Normal Distribution
nothing is called 95% until it has been counted
Series

Nuisance — the series

12 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. The cheap repair needs a number nobody has. The obvious alternative to re-randomising is to simulate the design under its null once and use the critical value that comes out — which is what the arm-dropping design does, where the critical value has to be solved for and is 2.313. It does not transfer here. The rule chases outcomes, so how imbalanced the allocation gets depends on how often anything succeeds, and the critical value moves from 1.668 at a success rate of 0.05 to 2.718 at 0.8. Calibrated at 0.3 and used at 0.8 the test's real size is 12.4%; used at 0.05 it is 0.12%. The randomisation test needs none of this, because it conditions on the outcomes that happened rather than on a rate they were supposed to come from.

    What the exactness buys

    Against a z test calibrated to reject exactly 5% of true nulls on this design, the randomisation test loses nineteen points of power. What it buys is that the calibration needs the success rate — which moves the critical value from 1.668 to 2.718 and is the quantity the trial was run to find out.

    part 1 · exact
  2. What the exactness costs, and the dial it is bought with. The median half-width of the interval each rule reports, at a requirement of 0.4 and a first look after 5 observations. The flat line is the interval a practitioner writes at the purely sequential rule's stopping time, which covers 91.72% rather than 95%. The curve is the blinded rule, which covers its nominal level at every block size: it reads b − 1 degrees of freedom where the other reads n − 1, and pays for the exactness in width. The best block size is 3, at 0.4712. Larger blocks give the stopping rule a better estimate and the interval a worse one, and the two costs go opposite ways, which is what puts the minimum in the middle.

    What the blindfold costs

    The exactly-covering rule pays for it in the width of the interval, and the block size is a dial between two costs that run in opposite directions. And on an interval whose width was fixed in advance, the same repair buys nothing at all.

    part 2 · blind
  3. The overshoot is the last block size and nothing else. A run stops at a multiple of its own block sizes and cannot land between them, so it ends past its own target by about half a block. Fixed sizes overshoot by 1.5, 2.2, 3.1, 4.7, 8.5 observations as the size goes 2, 3, 5, 8, 16. Every schedule here ends in blocks of two and every one of them lands where blocks of two land — 1.62, 1.32, 1.37 against 1.48 — while having spent most of the run inside blocks four and eight times larger. That is the one thing on this page a schedule genuinely takes from both ends.

    What a schedule actually buys

    Big blocks early and small blocks late is the right instinct and it does not take both ends of the trade, because there are not two ends to take. What it does take is the overshoot — about four per cent of the observations — and a steadier stopping point.

    part 3 · pace
  4. Three sets of weights, five designs, and no estimator that is exact everywhere. Coverage of the same interval under three weightings. h_b is the inverse variance when the arms share a variance or the allocation is constant; equal weights are right when every block has the same two counts; the estimated precision weights are right in the limit and exact nowhere, because the decomposition needs the weights to be the constants they are only estimating. In the corner — two variances, changing sizes, changing allocation — the two exact estimators are the ones that miss, at 98.45% and 95.65%, and the one with no theorem behind it is at 95.05%. That is the whole statement: there is an exact estimator under either condition, and none under both.

    Which weights are the inverse variances

    There is an exact estimator when the two arms share a variance and another when every block has the same two counts, and between them they cover every trial anybody designs on purpose. In the corner where neither holds, both cover 98.45% instead of 95%, and the only estimator at its level is the one with no theorem behind it.

    part 4 · contrast
  5. Exact in the corner, where nothing was. Coverage of a nominal 95% interval on five designs, at a required half-width of 0.3. The first four are the two-arm field's own and the fifth is its corner — two variances, block sizes that swing by eight, and an allocation that alternates between five to one and one to five — where neither of that field's two conditions holds. The effective-size weights over-cover there at 98.40%; the weights h_b(λ) = (1/m_A + λ/m_B)⁻¹ cover at 94.84%, and at 94.84% when λ is estimated from the within-arm contrasts rather than known. Nothing here is supposed to move.

    Weights that need only a ratio

    A fixed-width interval about a difference is exact under either of two conditions and under neither in the corner. It is exact there too, and the only thing it needs is how much larger one arm's variance is than the other's.

    part 5 · corner
  6. A wrong weight costs width; a random weight costs level. Five weightings on a trial whose variance ratio drifts by a factor of 20.1 between the first block and the last, over 4000 runs. The rule that knows every λ_b covers at 95.1% and sets the width. One ratio for the whole trial is wrong for every block and costs nothing in level — 94.8% — while being 20% wider; equal weights are calibrated by an identity and 22% wider. The ratio estimated inside each block is the only rule aimed at the quantity that actually varies, and it is the only one that misses the level, at 92.0%: a weight computed from a handful of degrees of freedom is mostly noise, and noise in a weight is not a wrong weight. Modelling the drift across blocks recovers the oracle's width at 94.8%.

    A ratio that changes between blocks

    A wrong weight costs width and a random weight costs level. The rule aimed at the quantity that actually varies is the only one that misses its own coverage, and the rule that models it across blocks recovers the whole of what knowing it is worth.

    part 6 · blocks
  7. An exact test rejecting a true hypothesis a fifth of the time. How often each analysis reports an effect when the average treatment effect is exactly zero and the effect varies between units, at 150 units with 25% treated. The permutation test on the difference in means reads 4.20% where the effect is constant — where the two nulls coincide and its exactness applies — and 22.93% where the effect varies with a standard deviation of 3. The same test on the studentised difference reads 6.27% there, and the ordinary large-sample t, which makes no exactness claim at all, reads 6.60%.

    The null the exactness is for

    A permutation test is exact under the hypothesis that the treatment changed nothing for anybody. Under the hypothesis it changed nothing on average, with a quarter of the units treated and the effect varying between them, it rejects a true null 22.93% of the time.

    part 7 · exact
  8. One statistic that is right under both hypotheses. Rejection rates for both statistics under both nulls, at 25% of 150 units treated, with the weak-null readings taken at an effect spread of 3. The difference in means is exact under the sharp null and rejects 22.93% of true weak nulls. The studentised difference is exact under the sharp null — 4.07% — and reads 6.27% under the weak one. The repair is a change of statistic inside the same construction: the same re-randomisations, the same fixed outcomes, a different number compared across them.

    A statistic that is exact twice

    Dividing the difference in means by its own separate-variance standard error before permuting takes the rejection rate under a true weak null from 20.47% to 6.07%, keeps the exactness under the sharp null at 4.07%, and costs 0.8 points of power against a real effect. At an even split it changes nothing at all, in every draw.

    part 8 · exact
  9. How often four permutation tests of a covariate-adjusted effect reject a true weak null, effect varying with the outcome, 25% of 150 treated. At an effect spread of 0, 0.5, 1, 1.5, 2: the adjusted coefficient's permutation test rejects 5.9%, 10.8%, 13.2%, 16.6%, 14.8%; over its sandwich standard error, 5.7%, 5.0%, 3.4%, 3.4%, 3.0%; the coefficient with a treatment-by-covariate interaction, 5.1%, 10.2%, 12.8%, 14.7%, 14.0%; the same over its sandwich standard error, 5.7%, 5.2%, 5.2%, 5.1%, 3.9%.

    The repair, with a covariate in it

    Studentising a permutation test's statistic made the difference in means exact under the sharp null and nearly right under the weak one. Carried to the covariate-adjusted estimate a trial actually reports, the repair works only if the adjustment includes the treatment-by-covariate interaction. With a quarter of 150 units treated and effects that vary with the outcome, the adjusted estimate's permutation test rejects 14.8% of true weak nulls; over its sandwich standard error, 3.0%; the interacted estimate over its sandwich, 3.9%. When the effect varies with the covariate itself, even an even split does not protect: the adjusted test rejects none of a thousand true weak nulls, and at an average effect of 0.2 its power is 5.2% against the interacted, studentised test's 40.7%.

    part 9 · exact
  10. How often an adjusted permutation test rejects a true sharp null when the covariate is chosen after randomisation, held fixed or re-chosen inside every shuffle. 150 units, half treated, six baseline covariates of which three are prognostic; 2,000 experiments and 99 re-randomisations each, read on the adjusted coefficient over its sandwich standard error. Choosing the covariate most correlated with the outcome: 5.0% held fixed, 5.0% re-chosen. Choosing the covariate with the smallest residual variance: 5.5% held fixed, 5.3% re-chosen. Choosing the covariate most imbalanced between the arms: 3.0% held fixed, 4.8% re-chosen. Choosing the covariate with the smallest adjusted p-value: 10.4% held fixed, 5.1% re-chosen. Naming the most prognostic covariate in advance: 5.5%.

    The covariate chosen afterwards

    A trial that chooses which baseline covariate to adjust for after randomisation has two ways to run its permutation test: hold the chosen covariate fixed across the shuffles, which is what an analysis that reports adjusting for it actually does, or re-make the choice inside every shuffle, which is exact by construction. Holding it fixed, a choice made on significance rejects 10.4% of true sharp nulls, a choice made on imbalance only 3.0% — and pays seven and a half points of power for it — while a choice that never reads the arm labels is the same test either way. The interacted, studentised statistic keeps its weak-null behaviour only when the covariate chosen is the one the effect varies with.

    part 10 · exact
  11. One pooled ratio, two periods. The calibration factor of weights built from one variance ratio pooled across the trial — the factor by which the interval's own scale estimate overstates the estimate's variance — when arm B's ratio is F times higher in one kind of period than the other, closed form on the corner's block compositions. With the small A-heavy blocks the noisier, the factor rises: 1.00 at F = 1, 1.50 at F = 1.5, 2.07 at F = 2, 3.12 at F = 3, 3.91 at F = 4, 4.86 at F = 6, 5.44 at F = 8. With the large B-heavy blocks the noisier, it falls: 1.00 at F = 1, 0.74 at F = 1.5, 0.65 at F = 2, 0.58 at F = 3, 0.55 at F = 4, 0.53 at F = 6, 0.53 at F = 8. A ratio estimated per period gives weights that are the inverse variances again, and its factor is exactly 1 at every F.

    A ratio that changes with the period

    The corner's exact interval needs one variance ratio, and the corner's own alternating allocation makes two kinds of period that need not share one. When they differ the pooled ratio is precise and wrong: at a fourfold difference the interval covers 100.0% one way round and 85.1% the other. A ratio estimated per period is exact again, blind to the effect, and costs under a tenth of the width — and a ratio that merely wanders at random costs width and no coverage at all.

    part 11 · corner
  12. What each stopping rule spends. The average number of observations the corner design takes before it stops, with the stopping rule reading pooled within-arm spreads (the upper bar) and each kind of period's own spreads (the lower), at a promised half-width of 0.3, 3,000 trials each. one ratio: 1,656 and 1,649; 2-fold, small blocks noisier: 1,020 and 1,015; 2-fold, large blocks noisier: 2,976 and 2,614; 4-fold, small blocks noisier: 751 and 669; 4-fold, large blocks noisier: 5,642 and 3,368; 8-fold, small blocks noisier: 711 and 490; 8-fold, large blocks noisier: 10,989 and 3,019. With one ratio the two rules are the same rule. With the large blocks noisier the pooled rule over-spends by a factor that grows with the difference, to 3.6 times at eightfold; with the small blocks noisier it over-spends less.

    A stopping rule that reads the periods

    A fixed-width trial in the corner stops when its estimated precision reaches the promise, and it estimated that precision from spreads pooled across a trial whose two kinds of period need not share a variance ratio. Read each kind of period's own spreads instead and the trial stops on 3,019 observations where it took 10,989, at an eightfold difference, with the interval it reports still covering 95.1%. The stopping time reads nothing but within-arm spreads, per period now as well as pooled, and that is all the exactness needed.

    part 12 · corner

All series