Normal Distribution
nothing is called 95% until it has been counted
Concept

Sharp null — where it appears

The hypothesis that the treatment changed nothing for any individual unit, which fixes each unit's outcome whatever arm it was assigned to. It is what makes a randomisation test's reference distribution computable, since every unit's outcome is then known under every assignment.

Named by 12 essays across 4 fields — each of them below, with the objects they name alongside it.

The allocations the rule could have made, from these exact patients. One 200-patient trial allocated by response-adaptive randomisation, re-randomised 999 times. No outcome is redrawn anywhere in this figure: each re-randomisation runs the same rule over the same patients in the same order, so what is drawn is the set of experiments that could have happened rather than a sampling distribution. The observed |z| is 1.417, 258 of the 999 re-randomisations reach it, and the p-value is (1 + 258)/(1 + 999) = 0.2590. The curve is the half-normal the ordinary analysis reads the same statistic against; its 5% point is 1.96 and this distribution's is 2.101.

The experiments that could have happened

An adaptive trial's allocation is a function of the outcomes it will later be compared against, so the ordinary analysis rejects a true null 9.2% of the time. Hold the outcomes fixed, re-run the rule that assigned them, and count — the same statistic against a reference distribution the trial could actually have drawn from is back at 4.0%.

exact · Reference Two diagnostics, one answer, two different moments. The two-chain statistic on a covariate probe and on the trial's own difference in arm means, at seven tolerances of a fourteen-unit rule, against the enumerated truth. Both are quiet wherever the set is one set and both fire wherever it is not, at every tolerance — which is what says the outcome probe is the same test rather than a resemblance of it. The difference between them is not accuracy and it is not power. It is when: the covariate probe can be run before a single outcome exists, when a practitioner can still loosen the rule or change the sampler, and it can be run again on a different function if it comes back quiet. The outcome probe runs after the trial, on the one column the trial produced, and what it can do with a positive verdict is repair the p-value rather than the design.

The statistic the p-value is about

The test for whether a balanced-assignment walk reaches its whole set is run on a covariate function chosen before the trial. Run on the difference in arm means it is the same test, and it is about the number the trial publishes.

after · Randomisation The probe a trial has is the probe a trial got. What the two-chain test says when it is run on the trial's own difference in arm means, over 24 outcomes on one fourteen-unit set. The set is in 2 mirror components — that is enumerated, not inferred — so every quiet reading is a miss. 29% of them are quiet. The reason is in the enumerated set rather than in the run: how far the two components are apart on a given probe ranges from 0.001 to 4.938 of a within-component spread across these outcomes, a factor of several thousand. Both covariate probes — chosen before any outcome existed, and replaceable if they had been quiet — report the split. An outcome cannot be chosen and cannot be replaced.

A probe nobody chose

On a set that is definitively in two pieces, seven of twenty-four outcomes report nothing at all. Every covariate probe reports it. What separates them is not accuracy — it is that one of them can be chosen and the other is what happened.

after · Randomisation The one thing a trial always reports is the one thing that survives. How wrong three p-values are when they are computed over the half of the admissible set a single walk can reach, rather than over all of it, at a fourteen-unit trial where the whole set can be enumerated. The two-sided p-value on the difference in arm means — the number a trial publishes — is wrong by exactly nothing, at every row, to machine precision. That is not luck: the two components are complement pairs and the difference in arm means is exactly negated by the complement, so the distribution of its absolute value is the same on both. A one-sided p-value on the same statistic is out by as much as 0.112, and the largest response observed in the treated arm — a safety reading rather than an effect, and the one statistic here that is not odd under the complement — by as much as 0.172. The defect survived because the commonest thing anybody computes is the one quantity it cannot touch.

Half a reference distribution

A walk that reaches half its admissible set reports the two-sided p-value exactly right, to the last digit, for ever. A one-sided one it puts on the wrong side of five per cent about once in thirty.

after · Reference Where the +1 matters, and why nobody has noticed that it does. The true size of the two rules at every B, computed rather than simulated: under the null the count of re-randomisations reaching the observed statistic is uniform over {0 … B}, so both sizes are integer arithmetic. With the +1 the size is (⌊α(B+1)⌋)/(B+1), which never exceeds 5%. Without it the size is (⌊αB⌋+1)/(B+1), which is larger except at B = 19, 39, 59 — the values with B + 1 a multiple of 1/α, and the values everybody uses. At B = 19 the two rules are the same rule; at B = 20 the uncorrected one is a 9.5% test. The marks are simulated on 500 trials of 120 patients, as the second route to the same numbers.

The plus one and the round number

A sampled randomisation test counts the observed allocation as one of its own reference draws, and the correction is invisible at B = 19, 39, 59 and 999 — every value anybody uses. At B = 20 the version without it is an 8.00% test where the corrected one is 3.80%, and the convention protecting everybody is a preference for round numbers minus one.

exact · Reference How long a walk has to be given. Every equal split of twelve units is enumerated, the 410 admissible ones are found, the transition matrix is built, and the distance from uniform is computed exactly at each step — no simulation anywhere. The walk is started at the least balanced admissible assignment, which is the state a rejection sampler is least likely to have handed it and the one a burn-in has to cover. It is 0.0849 away after twenty steps and 0.00008 after ninety. A real cost, and a small one, and naming it is what stops it being assumed to be zero.

Walking the admissible set

A rerandomisation test hunts for admissible assignments and throws away the rest. A walk visits them instead — and it is exactly uniform only because it stands still when a proposal fails, which is the step that looks like waste.

joint · Randomisation Where the set stops being one set. How many of the 3,432 equal splits of fourteen units a balancing rule admits, as the tolerance tightens, with the number of components single swaps leave it in. The set falls from 886 to 84 assignments, and somewhere in that fall it stops being connected: at 0.8 it is in 2 pieces and every assignment's complement is in the other one. Nothing about the rule changes at that point and nothing a chain reports changes either, which is the whole difficulty — the acceptance rate, the stationary distribution and the detailed balance are all in order on both sides of it.

Before the trial and after

The same diagnostic run at two moments answers two different questions. Before, a positive verdict changes the design. After, it changes which number gets reported — and only for the numbers the defect can reach.

after · Assignment The allocations this trial could have made, and the ones it could not. One 120-patient trial allocated by minimisation at p = 1, re-randomised 399 times. No outcome is redrawn anywhere in this figure: each re-randomisation runs the rule again over the same patients in the same order with the same recorded factors, so what is drawn is the set of experiments that could have happened. The bars are that set; the outline is what shuffling the labels gives, which is the reference distribution of a coin and is what every off-the-shelf permutation routine assumes. The coin's is wider — its 5% point is 1.95 against the rule's 1.09 — because a coin's allocations are less balanced and a less balanced allocation gives a larger statistic. Reading this trial against it makes the test conservative rather than anti-conservative, which is the opposite error from the outcome-adaptive case and for the same structural reason.

The reference the covariates supply

Hold the outcomes fixed, re-run the rule that assigned them, count. The same construction cost nineteen points of power in the adaptive field, because its rule chased outcomes and its critical value depended on a rate nobody has. Here the rule reads only what was recorded before anything happened, and the same unadjusted statistic goes from 20.3% power to 55.0% by being read against the right distribution.

covadapt · Assignment An exact test rejecting a true hypothesis a fifth of the time. How often each analysis reports an effect when the average treatment effect is exactly zero and the effect varies between units, at 150 units with 25% treated. The permutation test on the difference in means reads 4.20% where the effect is constant — where the two nulls coincide and its exactness applies — and 22.93% where the effect varies with a standard deviation of 3. The same test on the studentised difference reads 6.27% there, and the ordinary large-sample t, which makes no exactness claim at all, reads 6.60%.

The null the exactness is for

A permutation test is exact under the hypothesis that the treatment changed nothing for anybody. Under the hypothesis it changed nothing on average, with a quarter of the units treated and the effect varying between them, it rejects a true null 22.93% of the time.

exact · Nuisance One statistic that is right under both hypotheses. Rejection rates for both statistics under both nulls, at 25% of 150 units treated, with the weak-null readings taken at an effect spread of 3. The difference in means is exact under the sharp null and rejects 22.93% of true weak nulls. The studentised difference is exact under the sharp null — 4.07% — and reads 6.27% under the weak one. The repair is a change of statistic inside the same construction: the same re-randomisations, the same fixed outcomes, a different number compared across them.

A statistic that is exact twice

Dividing the difference in means by its own separate-variance standard error before permuting takes the rejection rate under a true weak null from 20.47% to 6.07%, keeps the exactness under the sharp null at 4.07%, and costs 0.8 points of power against a real effect. At an even split it changes nothing at all, in every draw.

exact · Nuisance How often four permutation tests of a covariate-adjusted effect reject a true weak null, effect varying with the outcome, 25% of 150 treated. At an effect spread of 0, 0.5, 1, 1.5, 2: the adjusted coefficient's permutation test rejects 5.9%, 10.8%, 13.2%, 16.6%, 14.8%; over its sandwich standard error, 5.7%, 5.0%, 3.4%, 3.4%, 3.0%; the coefficient with a treatment-by-covariate interaction, 5.1%, 10.2%, 12.8%, 14.7%, 14.0%; the same over its sandwich standard error, 5.7%, 5.2%, 5.2%, 5.1%, 3.9%.

The repair, with a covariate in it

Studentising a permutation test's statistic made the difference in means exact under the sharp null and nearly right under the weak one. Carried to the covariate-adjusted estimate a trial actually reports, the repair works only if the adjustment includes the treatment-by-covariate interaction. With a quarter of 150 units treated and effects that vary with the outcome, the adjusted estimate's permutation test rejects 14.8% of true weak nulls; over its sandwich standard error, 3.0%; the interacted estimate over its sandwich, 3.9%. When the effect varies with the covariate itself, even an even split does not protect: the adjusted test rejects none of a thousand true weak nulls, and at an average effect of 0.2 its power is 5.2% against the interacted, studentised test's 40.7%.

exact · Nuisance How often an adjusted permutation test rejects a true sharp null when the covariate is chosen after randomisation, held fixed or re-chosen inside every shuffle. 150 units, half treated, six baseline covariates of which three are prognostic; 2,000 experiments and 99 re-randomisations each, read on the adjusted coefficient over its sandwich standard error. Choosing the covariate most correlated with the outcome: 5.0% held fixed, 5.0% re-chosen. Choosing the covariate with the smallest residual variance: 5.5% held fixed, 5.3% re-chosen. Choosing the covariate most imbalanced between the arms: 3.0% held fixed, 4.8% re-chosen. Choosing the covariate with the smallest adjusted p-value: 10.4% held fixed, 5.1% re-chosen. Naming the most prognostic covariate in advance: 5.5%.

The covariate chosen afterwards

A trial that chooses which baseline covariate to adjust for after randomisation has two ways to run its permutation test: hold the chosen covariate fixed across the shuffles, which is what an analysis that reports adjusting for it actually does, or re-make the choice inside every shuffle, which is exact by construction. Holding it fixed, a choice made on significance rejects 10.4% of true sharp nulls, a choice made on imbalance only 3.0% — and pays seven and a half points of power for it — while a choice that never reads the arm labels is the same test either way. The interacted, studentised statistic keeps its weak-null behaviour only when the covariate chosen is the one the effect varies with.

exact · Nuisance

Named alongside it

The objects these essays reach for when they reach for this one.

Randomisation testReference distributionPermutation testExact testAssignment mechanismCovariate balanceImbalanceMarkov chain Monte CarloStatistical powerConnected componentExact enumerationMonte Carlo

All concepts