Normal Distribution
nothing is called 95% until it has been counted
Concept

Two-stage design — where it appears

An experiment that takes a first batch, estimates something from it, and lets that estimate decide the size or the settings of the second. Whether the second stage may read the first's estimate of the thing being reported is the question every such design turns on.

Named by 9 essays across 4 fields — each of them below, with the objects they name alongside it.

The best of 8 arms, tested as though it were the only one. 8,000 trials with no effect in any arm. Stage one runs 8 arms at 60 each, the best is carried forward, and stage two adds 30 more to it and to the control. The histogram is where the final statistic lands and the curve is the standard normal it is being read against — shifted right, because the arm was chosen for being ahead. 10.5% of these trials clear 1.96 against a claimed 5%, and the value that actually holds the rate for this design is 2.313.

Dropping the losers

Carrying the best of eight arms forward and testing it at 1.96 rejects a true null 10.3% of the time — the hypothesis was chosen by looking at the data, so the statistic is a maximum wearing a single comparison's clothes. The value that holds the rate is 2.313, and it has to be solved for.

adaptive · Stopping One rule keeps its promise and the other keeps its budget. Both stopping rules at five requirements, 1,500 experiments each, with a first stage of 5. The upper curve is the two-stage rule: 97.1%, 96.2%, 95.9%, 96.1%, 96.0% — at or above 95% at every point, which is a theorem rather than a tendency, because its interval is built from a spread estimated before the stopping point was chosen. It pays 2.06×, 2.01×, 2.00×, 1.99×, 1.99× the observations that knowing σ would need. The lower curve is the rule that re-estimates after every observation: 94.5%, 89.3%, 91.0%, 91.5%, 94.3%, on 0.98×, 0.87×, 0.88×, 0.93×, 0.96×. The second rule is the one anybody would run and the first is the one whose claim is true.

Stopping when it is precise enough

An experiment that runs until its estimate is precise enough is the natural design and the one with a theorem against it. Its two-stage cousin keeps its promise exactly, for every unknown spread, and pays twice the observations for it.

guarantee · Stopping One experiment finding out where to look. A single run of the fully sequential design: 40 runs, the first 8 placed at the guess K = 1, then the model refitted and the design revised after every 2. The marks are the settings the runs were made at. The horizontal lines are where a design built at the truth K = 3 would have put them — 1.875 and 10.00 — and the rule walks onto them without being told: its estimate of K after the first eight runs was 2.694, and by the end 2.765 against a truth of 3. The whole experiment is 96.5% as efficient as the design that knew the answer, where running all 40 at the guess would have been 81.1%.

The design that stops guessing

Every repair so far protects a guess. The alternative is to run part of the experiment, estimate the parameter from it, and design the rest at the estimate — which recovers most of what a threefold wrong guess costs, and has a best moment to stop guessing that is earlier than anyone expects.

robust · Local design What a pilot buys, σ = 1 against 3. Each point is 6,000 two-stage experiments of 100 units: a pilot of m per arm, then the rest split by the pilot's own estimate of the two spreads. Above the line the pilot has made the experiment worse than not bothering. The best pilot here is 8 per arm at 0.809, against 0.800 for a designer who knew the spreads — so the rule recovers 96% of what knowing them is worth. A larger pilot estimates the ratio better and has less left to apply it to, which is why the curve turns.

Allocating on a guess

Every allocation rule in this field is a function of quantities the experiment is being run to find out. Fed a pilot's estimate of them, the rule that minimises the variance makes the experiment worse than not bothering — until the arms differ by about a factor of two, which is further than anyone would guess.

allocation · Allocation Four intervals at one stopping time. 3,500 experiments under the sequential rule with a first stage of 5 and a required half-width of 0.4, which is a demand that knowing σ would meet with 24.0 observations and which the rule meets with 20.8. The rule's own interval covers 90.3%. Replacing the fixed width by a t interval on the same data gives 92.0%. Keeping the rule's own random sample size and drawing a fresh sample of that size gives 89.8% at the fixed width and 95.5% for a t interval — so the sample size being random costs nothing, and the sample size being chosen by the data the interval is built from costs the rest. The spread estimated at the stopping moment is 17.1% below the truth, which is the same fact one level down.

The interval after a stop it chose

A rule that stops when the estimated precision is good enough stops on the samples whose estimate was small. Its interval covers 90% and claims 95%, and a fresh sample of the same random size covers 95.4%.

guarantee · Stopping What the interval covers, after a design that read the data. 800 experiments of 12 runs, all at the same truth. The first pair is a design fixed in advance; the second is one whose settings were chosen from the first stage's own outcomes. If choosing the design from the data broke the inference, the second pair would sit below the first, and it does not — 91.3% against 92.6%. What does move the coverage is the shape of the interval rather than the design: the Wald interval assumes the estimate is normal around its own standard error and falls short under both designs, and the profile interval, computed from the same residual sums of squares with no derivative in it, covers 94.3% and 94.9%.

What a design chosen from the data costs

Two earlier measurements found what happens when a rule reads the data, and the error rate broke both times. A design that reads the data to decide where to put its runs breaks nothing — and the control that proves it also finds what the real shortfall is.

robust · Local design A third stage, between the two rules. Four rules for one requirement: a half-width of 0.4 at 95%, which knowing σ would meet with 24.0 observations, after a first stage of 5. Each bar is the mean number of observations as a multiple of that, and the note beside it is the coverage. The two-stage rule spends 2.03× and covers 95.42%, computed exactly. The three-stage rule — a second stage to half of what the first spread asks for, the spread re-read on everything, a third stage to the rest — spends 0.86× and covers 88.42%; given 8.50 more observations in every experiment's last stage it covers 95.00% on 1.20×. The rule that re-estimates after every observation spends 0.88× and covers 90.5% over 4,000 drawn experiments; it is the only row that has to be drawn, because it is the only one whose stopping point shares observations with the mean.

The fifth a third stage never reaches

A three-stage stopping rule keeps the two-stage rule's independence and the sequential rule's budget. With a first stage of five it covers 88.4% — worse than the rule it was meant to repair — because in a fifth of experiments the second stage buys nothing.

guarantee · Stopping The error of the chosen arm's estimate against a fixed second stage's size, and a second stage sized by the interim, 8 arms, one ahead by half a standard deviation. Fixed second stages of 10, 25, 40, 55, 70 patients: both stages combined err by 0.172, 0.155, 0.143, 0.133, 0.126; the mixture-shrunk estimate by 0.208, 0.183, 0.164, 0.151, 0.140. The rule that sizes the second stage by the interim spends 41.4 on average and errs by 0.142 and 0.159; its second stage alone errs by 0.225.

A second stage sized by the first

A drop-the-losers trial can size its second stage by what the first stage says about the arms' shape — more patients when the chosen arm's lead looks like luck, fewer when it looks real. The rule does what it was built to do: at sixteen arms with a genuine winner it spends 51.0 patients a group when the interim chose the wrong arm and 24.2 when it chose the right one. And it buys nothing. Against a fixed second stage of the same average size, every estimate of the chosen arm errs by the same to within a few thousandths; the published combination's bias rises from 0.014 to 0.022, and the second stage alone, still unbiased, errs by 0.319 against 0.281.

adaptive · Curse How far the published estimate of the chosen arm sits from the best arm's effect, by how six hundred patients are split between choosing and estimating. Eight arms and a control, six hundred patients: a stated number an arm in the first stage and the rest split equally between the chosen arm and the control in the second. Root mean squared error of the published estimate about the best arm's true effect, over 5,000 trials a split. All eight alike: best at 16 an arm (0.092), 0.183 at the familiar sixty; One modestly ahead: best at 56 an arm (0.151), 0.153 at the familiar sixty; One clearly ahead: best at 48 an arm (0.146), 0.154 at the familiar sixty; Two clearly ahead: best at 40 an arm (0.128), 0.151 at the familiar sixty; Spread evenly: best at 40 an arm (0.131), 0.146 at the familiar sixty. The split that guards every world puts 32 an arm before the choice and costs at most 21.3% over each world's best.

Patients spent before the choice

A trial that picks the best of eight arms and then confirms it splits its patients between choosing and estimating, and the familiar shape — sixty an arm before the choice, thirty after — spends ninety per cent of a six-hundred-patient budget on the choice. Measured against the effect a reader is told about, the best arm's, that split is nearly right when one arm is modestly ahead and twice as wrong as it needs to be when the arms are alike. A split of thirty-two an arm before the choice is never more than 21.3% worse than the best split for whichever world holds; sixty an arm can be 98.8% worse. The two targets an estimate could serve — the chosen arm's effect and the best arm's — pull the split in opposite directions, and only one of them is what gets published.

adaptive · Curse

Named alongside it

The objects these essays reach for when they reach for this one.

Adaptive designExperimental designSample sizeCoverageMichaelis–MentenMonte CarloSelection biasConfidence intervalFixed-width intervalRandom sample sizeSequential designStopping rule

All concepts