Normal Distribution
nothing is called 95% until it has been counted
Concept

Sequential analysis — where it appears

Deciding whether to continue an experiment from the data it has already produced, which changes what the final estimate and interval mean. Every quantity computed at the end has a distribution that depends on the rule, so a rule that is not recorded leaves those quantities uninterpretable.

Named by 8 essays across 5 fields — each of them below, with the objects they name alongside it.

The stopping rule costs more than the weighting does. Coverage over 2000 runs of a trial whose variance ratio drifts by a factor of twenty, at three ways of deciding when to stop. Twelve blocks fixed in advance is the top line and reproduces what a trial of fixed length delivers. Stopping when the reported interval is short enough is the bottom line, and it costs between 3.0% and 5.5% of coverage — including for the rule that is told every block's true ratio, which is what says the shortfall belongs to the stopping and not to the weights. Stopping on a width predicted from the within-arm sums of squares is the middle line, and it is back at the fixed-length values. The standard error on each point is 0.49%.

A width the trial has to stop for

The weighting that covers at 94.9% on twelve blocks covers at 91.5% when the trial stops as soon as its interval is short enough — and so does the rule that is told every block's true variance ratio. The shortfall is the stopping, not the weights.

stop · Stopping Two promises, and no rule here keeps both. A fixed-width procedure promises two things: that the interval covers at its nominal rate, and that it is no wider than the width asked for. Over 1500 runs of the modelled weighting, a rule that stops when the interval it will report is short enough keeps the width — only 2.0% of runs come out wider than 0.34 — and covers at 91.13% against a nominal 95%. A rule that stops on a width predicted from the within-arm sums of squares covers at 94.80% and comes out wider than promised on 42.3% of runs. The two promises are in conflict because keeping the second one exactly requires conditioning on the very quantity that has to be independent of the stopping time for the first.

Stopping on the arms

The width a trial will report is predictable from quantities the interval is not about. A rule that stops on the prediction covers at 94.5% where one that stops on the interval covers at 91.5, and it costs two blocks and half of the width promise.

stop · Width Exact coverage, at every block size. Coverage of the interval each rule reports, at a nominal 95%, over 2,500 runs each with a standard error of 0.44 points. The blinded rule stops on the within-block contrasts and reports an interval built from the block means, and those two are independent whatever the rule does — so the interval is an ordinary t interval on b − 1 degrees of freedom and its coverage is exact. It is exact at every block size drawn. The interval a practitioner writes at the purely sequential rule's stopping time covers 91.72%, and Stein's two-stage rule is exact for the same reason as the blinded rule and spends 2.10 times the observations to be so. The bars are truncated at 86% so the differences can be seen.

The rule that cannot see the mean

A sequential rule stops when its own estimate of the spread is small, which is more often on the samples whose spread came out low — so the interval afterwards is short. There is a way to keep updating the estimate and stop being able to see the mean at all.

blind · Stopping What a fixed-width interval covers, by the number of blocks the trial ran before it stopped. Two thousand runs of each rule, the modelled weighting, a promise of 0.34. Reading its report: 4–8 blocks, 22.3% of runs, 78.2%; 9–12 blocks, 16.6% of runs, 90.4%; 13–16 blocks, 18.4% of runs, 96.2%; 17–20 blocks, 17.4% of runs, 96.0%; 21–28 blocks, 17.9% of runs, 96.4%; 29–36 blocks, 7.4% of runs, 99.3% — 91.45% overall. Reading the arms: 4–8 blocks, 0.0%, none; 9–12 blocks, 0.9%, 94.4%; 13–16 blocks, 30.4%, 95.6%; 17–20 blocks, 50.0%, 93.9%; 21–28 blocks, 18.0%, 94.4%; 29–36 blocks, 0.7%, 92.3% — 94.50% overall.

The trials that stopped early

A fixed-width trial that stops when its own interval is short enough covers 91.45% — an average of 78.2% among the 22.3% of runs that stop within eight blocks and 96% to 99% among those that run longer. Widening every interval by 17.1% brings the average to 95% and leaves the early stops at 85.6%, while 92.8% of runs now report an interval wider than the width they promised. Even doubling every interval leaves the early stops short.

stop · Width What the exactness costs, and the dial it is bought with. The median half-width of the interval each rule reports, at a requirement of 0.4 and a first look after 5 observations. The flat line is the interval a practitioner writes at the purely sequential rule's stopping time, which covers 91.72% rather than 95%. The curve is the blinded rule, which covers its nominal level at every block size: it reads b − 1 degrees of freedom where the other reads n − 1, and pays for the exactness in width. The best block size is 3, at 0.4712. Larger blocks give the stopping rule a better estimate and the interval a worse one, and the two costs go opposite ways, which is what puts the minimum in the middle.

What the blindfold costs

The exactly-covering rule pays for it in the width of the interval, and the block size is a dial between two costs that run in opposite directions. And on an interval whose width was fixed in advance, the same repair buys nothing at all.

blind · Nuisance What each stopping rule spends. The average number of observations the corner design takes before it stops, with the stopping rule reading pooled within-arm spreads (the upper bar) and each kind of period's own spreads (the lower), at a promised half-width of 0.3, 3,000 trials each. one ratio: 1,656 and 1,649; 2-fold, small blocks noisier: 1,020 and 1,015; 2-fold, large blocks noisier: 2,976 and 2,614; 4-fold, small blocks noisier: 751 and 669; 4-fold, large blocks noisier: 5,642 and 3,368; 8-fold, small blocks noisier: 711 and 490; 8-fold, large blocks noisier: 10,989 and 3,019. With one ratio the two rules are the same rule. With the large blocks noisier the pooled rule over-spends by a factor that grows with the difference, to 3.6 times at eightfold; with the small blocks noisier it over-spends less.

A stopping rule that reads the periods

A fixed-width trial in the corner stops when its estimated precision reaches the promise, and it estimated that precision from spreads pooled across a trial whose two kinds of period need not share a variance ratio. Read each kind of period's own spreads instead and the trial stops on 3,019 observations where it took 10,989, at an eightfold difference, with the interval it reports still covering 95.1%. The stopping time reads nothing but within-arm spreads, per period now as well as pooled, and that is all the exactness needed.

corner · Nuisance Three repairs at one requirement. Coverage against spend at a half-width of 0.4 and a first stage of 5, all exact. A floor on the second stage at 5, 8, 10, 12, 15, 18, 21, 24 observations covers 88.4%, 91.4%, 92.4%, 93.1%, 93.9%, 94.6%, 95.3%, 96.0% at 0.86, 0.94, 0.97, 1.00, 1.02, 1.05, 1.09, 1.14 times the oracle's spend. The second stage reading with t, with its fraction dialled from 0.2 to 1, covers 88.1%, 89.0%, 90.6%, 92.1%, 93.4%, 94.4%, 95.2%, 95.9%, 96.4% at 0.84, 0.91, 1.02, 1.17, 1.34, 1.52, 1.70, 1.89, 2.08 times. A constant at the end covers 88.4%, 89.4%, 90.4%, 91.3%, 92.1%, 92.9%, 93.6%, 94.2%, 95.0%, 95.7% at 0.86, 0.90, 0.94, 0.98, 1.02, 1.06, 1.10, 1.14, 1.20, 1.26 times.

A floor that reads the first stage

A three-stage stopping rule fails at a first stage of five because a low first reading asks for nothing more, and the only floors that repair it were set near the answer, by somebody who already knew roughly what the spread was. Inflating the first reading by Stein's own factor needs no such knowledge, and it does cut the experiments left stuck from a fifth to a fifteenth. But those still stuck cover less than before, the rule needs a different dial at every requirement, and a floor of 26 observations covers 96.5% at the spend where the rule covers 92.1%, even met by a spread it was not set for.

guarantee · Stopping Coverage lost against the honest rule. Coverage lost in points against the rule that reads only within-block spreads, on the same observations, blocks of 2, 6000 runs, at half-widths 0.5, 0.35, 0.25, 0.15. the reported spread in full: 0.32 ± 0.27, 2.53 ± 0.37, 5.33 ± 0.41, 3.75 ± 0.35. a fixed half: 0.72 ± 0.26, 2.43 ± 0.29, 0.98 ± 0.26, -0.25 ± 0.20. κ = (b − 1)/(N − 1): 0.37 ± 0.21, 1.18 ± 0.25, 0.58 ± 0.23, -0.18 ± 0.19. the same, phased in as b/(b + 20): 0.22 ± 0.10, 0.35 ± 0.16, 0.23 ± 0.18, -0.10 ± 0.19. the same, read in only after twenty blocks: 0.00 ± 0.00, -0.02 ± 0.04, 0.17 ± 0.19, -0.17 ± 0.19.

A rule that sets its own κ

A stopping rule may read the spread inside its blocks and may not read the spread between them, and the dial between the two costs nothing for a long stretch and then a lot. The share an efficiency argument recommends is each estimate's share of the degrees of freedom, and on blocks of two at a half-width of 0.15 that loses nothing and steadies the experiment by 30%. At 0.35 it loses 1.18 points. Deferring the forbidden reading until twenty blocks costs nothing measurable at any requirement, and it keeps most of the steadiness where steadiness can be had.

pace · Stopping

Named alongside it

The objects these essays reach for when they reach for this one.

CoverageFixed-width intervalStopping ruleBlindingDegrees of freedomInterval widthOptional stoppingSample sizeEfficiencyInverse variance weightingRandom sample sizeVariance ratio

All concepts