Stopping when it is precise enough

Every sample size on this site so far has been a number decided in advance. The design field computes one from a stated effect and a stated power; the adaptive field re-estimates one at an interim look and shows what that costs; and both keep the arithmetic that decides when to stop outside the data.

The obvious thing an experimenter actually does is different. Run until the answer is precise enough — until the interval is narrow enough to be useful, until the standard error is a small enough share of the estimate — and then stop. That is a stopping rule whose trigger is the estimate itself, and it is the adaptation this site has not yet measured.

Two rules for one requirement #

The requirement is stated as a width: produce an interval for a normal mean of half-width d = 0.4, at 95%. If σ were known it would take 24.0 observations and there would be nothing to discuss. It is not known, and the two ways of proceeding differ in exactly one respect.

Stein’s two-stage rule takes n₀ observations, estimates the spread once from them, and takes as many more as that one estimate asks for: N = max(n₀, ⌈t²s₀²/d²⌉), then reports x̄ₙ ± d.

The purely sequential rule takes one observation at a time, re-estimating the spread after each, and stops the first time n reaches z²s²ₙ/d².

The second is what anybody would write. The first has a theorem.

The theorem, and what it rests on #

Stein’s rule covers at least its nominal level for every σ, and the reason is a sentence: √N(x̄ₙ − μ)/s₀ has a t distribution on n₀ − 1 degrees of freedom whatever the second stage does, because s₀ is computed from the first stage alone and the mean of the whole sample is independent of it. The stopping point is a function of s₀; the error the interval has to cover is not.

That independence is the whole of it, and it is exactly what the sequential rule gives up.

Counted at nine combinations of first stage and required width, the two-stage rule covers 97.0%, 95.9%, 95.6%, 98.3%, 95.6%, 95.5%, 99.9%, 96.6% and 96.0% — at or above its claim at every one of them, which is what a theorem looks like when it is measured rather than cited. The sequential rule covers 93.7%, 90.8%, 93.9%, 97.5%, 93.2%, 94.3%, 99.9%, 95.1% and 94.9%.

What the exactness costs #

The two-stage rule pays for its theorem in observations, and the bill is large.

At d = 0.4 with a first stage of five, the two-stage rule averages 48.2 observations — 2.01 times what knowing σ would need — with a standard deviation of 32.7 and a 5th-to-95th range of 9 to 111. Everything is committed to one variance estimate from five observations, and a variance estimate from five observations is a wild thing: a low draw ends the experiment almost immediately and a high one runs it for a hundred.

The sequential rule averages 21.3 observations, 0.89 of the oracle’s requirement, with a standard deviation of 8.5. Every new observation revises the target, so the estimate the rule acts on is always the best one available and the total lands close to the right answer.

Twice the observations, or an interval that is four points short of its claim. That is the trade, and it is not obviously worth taking in either direction.

Why the sequential rule undershoots #

The sequential rule’s average sample size is below the oracle’s requirement — 21.3 against 24.0 — and that is not a rounding artefact. It is the mechanism of everything the next essay measures, visible one level earlier.

The rule stops the first time n reaches z²s²ₙ/d². It approaches that boundary from below, taking one observation at a time, so it stops on the first n whose sample spread happens to be small enough — and among all the samples that could have been drawn at that n, the ones that stop are the ones with a low s. The estimated variance at the moment of stopping averages 0.850 of the truth.

A rule that stops when its own noise estimate is low is a rule that stops early, and the average of a quantity conditioned on it having triggered a threshold is not the average of that quantity. The undershoot in the sample size and the shortfall in the coverage are the same fact counted twice.

The two-stage rule cannot do this. Its s₀ is drawn once, before any decision, and it overshoots on average for the opposite reason: N is a convex function of s₀ — it goes as s₀² — so the average of N over draws of s₀ is above N at the average s₀, by Jensen’s inequality and by nothing else. Its 2.01× is not a design choice; it is the curvature of a square.

The overspend is t² over z² #

The two-stage rule’s 2.01× is not a measurement that has to be repeated at every setting. It has a closed form, and the form says the factor is a property of the first stage and of nothing else.

The rule takes N = ⌈t²s₀²/d²⌉ with t on n₀ − 1 degrees of freedom, and s₀² is an unbiased estimate of σ², so E[N] is t²σ²/d² up to the rounding. The oracle needs z²σ²/d². The ratio is

t²(n₀ − 1) / z²

with no σ, no d and no target width in it. At a first stage of five that is 2.7764²/1.95996² = 2.006, against a counted 2.01 and an oracle requirement of 24.0 that becomes 48.1 — the counted figure is 48.2, the extra tenth being the ceiling and the max. At a first stage of twenty it is 2.0930²/1.95996² = 1.140, against counted overspends of 1.14 and 1.18.

Two things follow immediately and neither needs another simulation.

The overspend does not depend on how tight the requirement is. Halving d quadruples both the oracle’s requirement and the rule’s, so the ratio is untouched. The trade this essay describes is the same trade at every precision, which is why the slider moves both distributions together.

And it can be designed to any level. Holding the overspend below 10% needs t² < 1.1z², which is t < 2.056 and therefore n₀ ≥ 27; below 5% needs n₀ ≥ 52. So an experimenter who can afford twenty-seven observations before committing gets Stein’s exact guarantee for a tenth more than the oracle’s requirement, and the 2.01× that makes the rule look unaffordable is entirely the price of a first stage of five.

That reframes the comparison. The theorem is not expensive; a first stage of four degrees of freedom is. What an experimenter is really choosing between is a guarantee that costs a fixed, computable premium and a rule that costs nothing and delivers four points less than it claims.

And the whole distribution is a chi-square #

The same expression gives more than the mean. N is a fixed multiple of s₀², and s₀² on n₀ − 1 degrees of freedom is σ²χ²(n₀−1)/(n₀−1), so the sample size the two-stage rule produces is a scaled chi-square and its whole distribution is known before the experiment.

Its coefficient of variation is √(2/(n₀−1)) = 0.707 at a first stage of five, against a counted 32.7/48.2 = 0.679 — the small shortfall being the floor at n₀ that truncates the left tail. Its 5th and 95th percentiles are 0.1777 and 2.3719 times its mean, which at 48.1 gives 8.6 and 114 against the counted 9 to 111.

So the alarming range is not a surprise about variance estimates; it is a chi-square on four degrees of freedom, drawn once and multiplied by a constant. And the repair is the same one: the spread falls as 1/√(n₀ − 1), so a first stage of twenty-seven — the one that caps the overspend at a tenth — also brings the coefficient of variation to 0.28, and the experiment stops being a lottery about its own length.

The first stage is the only tuning either rule has #

Both rules take n₀ as an input and nothing else, so whatever can be improved has to be improved there.

At n₀ = 20 the two-stage rule spends 1.18 and 1.14 times the oracle’s requirement at the two tighter widths rather than 2.01 and 1.99, and its coverage is 96.6% and 96.0% rather than 95.9% and 95.6%. The sequential rule’s shortfall also shrinks: 95.1% and 94.9% against 90.8% and 93.9%.

So both defects are defects of a small first stage, and both are repaired by making it larger — at which point the experiment is mostly a fixed-size experiment and the rule has less to do. The uncomfortable case is exactly the one an experimenter faces: a first stage small enough to be worth running before committing.

The same question, inside a design #

The rules above stop an experiment that has nothing to decide except when to stop. The field’s own model has both decisions available: where the runs go and how many there are, each of which can be fixed in advance or read off the data.

The setup is the one the robust field uses. A two-parameter non-linear model, a true K of 3, a guess of 1, runs taken in batches of four after a first stage of eight, and a stated precision — stop when the standard error of K̂ is 8% of the estimate.

Fixing the design at the guess and letting the rule stop takes 58.2 runs. Redrawing the design at the current estimate after every batch takes 46.2 — a saving of 21%, bought entirely by not spending the experiment at settings chosen for a value of K that is wrong by a factor of three.

That is the design half of the adaptation, and it is free in the sense the robust field established: a design chosen from the data leaves the interval afterwards no worse than a design fixed in advance. What is not free is the other half, and the third row of that figure is the beginning of the next essay.

What the saving is, and what it is not #

The 21% is worth reading carefully, because two different things could produce it and only one of them did.

Both experiments stop at the same precision: the adaptive one at a standard error of 7.74% of the estimate and the fixed one at 7.79%, against a target of 8%. Neither is buying its saving by finishing earlier in a weaker sense — they finish at the same place, measured the same way, and the adaptive one gets there on twelve fewer runs.

That number is a function of how wrong the guess is rather than of the rule. Redrawing the design at the current estimate is worth nothing when the guess is right — the design being redrawn to is the design already being run — and worth more the further the truth is from the guess. At a guess wrong by a factor of three it is 21%; the robust field measures the same quantity at a fixed run count and finds a twelvefold wrong guess taking an experiment from 34.6% efficient to 90.4%.

So the design half of the adaptation is the profitable half, and it is profitable in proportion to the ignorance it is repairing. That is the pattern the whole of this field’s first half establishes for designs that protect a range, arriving here for a design that reads the data instead — two answers to one complaint, and this one is the cheaper when the guess is bad.

What a random sample size does and does not do #

It is worth separating two things that get confused, because one of them is harmless.

A random sample size is not a problem in itself. The number of observations is a random variable under both rules here, and under the two-stage rule the interval covers exactly. Randomness in N costs nothing when N is independent of what the interval is built from.

A sample size that is a function of the same numbers the interval is built from is a different object, and it is the one every rule that stops on its own estimate produces. The next essay measures what that costs and isolates it with a control; the point here is that the two properties are separable, that the two-stage rule has the first without the second, and that this is the only reason it can keep a promise the sequential rule cannot.

What an experimenter should take from the pair #

Neither rule is the answer, and saying which is better requires knowing something the rules do not.

If the cost of an observation is low and the cost of a wrong interval is high — a simulation study, a measurement that can be repeated cheaply, an interval that will be quoted — the two-stage rule’s theorem is worth its 2.01×, and its overspend can be halved by a larger first stage at no cost to the guarantee.

If observations are the binding constraint — patients, destructive tests, anything where twice the sample size is not available at any price — the sequential rule delivers the requested precision on about the number of observations that precision actually needs, and its interval is four points short of its claim. Four points is not nothing and it is not a catastrophe, and it is a stated quantity rather than a surprise, which is the difference this site keeps insisting on.

What is not defensible is the third option, which is what usually happens: run the sequential rule, report the interval, and describe it as a 95% interval. The number 95 in that sentence has not been counted, the procedure it refers to includes the stopping rule, and the rule is part of the result in exactly the sense the stopping-rule field established for p-values.

Where this sits against the two stopping fields #

The sequential field measures what happens when a hypothesis test is looked at repeatedly: testing five times at the nominal level rejects a true null 14% of the time, and the repair is to spend the error rate across the looks.

The adaptive field measures what happens when a sample size is re-estimated at an interim look and shows that a blinded re-estimation holds its error rate exactly.

This essay is neither. There is no hypothesis and no interim analysis; there is a precision requirement and a rule that stops when it is met. The error rate that goes wrong is not a rejection rate but a coverage, the quantity that leaks is not multiplicity but the correlation between the stopping time and the spread, and the repair that works in the adaptive field — stay blinded to the arms — has nothing to be blinded to here.

What is claimed here, and what is not #

This essay claims fixed-precision stopping: the two rules, the theorem one of them has, what the theorem costs in observations, and the same question inside a design that also chooses where its runs go.

What stays out and is named as a decision: three-stage and accelerated procedures, which sit between the two rules measured here and would need their own accounting; stopping on a relative precision for a normal mean, where the target itself is estimated and the rule can fail to stop; and the sequential rule’s asymptotic theory, which says the shortfall vanishes as d → 0 and is not what an experiment at d = 0.4 with a first stage of five is entitled to.

The checks, and what they are checked against #

Three claims are gated in this field’s library. The two-stage rule is required to cover at least its nominal level at every one of nine combinations of first stage and required width, and to spend more observations than knowing σ would need at every one of them — both halves, because the theorem without the cost is not the trade. The sequential rule is required to fall short at the sharpest of those settings and to do it on fewer observations. And the adaptive design is required to reach the same stated precision on at least ten per cent fewer runs than the design fixed at the guess, with both required to stop at the precision they were asked for, which is what makes the run counts comparable at all.