The rule that cannot see the mean

An experiment that is allowed to decide when it has seen enough has an obvious defect and a known repair, and the pair are worth restating because everything below is an attempt to have the repair without the price.

The defect: a rule that stops when the estimated spread is small enough stops early on the samples whose spread estimate came out low. Its estimate at the stopping moment is 0.83 times the truth, and the interval a practitioner writes at that moment covers 91.7% rather than 95%.

The repair: Stein’s two-stage rule, which estimates the spread once from a first stage, uses that estimate to fix the total, and never looks again. Its coverage is exactly nominal for every σ, because the first stage’s spread and the final mean are independent by construction. It costs 2.07 times the observations knowing σ would have needed.

The obvious question is whether the estimate has to be frozen to be safe, or whether it can go on being updated in a way the interval cannot see. For a normal sample it can, exactly, and the construction is four lines long.

What the interval is allowed to be built from #

Let the observations arrive in blocks of m. Write each block’s mean, and the m − 1 contrasts within that block — the deviations of its observations from their own mean.

For normal data those two sets of numbers are independent. The block means are iid normal with variance σ²/m; the within-block contrasts are orthogonal to every block mean, and orthogonality is independence for jointly normal quantities. That is true block by block, at every b, whatever happens next.

So split the experiment’s two jobs between them:

The rule reads only the contrasts. Pool them into an estimate of σ² on b(m − 1) degrees of freedom, and stop when the number of observations is at least z²σ̂²/d² — the count a fixed-width interval of half-width d would need if that estimate were the truth.

The interval reads only the block means. Their sample mean is the experiment’s estimate; their sample variance estimates σ²/m; and the interval is $\bar{x} \pm t_{b-1}s_B/\sqrt{b}$.

The stopping time is a function of the first set. The interval is a function of the second. Nothing the rule looked at appears in the interval, and nothing the interval is built from could have influenced when the rule stopped.

Two at a time is the smallest a block can be #

One consequence of the arithmetic is worth stating before the theorem, because it is the only constraint the construction has.

A block of one observation has no contrasts in it. The whole estimate of σ² is built from deviations within blocks, so blocks of one give b(m − 1) = 0 degrees of freedom and there is nothing to stop on. The smallest usable block is two, and with two the rule needs at least two blocks before it has an estimate at all — so the first decision cannot be taken before four observations, whatever the requirement is.

That is a real cost at very small samples and it is nearly nothing at any other size. It also explains the shape of everything below: at blocks of two, half the degrees of freedom in the sample go to the rule and half to the interval, and the whole question of how to set the block size is a question about that split.

Why the coverage is exact #

Condition on stopping after b blocks. Because b is a function of the contrasts alone, conditioning on it says nothing about the block means, which are still b independent draws from N(μ, σ²/m). So the quantity

$$\frac{\bar x - \mu}{s_B/\sqrt{b}}$$

is a t on b − 1 degrees of freedom conditionally on b, and therefore unconditionally. Coverage is exactly the nominal level, for every σ, at every block size, under any stopping rule that reads only within-block contrasts.

It is Stein’s theorem with the freezing removed. Stein gets independence by never updating the estimate after the first stage; this gets it by updating the estimate in a direction the mean cannot see. The rule may look as often as it likes and use everything it has seen, and the interval remains exact, which is the property a stopping rule is normally assumed not to be able to have.

Counted at a requirement of d = 0.4 with a first look after five observations, over four thousand runs at each block size: 94.97%, 94.80%, 95.08%, 95.20% and 95.30% at blocks of 2, 3, 5, 8 and 10, against a standard error of 0.34 points. At the gentler requirement of d = 0.25 the same five are 95.17%, 95.00%, 95.25%, 94.55% and 94.85%.

Ten measurements, none of them more than two standard errors from the nominal level, and no parameter anywhere that could have been tuned to put them there.

The same fact, read as a question about information #

It is worth restating the defect in the language the repair is written in, because in that language the repair is obvious rather than clever.

A purely sequential rule stops on σ̂ₙ, computed from every observation it has. The interval afterwards is computed from every observation it has. The two share all their numbers, and the stopping rule is a selection — it selects, out of all the ways the experiment could have gone, the ones where those shared numbers were small in a particular way. Anything computed from them afterwards is computed on a selected sample, which is the rule-is-part-of-the-result argument this site makes about p-values, arriving in an interval.

Written that way, the repair follows: partition the numbers, let the selection act on one part, and report from the other. What makes the normal case special is not the idea but its exactness — that there is a partition under which the two parts are not merely uncorrelated but independent, so the selection carries exactly no information about the reported half.

Blinding the rule is not enough on its own #

The construction has two halves and it is easy to keep only the first, so it is worth measuring what the first half is worth alone.

Run exactly the same rule — stopping on the within-block contrasts, which is the blinded part — and then report the ordinary interval a practitioner would write: $\bar{x} \pm t_{n-1}s/\sqrt{n}$ on all n observations, using every number the experiment collected.

That covers 93.3%.

The rule is blind; the interval is not. The full sample variance contains the within-block contrasts the rule stopped on, so the interval is built partly from the numbers that decided when to stop, and the same shortfall comes back — smaller, because only part of the estimate is contaminated, and unmistakable. Both halves are load-bearing. A rule that reads something the interval does not use is only exact if the interval also refuses to read what the rule used.

And the sharpest version of the same point is the interval built from the stopping estimate itself — the σ̂ in hand at the moment the rule fired, which is the most natural thing in the world to reach for. That interval covers 85.1%. It is short for exactly the reason the purely sequential rule’s is: the rule stops when that estimate is low, so the estimate at stopping is low, so the interval is narrow on precisely the runs where it should not be.

What the numbers look like in one run #

An example makes the split concrete. Take blocks of two, a requirement of d = 0.4, and a true σ of 1, so the experiment needs about 24 observations and does not know it.

The first two blocks give four observations. Each block contributes one contrast — the difference between its two values, halved and squared — so σ̂² arrives on two degrees of freedom and is consequently terrible. The rule compares 4 against z²σ̂²/d² and almost always takes another block.

Each subsequent block adds one degree of freedom to the rule’s estimate and one number to the interval’s. By the time the rule fires at, say, 20 observations, the estimate that stopped it has 10 degrees of freedom and the interval that gets reported has 9 — and the two sets of numbers, ten differences and ten sums, are between them a rotation of the same twenty observations that loses nothing.

The interval is then t₉ × s_B/√10, and the t on nine degrees of freedom is 2.26 where a normal quantile is 1.96. That factor is the whole price, and it is the price of not being allowed to use the other half of the sample twice.

What it spends #

Exactness with no observation cost would be a free lunch, and there is no free lunch here — the price is the width of the interval, which is the subject of the next essay. But the observation count is worth putting down now, because it is where this rule differs most sharply from the two it sits between.

Knowing σ, the experiment would need 24.0 observations. The blinded rule with blocks of two spends 20.3 — fewer than the oracle, because it stops at the first crossing and a first crossing from below usually happens slightly early. The purely sequential rule spends 20.8. Stein’s two-stage rule spends 49.8.

Stein’s rule spends twice the oracle count for the same exactness the blinded rule gets at 0.85 of it. The reason is entirely in that distribution: a sample size fixed from the spread of five observations is a sample size chosen by a very noisy estimate, and because the rule can only ever take more observations than the first stage suggested, the noise costs it in one direction only.

Where the degrees of freedom go #

The block size is the construction’s only dial, and what it dials is how the sample’s degrees of freedom are shared between the two halves. That split can be written down exactly.

With n observations in blocks of m there are b = n/m blocks, so the rule’s estimate of σ² carries b(m − 1) = n(m − 1)/m degrees of freedom and the interval carries b − 1 = n/m − 1. The rule’s share rises towards n as the blocks grow; the interval’s falls like 1/m.

At the twenty observations these requirements typically take, that is stark. Blocks of two give the rule nine or ten degrees of freedom and the interval nine, and the interval’s t multiplier is t₉ = 2.26 against a normal’s 1.96 — a penalty of 1.15. Blocks of five give the interval three degrees of freedom and t₃ = 3.18, a penalty of 1.62. Blocks of ten leave the interval with one, and t₁ = 12.7 — a penalty of 6.5.

So the dial has a side. Going from blocks of two to blocks of ten improves the rule’s estimate by a factor of 1.8 in degrees of freedom and costs the interval a factor of 5.6 in width. The two costs do go opposite ways and they are not remotely the same size, which settles the question the essay leaves open in the direction of the smallest block the construction admits.

That also explains why the coverage table is flat while nothing else is. Coverage is a theorem at every block size — it holds because the two sets of numbers are independent, and independence does not care how many of each there are — so the block size cannot show up there at all. It shows up entirely in the multiplier, and the multiplier is the one quantity in the construction that a reader can compute without running anything.

Blocks of two are the answer, and the reason the question looks open is that the coverage column cannot see it.

Forty-one per cent of Stein’s observations #

The observation counts are worth reading as a ratio to the one construction that shares this rule’s guarantee.

Stein’s two-stage rule spends 49.8 observations for exact coverage; the blinded rule spends 20.3 for the same exactness. The blinded rule reaches the guarantee on 41% of the two-stage rule’s sample, and on 85% of what knowing σ outright would have required.

Stein’s overspend is not a mystery either: with a first stage of five it commits to N = ⌈t²s₀²/d²⌉, and since s₀² is unbiased the expected count is t²(4)/z² times the oracle’s, which is 2.006 — against the counted 49.8/24.0 = 2.075. The closed form for that ratio is the first stage’s own arithmetic, and it says the whole of the two-stage rule’s premium is the four degrees of freedom it froze.

The blinded rule pays no such premium because it never freezes anything: every block adds a degree of freedom to the estimate that decides when to stop, so the estimate at the moment of stopping is built on ten degrees of freedom rather than four. It is the same guarantee bought with a better estimate of σ, and the price it pays instead is the t multiplier of the previous section — 1.15 at blocks of two, against Stein’s 2.075 in observations.

What the construction needs, and what it does not #

The independence used above is a fact about the normal distribution: it is what makes contrasts and means independent rather than merely uncorrelated. So the exactness is exact for normal data and approximate otherwise, in the way everything of this kind is.

What it does not need is worth listing, because the list is longer than expected.

It does not need the stopping rule to be the one described. Any rule that reads only the within-block contrasts gives an exact interval — stop on a target precision, stop on a budget, stop when a colleague says so, provided the colleague has only seen contrasts. The theorem is about the information the rule uses and not about the rule.

It does not need the blocks to be equal, though unequal blocks complicate the variance of the block means and are not measured here.

It does not need the experimenter to know σ, or to have a first-stage estimate, or to commit to anything in advance. The two-stage rule needs a first stage precisely because that is where its independence comes from; here the independence comes from the geometry of the sample, and every observation is used for something.

And it does not need the observations to arrive in blocks in any physical sense. The blocking is an arithmetic device applied to a stream: take them two at a time, or five at a time, and the split into contrasts and means is a rotation of the data rather than a fact about the experiment.

What is claimed here, and what is not #

This essay takes an exactly-covering sequential rule for a fixed-width interval, and the claim is the theorem together with its measurement at ten settings.

What stays out and is named as a decision: non-normal data, where the independence becomes asymptotic and the whole argument becomes an approximation; the two-sample and regression versions, where the contrasts orthogonal to the estimate of interest are a larger space and the same construction should work with more degrees of freedom to spare; and any comparison with the literature’s own sequential fixed-width procedures, of which there are many and none of which are implemented here.

The boundary against the field before this one is the defect. That the sequential rule’s estimate at stopping is 0.83 of the truth, that the shortfall is the dependence rather than the randomness of the sample size, and that the two-stage rule is exact are all established there; this essay takes the diagnosis as given and builds the thing it implies.

The checks, and the refusals that make them mean something #

Two claims are gated in this field’s library, and the first is held to a stricter standard than anything else in this subject: the blinded interval’s coverage is required to be within three standard errors of its nominal level at four block sizes and two requirements — eight separate measurements — because it is a theorem rather than a tendency, and a theorem that is only nearly right in simulation is a theorem with a mistake in it. The second is the comparison: the interval written at the purely sequential rule’s stopping time is required to be below its level at the tighter requirement, so that the exactness above is being measured against something that fails.

The refusal is the interval built from the contrasts the rule stopped on. It is the same numbers, in hand at the same moment, and it takes the coverage from 95.0% to 85.1% — a larger failure than the defect the construction was built to repair. The check throws when it does, because a construction whose whole content is which numbers may be read by which half of the procedure has to be able to detect the half being read twice.