How much of the forbidden spread a rule may read

Every rule in this field rests on one restriction. Observations arrive in blocks; the rule decides when to stop from the within-block contrasts, which say how noisy the observations are; and the interval it reports is built from the block means, which say where the mean is and how much it varies from block to block. The two are independent for normal observations whatever the rule does with the first, so the interval is exact. The essay that measured the boundary showed what happens to three rules that cross it, and the worst of them — stopping when the interval about to be reported is narrow enough — lost eight points of coverage while appearing to save observations.

That essay treated the restriction as a line: a rule reads the forbidden quantity or it does not. It named, and left, the obvious next question — the size of the violation as a function of how strongly the rule reads the means, which is a dial nobody turned. This essay turns it. The answer is not the one a line would suggest, and it changes what a protocol should worry about.

A dial between the allowed spread and the forbidden one #

The honest rule stops when the observations in hand reach $z^2\hat s^2/d^2$, the number a fixed-width interval of half-width d would need if ŝ² were the truth, with ŝ² the pooled within-block estimate. The offending rule uses the between-block estimate $S_B^2$ instead — the very quantity the reported interval divides by. Between them is a family:

$$\hat s^2 \;=\; (1 - \kappa)\,\hat\sigma^2_{\text{within}} \;+\; \kappa\,S_B^2$$

At κ = 0 the rule is honest and exact. At κ = 1 it stops on the reported spread. In between it reads some of each, and because both estimate the same $\sigma^2$, every rule in the family is a perfectly sensible-looking way to decide whether there is enough data. A practitioner who pools all the information about $\sigma^2$ available at each look, which is what an efficiency argument recommends, is somewhere on this dial.

Everything here uses blocks of two, as the field’s earlier rule did, and runs every κ on the same observations, so a difference in coverage between two blends is measured on identical data and its standard error is the paired one. That is what lets a loss of a fraction of a point be told from noise.

The loss is not proportional to the reading #

At a half-width of 0.25, where knowing σ would need 61.5 observations, the honest rule covers 94.65% over six thousand runs. Reading a fifth of the forbidden spread loses 0.20 points, with a paired standard error of 0.18 — nothing measurable. Reading half of it loses 0.98 ± 0.26. Reading two thirds loses 1.52. Then the curve turns: 3.18 at κ = 0.8, 5.10 at 0.9 and 5.50 at full reading. The first half of the dial costs one point; the second half costs four and a half.

At a half-width of 0.15, where the experiment needs about 171 observations, the curve is flat for longer. Reading up to two thirds of the forbidden spread costs nothing distinguishable from zero — the losses run between −0.28 and 0.08 points — and κ = 0.8 costs 0.25 ± 0.24. At 0.9 the loss is 0.78 and at full reading 3.73.

At a half-width of 0.35, where about 31 observations are needed, there is no flat stretch. A tenth of the forbidden spread already costs 0.30 ± 0.15 points and a fifth 0.87. The loss peaks at 3.97 at κ = 0.8 and falls back to 2.70 at full reading, for a reason the next sections give.

So the answer to how much may a rule read depends on the size of the experiment, and in the direction that matters. The longer the experiment, the more of the forbidden quantity a rule can lean on before it pays.

Why the length of the experiment buys tolerance #

The mechanism is the one the field already knows, measured one level down. A rule that stops when a blend including $S_B^2$ is small enough stops preferentially when $S_B^2$ happens to be small, and the reported interval is then built from an $S_B^2$ that is biased low.

Read honestly, $S_B^2$ at stopping averages 0.9818, 0.9941 and 0.9954 of $\sigma^2$ at the three requirements — on its target to within the noise. Read in full it averages 0.652 at the loosest, 0.720 at the middle and 0.890 at the tightest. The ordering is the ordering of the coverage losses at small κ, and it is the ordering of how many blocks each experiment has: about 12, 27 and 81 under the honest rule. An estimate built from eighty blocks barely moves from one look to the next, so a rule that waits for it to be small has little to wait for; the same estimate on twelve blocks fluctuates enough that waiting for a low value finds one.

The convex shape is visible here too, and far more precisely. At a half-width of 0.25 the estimate falls from 0.9941 to 0.9361 of $\sigma^2$ over the first half of the dial and from there to 0.7197 over the second. A mean of an estimate is resolved to a few thousandths in six thousand runs where a coverage rate is resolved to a quarter of a point, so this is the diagnostic to run on a proposed rule: simulate it, and look at where $S_B^2$ lands when it stops.

That is also why the cost is convex in κ rather than proportional. With most of the weight on the within-block estimate, the blend’s fluctuations are mostly fluctuations in the allowed quantity, and the rule stops when that is small, which costs nothing. The forbidden part has to carry most of the blend’s movement before the rule’s stopping time is driven by it, and when the experiment is long it moves least — so the share at which it takes over moves right as the requirement tightens.

What the offending rule looks like from outside #

A violation that showed itself in the output would be easy to catch. This one does the opposite.

At a half-width of 0.25 the honest rule stops after 59.18 observations and reports an interval whose half-width averages 0.2815. As κ rises the observations barely change until κ = 0.65, and the reported interval gets steadily narrower: 0.2557 at κ = 0.5 and 0.2546 at 0.65, about a tenth below the honest rule. Only past 0.8 does the rule start stopping visibly early — 46.80 observations at full reading — and by then the interval has stopped narrowing, because fewer observations mean fewer blocks and a larger t quantile on fewer degrees of freedom.

So across the middle of the dial an experimenter comparing two rules on their output sees the offending rule produce narrower intervals from the same number of observations, which is exactly what a better rule would produce. The coverage loss is the only symptom, and at κ = 0.5 it is one point against a counting error of a quarter of a point in six thousand runs — invisible in any single trial and in most simulations anyone would run to check a protocol.

The non-monotone curve at the loosest requirement comes from the same t quantile. At a half-width of 0.35 the rule reading the forbidden spread in full stops after 23.83 observations in 8.92 blocks, against 29.36 in 11.68 at κ = 0.5; the interval’s t quantile on seven or eight degrees of freedom is wide enough to win back some of what the biased spread lost, and the reported half-width rises from 0.3749 to 0.4302. The loss falls from its peak not because the rule reads less but because it stops so early that the interval’s own correction for few degrees of freedom partly repairs it.

A schedule turned the same way moves in a line #

The field’s other way of reading the means is through the block size rather than the stopping time: a schedule that takes a larger block when the between-block spread is running high. The earlier essay found it over-covers by about a point. The same dial applies — a share κ of the time the schedule obeys the spread, and otherwise it tosses a coin between the large and small block, which reads nothing and is exact.

The schedule’s coverage changes by −0.12, +0.40, +0.42 and +1.00 points at κ = 0.25, 0.5, 0.75 and 1, a straight line through zero with a slope of 1.01 points per unit of κ and no departure from it larger than 0.20 points. Every bit of reading costs the same small amount, in the over-covering direction.

The two shapes differ because the two rules decide different things. A schedule chooses how much more data to collect before the next look, and a large block taken because the spread was high adds observations that then dilute the high spread — a mild, proportional feedback. A stopping rule chooses whether to collect any more at all, and stopping is the one decision after which nothing can be diluted. The independence the construction needs is broken by both; only the second breaks it in a way that compounds.

The field’s earlier offenders, placed on the dial #

The dial gives a coordinate to rules the field has already measured, and placing them on it explains why they cost what they cost.

The rule the earlier essay found most expensive stops when the interval it is about to report — t on b − 1 degrees of freedom times $\sqrt{S_B^2/N}$ — is narrower than d. That reads the forbidden spread and nothing else, so it sits at κ = 1, and then goes past it: it also multiplies by the t quantile, which is large exactly when the number of blocks is small, so it waits longest when $S_B^2$ is least reliable and most likely to drift low. At a half-width of 0.25 it covered 86.87% against the honest rule’s 94.87% over fifteen hundred runs; the blend at full reading, which uses z and stops on the observation count, covers 89.15% against 94.65% over six thousand. The two are the same violation, and the rule that reads the reported interval directly is the more aggressive form of it.

The purely sequential rule of the guarantee field is the same violation without blocks. It reads the sample spread of all the observations and reports an interval built from that same spread, so in this field’s terms there is no within-block estimate to blend with: it is κ = 1 by construction, and its coverage of about 90% at a short first stage is of a piece with the losses here at the loosest requirement.

And the two schedules the earlier essay measured — one that shrinks the block when the between-block spread is high, one that grows it — sit on the schedule’s line rather than the stopping rule’s curve. They cost a point or so in either direction because a schedule’s violation is proportional and small. The stopping rule is the only one of the three that has a ledge, and it is the one a practitioner is most likely to write, because “stop when the interval is narrow enough” sounds like the definition of a fixed-width procedure rather than a departure from one.

What blocking buys, seen from the dial, is the existence of an allowed spread at all. The blinded construction and the rule that stops on the arms both rest on there being a within-block quantity that says how noisy the data are without saying anything about where the mean is. Without it there is nothing to put at κ = 0.

What a protocol should take from the dial #

Three things, and the first is the opposite of the usual advice.

A little reading of the forbidden spread is usually harmless, and how little depends on the experiment. At a requirement needing a hundred and seventy observations, a stopping rule that pools within- and between-block information with up to two thirds of the weight on the forbidden half loses nothing measurable. At one needing thirty observations, a tenth of the weight already costs a third of a point. The dial says the restriction is strict only for short experiments — which are also the ones where the honest rule’s own costs are highest, so the temptation to pool is greatest exactly where pooling is least safe.

The violation is invisible in the output across the middle of the dial. The offending rule produces narrower intervals on the same observations. A protocol cannot rely on anything in its results to reveal the problem; it can only rely on how the rule was written, which is the earlier essay’s conclusion about stating a schedule as a function, now with a number attached to how much the function may include.

A schedule and a stopping rule are not the same risk. A block-size schedule that reads the spread over-covers by about a point at most, and proportionally. A stopping rule that reads it can lose five, and the loss arrives late on the dial. If anything has to read the forbidden quantity, it should be the schedule.

The boundary with the rule that cannot see the mean is that a blinded rule reads nothing forbidden by construction, so its κ is zero whatever else it does; this essay is about how far an unblinded rule may drift from zero. The boundary with what the blindfold costs is the direction: that essay prices the honest rule’s restriction, and this one prices relaxing it.

Still open: a rule that sets its own κ #

Every rule here uses a fixed blend. The dial suggests a rule that reads more of the forbidden spread when it is safe — late in a long experiment, when $S_B^2$ rests on many blocks — and none when it is not. The natural candidate weights the two estimates by their degrees of freedom: the within-block estimate has N − b and the between-block one b − 1, so the pooled estimate of $\sigma^2$ that an efficiency argument recommends puts κ = (b − 1)/(N − 1) on $S_B^2$, which with blocks of two is just under a half and changes little as the experiment grows. Whether that particular κ lands in the flat stretch at every requirement — the measurements here say it would at a half-width of 0.15 and 0.25 and would not at 0.35 — and what a rule that lowers κ when b is small would cost in observations, is the measurement that would say whether the pooled estimate is ever safe to stop on.

What is claimed here, and what is not #

This essay takes the size of the violation as a function of how much of the forbidden spread a stopping rule reads. The claims are that over six thousand paired runs at three requirements, a rule whose stopping spread blends a share κ of the between-block estimate into the within-block one loses coverage convexly in κ; that the share it can read before any measurable loss grows as the requirement tightens — the loss is measurable from a tenth at a half-width of 0.35, from about a third at 0.25 and only from nine tenths at 0.15; that the between-block estimate at stopping is pulled down to 0.652, 0.720 and 0.890 of $\sigma^2$ at full reading, in the same order; that across the middle of the dial the offending rule reports narrower intervals on the same observations; and that a block-size schedule turned the same way changes coverage linearly and by at most a point.

What stays out, named as a decision: blocks larger than two, where the within-block estimate has more degrees of freedom and the dial would shift; two arms, where the forbidden quantity is a difference and the blend would be between three estimates; and any repair — a rerandomisation over the rule’s own reference set would restore a level at any κ and is a different construction with a different cost.

The checks, and the refusals that make them mean something #

The dial figure checks at every requirement that full reading costs more than four paired standard errors and that the onset moves right as the requirement tightens. The mechanism figure checks that the between-block estimate at stopping falls by more than five hundredths of $\sigma^2$ from honest to full reading at all three. The schedule figure checks that the schedule’s coverage stays within three standard errors of a straight line. The refusal is the honest end of the dial read as if it were the dishonest one: κ = 0 claimed to lose a point of coverage at a half-width of 0.25, which the checks must reject, because a check that passed it would be measuring the noise between runs rather than the reading.