A rule that sets its own κ

A trial stopping on a fixed-width requirement has two estimates of its variance, and it may use only one of them to decide when to stop. The within-block spread is independent of every block mean, so a rule that reads it stays exact; the interval the trial reports, built from the between-block spread, is untouched by the decision. Reading the between-block spread is the offence the schedule that reads the mean was the first example of: the rule then stops when the thing it reports happens to look small.

Yet both estimate the same $\sigma^2$, and an efficiency argument says to pool them. How much of the forbidden spread a rule may read turned that into a dial. With $\hat s^2 = (1-\kappa)\hat\sigma^2_{\text{within}} + \kappa S_B^2$, a share $\kappa = 0$ is honest and $\kappa = 1$ stops on the reported spread. It found that the cost is not proportional to the share. At a tight requirement a rule can read four fifths of the forbidden spread before anything is lost, and the last fifth costs nearly four points.

So every rule on that dial used a fixed blend, while the shape of the curve says that a good blend depends on where the experiment is. Its closing suggestion was to let the rule choose. The pooled estimate an efficiency argument recommends gives each estimate its share of the degrees of freedom: the within-block spread has $N - b$ and the between-block spread $b - 1$, so

$$\kappa = \frac{b-1}{N-1},$$

which on blocks of two is just under a half. That essay’s measurements said this would land in the flat stretch at half-widths of 0.15 and 0.25 and off it at 0.35. It also asked the part the dial never asked: what would a rule that lowers $\kappa$ when $b$ is small cost in observations?

All of this can be run on one set of observations, and one more question goes with it. Reading the forbidden spread has to be worth something, or no one should do it. What exactly does it buy?

What it loses, on the same observations #

Every rule at a requirement runs on the same stream of observations as the honest rule that reads the within-block spread alone. The difference in coverage is therefore paired, and a loss of a third of a point can be told from noise. The figure at the head of this essay is the result on blocks of two, at 6,000 runs per requirement, as points of coverage lost.

The earlier essay’s prediction holds up. At a half-width of 0.15, the rule setting its own $\kappa$ loses −0.18 points, plus or minus 0.19: nothing. At 0.35 it loses 1.18 ± 0.25, nearly five paired standard errors, and at 0.5 it loses 0.37 ± 0.21. At 0.25, which was predicted to be safe, it loses 0.58 ± 0.23, about two and a half standard errors: a small thing, but it is not nothing.

Against the fixed dial the rule is a modest improvement. The fixed half at the same four requirements costs 0.72, 2.43, 0.98 and −0.25, and the reported spread in full costs 0.32, 2.53, 5.33 and 3.75. So a $\kappa$ read off the counts is better than the fixed half at every requirement but the last, where neither loses anything. Yet at the requirement the dial says to worry about, it is no more safe than it was bound to be.

Two features of that plot are worth looking at twice. The loss peaks in the middle of the range, not at either end. And at the loosest requirement, 0.5, the reported spread in full loses less than a fixed half does. Both are because of how few blocks there are. At 0.5 the experiment ends at about five blocks, of which the first two are the opening pair. The between-block estimate reads three or four degrees of freedom, it is enormously noisy, and an interval on so few blocks has a $t$ multiplier so large that a further error in the stopping decision barely registers: the half-width is about 0.65 to 0.77 against a requirement of 0.5. A trial at that requirement overshoots its own target and cannot lose much. At 0.35 the block count is about twelve, and the two forces have met: the interval is no longer rescued by its $t$ point, and $S_B^2$ is still noisy enough to stop an experiment early.

What it buys #

Here is what has been missing from all of this. A rule that reads the within-block spread on blocks of two is reading half of all the information about $\sigma^2$ in the experiment. A pooled reading has twice the degrees of freedom, so its stopping size should be steadier. It is.

At a half-width of 0.15, the honest rule’s stopping size has a standard deviation of 27.0 observations around a mean near 170. The rule setting its own $\kappa$ brings that to 19.0. At 0.25 it is 16.3 to 12.3. At 0.35, 10.3 to 8.7. At 0.5, 4.9 to 4.2. On means of about the same size, that is the same experiment on a shorter leash.

That has real consequences for a trial budget. A planner who must reserve enough capacity for about 95% of runs, roughly the mean plus 1.65 standard deviations, reserves 214 observations at the tightest requirement under the honest rule and 202 under the pooled one, and the mean itself barely moves, 169.0 against 170.4. That is twelve observations of capacity, about 6%. It is useful and not spectacular, and it is what a schedule actually buys was pricing from another direction: a predictable sample size is worth having, and it can be had without changing the sample size it predicts.

Note how unbalanced the two columns are. Reading all of the forbidden spread makes the size less steady, not more: 51.1 against the honest 27.0 at 0.15. That is because the rule then stops whenever the between-block estimate dips, and it has only about 74 blocks’ worth of noisy readings to dip on, with no within-block anchor. The steadiness comes from pooling, not from the forbidden estimate itself. A blend is better than either piece, which is the efficiency argument at work.

Two versions that wait #

The suggestion was a rule that reads more of the forbidden spread when it is safe. The curve in the first figure says safe means a large block count, and two simple versions follow.

Phased in. $\kappa = \frac{b}{b+20} \cdot \frac{b-1}{N-1}$ reads a growing share of the pooled weight as blocks accumulate. It is at half strength at twenty blocks.

Deferred. $\kappa = 0$ until twenty blocks have been read, and the pooled share after that. That is the simplest rule a protocol could state: the same honest stopping rule, plus one number.

Twenty is a round number with nothing tuned about it, and it is a choice this essay has made rather than found. But it was chosen with an eye to the first figure, so the two rules’ losses should be read as a demonstration of the principle and not as a certified setting.

The losses are in the figure at the head. On blocks of two, phased in, they run 0.22, 0.35, 0.23 and −0.10, none more than about two standard errors. Deferred, they run 0.00, −0.02, 0.17 and −0.17. The deferred rule is not distinguishable from the honest rule at any requirement. It cannot be at the loosest: that experiment rarely reaches twenty blocks, so the two rules are the same rule on the same data, and the zero is exact rather than measured.

What the trade picture shows is that nothing has been given up where something was on offer. At 0.15 the deferred rule’s standard deviation is 19.4 against the undeferred 19.0, and at 0.25 it is 14.8 against 12.3: a real part of the steadiness has been given up there, more than half of the undeferred gain, for a loss that has dropped from 0.58 to 0.17 points. At 0.35 and 0.5, where the undeferred rule was steadying the experiment most cheaply in observations and most expensively in coverage, the deferred one simply does not try. That is a fair exchange. It says: wherever the between-block estimate is too noisy to be read safely, it is also too noisy to steady the stopping decision much.

The phased rule is a middle position, at 19.3, 13.4 and 9.4, with losses up to 0.35. It is not obviously better or worse. The deferred one is much simpler to write down and it has the unusual property of being exactly honest over its first stretch, which the phased rule is not.

Blocks of four and of eight #

The self-setting rule behaves differently as the block size grows, because $\kappa$ itself falls. The between-block estimate’s share of the degrees of freedom on blocks of $m$ is about $1/m$.

On blocks of four, $\kappa$ settles near 0.24 and the loss runs 0.13, 0.52, 0.52 and 0.07. On blocks of eight, near 0.12 and 0.22, 0.25, 0.33 and 0.18. At the tightest requirement the steadying is smaller, as it should be with less to pool: 22.0 to 18.9 on blocks of four, 20.4 to 18.9 on blocks of eight. All three come out at about the same stopping standard deviation there, near 19: that is the limit of what the total degrees of freedom can do, and the block size changes only how much of it the honest rule already had.

So one more thing is worth noting about the honest rule. On blocks of eight, it has already 88% within-block. Two degrees of freedom, one total showed that the within-block and between-block counts sum to a constant, and so the bigger the block, the better the honest stopping rule reads and the worse the interval does. That trade has been in this field from the start, and here it reappears as a limit on how much a clever $\kappa$ can buy: a trial that has already chosen large blocks has very little left to gain from reading the forbidden spread at all.

What the fixed-width promise is measured in #

A reader might reasonably ask why the half-width is not the headline. The pooled rule’s mean half-width at 0.25 is 0.257 against the honest rule’s 0.282, closer to the 0.25 the protocol promised, and at 0.35 it is 0.384 against 0.448. On that one number it looks like a clean win, and a report of a trial run this way would say so.

But the half-widths are smaller for the same reason the coverage is lower. The trial stops when the between-block estimate is low, and it is the between-block estimate the half-width is built from. The rule is not estimating a shorter interval, it is selecting for one. This is why the same setup could be read as success at one level of the report and as failure at another, and it is the failure the honest rule was built to avoid: an interval’s reported width is not evidence of its coverage once the width helped to decide when to stop. The sums under a moving level met the other side of that, an interval that gets wider from the same kind of borrowing and says nothing either about its own coverage.

Why the efficiency argument is the wrong kind of argument #

It is worth saying why a rule built from the textbook’s best estimate of $\sigma^2$ can still go wrong. The degrees-of-freedom weights are optimal for estimating the variance at a fixed time. That is a statement about the variance estimate’s own sampling error, and at a fixed sample size it is correct. The stopping rule is not using the estimate at a fixed time. It reads it repeatedly and stops the first time it is small enough, and the event “the estimate is small enough now” selects for low values. In the within-block estimate those low values do not matter, because they are independent of the interval. In the between-block one they do, because they are the interval. A stopped experiment’s $S_B^2$ is low for the reason it stopped, and it is the number that goes into the reported half-width.

So the weights were derived for a problem with no stopping and no independence requirement. The right weight for this problem has a term the efficiency calculation does not know about, which is how much the estimate’s dip costs at the moment of stopping. That is small if the experiment has many looks at a slowly changing estimate, and it is large in the middle of the range, where a single block can still move $S_B^2$ across the threshold.

That is an account and not a proof. It does predict the block-count dependence: a loss that peaks where blocks are about ten or twelve, and disappears once they number in the dozens. That is what the first figure shows.

How the numbers were made #

The rule at each look reads $(1-\kappa)\hat\sigma^2_{\text{within}} + \kappa S_B^2$, with $\kappa$ a function of the current $b$ and $N$, and stops when the observations in hand reach $z^2 \hat s^2 / d^2$, after a start of two blocks of five. It takes a further block of $m$ otherwise. The interval reported is $t_{b-1} \sqrt{S_B^2 / N}$ in every case, built on the between-block estimate. Six thousand runs at each of four requirements and three block sizes run every rule on the same observations, and all the coverage losses are paired. The standard error of an unpaired coverage near 95% at this many runs is 0.28 points.

Three things have to hold. On blocks of two the rule setting its own $\kappa$ must lose more than three paired standard errors at 0.35 and less than two at 0.15, and it must steady the experiment by more than a fifth at 0.15. The deferred rule must lose less than two and a half standard errors at every requirement. And the claim that the rule setting its own $\kappa$ is safe at every requirement, at two paired standard errors, must be refused, as it is at 0.35 and 0.25.

Twenty is the only tuned number here. A rule with a threshold of ten has not been measured and should not be assumed to behave the same way.

Still open: the threshold, and a rule that watches its own estimate #

Where the deferral should start. A threshold of twenty blocks was chosen by eye from a curve that peaks near twelve. The natural study sweeps it at each requirement, at higher precision than these runs, and asks for the lowest threshold at which the paired loss stays below some tolerance, since every block below it is steadiness given up for nothing. Whether that threshold depends on the block size or only on the block count is the part of the answer a protocol could use.

A rule that compares the two estimates. The offence is reading $S_B^2$ when it has dipped. A rule could read it only when it agrees with the within-block estimate to within what the block count permits, and fall back to the within-block one otherwise. That is a pretest inside a stopping rule, and it could go either way: it removes exactly the dips that do the damage, and it also reads the forbidden number in order to decide whether to read it. Whether what is left is exact, or nearly so, is measurable from these runs with one more branch.

A law that is not normal. Independence of the within-block spread and the block means depends on normality, as it did for the three-stage rule and for the stop the rule chose. Under a skewed law even the honest rule is not exact, and the difference between honest and pooled might narrow. Nothing here has been run off the normal.

What the dial now says #

This essay takes a stopping spread blending the within-block and between-block estimates at a share set by the degrees of freedom. Its claims are that on blocks of two this rule loses −0.18 ± 0.19 points at a half-width of 0.15 and 1.18 ± 0.25 at 0.35; that it steadies the stopping size from a standard deviation of 27.0 to 19.0 at 0.15; that a rule deferring the reading until twenty blocks has no measurable loss at any requirement and keeps most of the steadiness at the tightest; and that on larger blocks the share falls and the loss with it.

What stays out, by decision: non-normal observations; a stopping rule on a precision other than a fixed width; and any certification of twenty as a threshold. The boundary against the essay on the dial is that it fixes $\kappa$ and measures the loss, and this one lets $\kappa$ be chosen and prices the other side of the trade, which is the steadiness of the stopping size that an offending rule was reading the spread to get.