A stopping rule that reads the periods

A fixed-width trial makes a promise about a difference of two arms: it will keep recruiting until its 95% interval for the difference is no wider than an agreed half-width, here 0.3. It keeps that promise by estimating, after each block, how precise the difference already is, and stopping when the estimate says enough. The rule that cannot see the mean found the condition under which such a rule’s interval is still exact afterwards: the stopping time must depend only on quantities independent of the difference itself, and within-arm spreads are such quantities, because they contain no mean.

The corner is the design where that rule is hardest to keep. Weights that need only a ratio is the field’s theorem for it: when the two arms have different variances and the allocation between them changes from block to block, a block’s difference is worth $h_b(\lambda) = (1/m_{bA} + \lambda/m_{bB})^{-1}$, with $\lambda$ the ratio of arm B’s variance to arm A’s, and weights proportional to those make the interval exactly $t$. A ratio that changes with the period then found what happens when the corner’s own alternating allocation makes two kinds of period whose ratios differ — small A-heavy blocks and large B-heavy ones. One ratio pooled across the trial is set by the large blocks, its weights are miscalibrated, and the interval covers 100% in one alignment and 85% in the other. A ratio estimated per kind of period repairs the interval.

It did not repair the stopping rule, and the essay said so. The trial still decided when to stop from pooled spreads, and so spent observations against a precision it was mismeasuring: at an eightfold difference between the periods, in one alignment, about three and a half times what its promise needed. A rule that summed per-period precisions would stop when the per-period interval reached its promised width. Whether it would remain exact was the question, and the argument for it is the blinded rule’s: a stopping time that depends only on within-arm spreads, now per period, should still be independent of the block differences. This essay runs that rule.

Two stopping rules on the same trials #

The setting is the corner exactly as the field built it: arm A at a standard deviation of 1 and arm B at 5, block sizes that alternate between six and forty-eight, an allocation that alternates with them so the small blocks are A-heavy and the large ones B-heavy, and a promised half-width of 0.3. A variance-ratio difference between the periods is introduced by multiplying arm B’s standard deviation by $\sqrt{F}$ in one kind of period and $1/\sqrt{F}$ in the other, at $F$ = 2, 4 and 8, in both alignments — the small blocks noisier, or the large ones.

The pooled stopping rule is the field’s: after each block, a block’s precision is read from the within-arm spreads pooled over every block so far, the precisions are summed, and the trial stops when the sum reaches $z^2/d^2$, with $z$ the normal 97.5% point. The per-period rule reads each block’s precision from the within-arm spreads of the blocks of its own kind, and otherwise does the same. Neither looks at any block’s difference. Each is run on three thousand trials per configuration, and at the end every trial reports the interval weighted by each kind of period’s estimated ratio.

With one ratio the two rules are the same rule and spend the same: 1,656 observations and 1,649. Once the periods differ they part, and in the alignment where the large blocks are the noisier ones they part a long way. At a twofold difference the pooled rule takes 2,976 observations and the per-period rule 2,614; at fourfold, 5,642 and 3,368; at eightfold, 10,989 and 3,019. The pooled rule’s spending grows almost in proportion to the difference, and the per-period rule’s barely moves. With the small blocks the noisier the pooled rule over-spends too, by less: 1,020 against 1,015 at twofold, 751 against 669 at fourfold and 711 against 490 at eightfold.

The two alignments over-spend for two different reasons, and the allocation explains both. A small block of six puts four units in arm A and two in arm B; a large block of forty-eight puts eight in A and forty in B. So the large blocks hold nearly all of arm B’s degrees of freedom — thirty-nine a block against one — and nearly all of the trial’s precision.

When the large blocks are the noisy ones, the pooled arm-B spread is theirs, and the small blocks, whose arm B is quiet, are credited with a fiftieth of the precision they carry. The trial keeps recruiting to make up a shortfall that is not there. When the small blocks are the noisy ones, the leak runs the other way: their arm-B spread, two hundred at an eightfold difference against three in the large blocks, enters the pooled estimate on one degree of freedom a block and lifts it to about eight — and the large blocks, which carry the precision, are credited with about three fifths of what they have. The second leak is diluted by the large blocks’ own degrees of freedom, which is why it costs less.

Still exact #

The question the lead asked was whether the cheaper rule keeps the interval honest. A stopping time chosen by looking at the data can bend an interval’s coverage; this one is chosen by looking at per-period spreads, which are more numerous and noisier than pooled ones, and which the per-period interval itself uses for its weights.

It does. Under the per-period stopping rule the per-period interval covers between 94.4% and 95.2% at all seven configurations; under the pooled rule, between 94.3% and 95.1%. With three thousand trials a configuration the standard error on a coverage is 0.4 points, and every one of the fourteen numbers is within two and a half of them of 95%. Making the stopping time read the same per-period spreads the weights read does not make the two dependent on the block differences, because neither reads a difference. That is the blinded rule’s argument, unchanged, applied one level down, and the measurement agrees with it.

The cost of exactness was nowhere: the per-period interval is exact under either stopping rule, as it was in the earlier essay. What the stopping rule changes is not whether the interval is right but how many observations were spent to make it the width it is.

The promise, kept #

Under the pooled rule the interval’s width is wherever the pooled rule’s mismeasurement left it. At an eightfold difference with the large blocks noisier it is 0.53 of the promise — an interval about half as wide as anyone asked for, bought with three and a half times the observations. At fourfold in the same alignment it is 0.78. With the small blocks noisier it is 0.91 at eightfold, and at twofold it is slightly above the promise. A fixed-width rule whose delivered width depends on a nuisance nobody is told about is not keeping a fixed-width promise.

Under the per-period rule the delivered width is between 1.02 and 1.11 of the promise at every configuration. It keeps the promise to within a few per cent everywhere, and the one thing it misses by is systematic: it is slightly wider than promised, never narrower, and widest at the configurations where the trial ran fewest blocks.

Where the remaining few per cent come from #

The stopping rule reaches its target at the normal point, $z^2/d^2$; the interval it then reports uses Student’s $t$ on one fewer degree of freedom than there are blocks, because its scale is estimated from the blocks’ scatter. With 18 blocks — the small-blocks-noisier configuration at eightfold, where the per-period rule stops earliest — $t_{17}$ is 7.6% above 1.96, and the interval is 11.0% above the promise. With 112 blocks the ratio is 1.1% and the excess 2.0%. Across the seven configurations the excess sits between one and four hundredths above the ratio, so Student’s correction carries most of it, and what is left is largest where the blocks are fewest.

The repair is obvious and has a cost. A rule that stopped at $t_{b-1}^2/d^2$ rather than $z^2/d^2$ would aim at the interval it will report, but its target would move with the block count, which is itself a function of when it stops. The fixed- width literature’s answer is to inflate the target by the expected ratio; this collection’s measurement says the inflation needed is a few per cent at the block counts this design reaches, and that a rule that ignores it promises a width it misses by exactly that.

What the rule is allowed to read #

The per-period rule reads more of the data than the pooled one, and in a sequential design reading more is where an interval’s honesty is usually lost, so it is worth being exact about what it reads. After each block it computes, for each kind of period, the pooled within-arm sums of squares of arm A and of arm B over the blocks of that kind: four numbers per kind, eight in all. It never reads a block’s mean, a block’s difference, or any arm’s total. Blinded and still exact established why that matters: under normal outcomes every within-arm sum of squares is independent of every arm mean, so any stopping time built from sums of squares alone is independent of the differences the interval is built from, and conditioning on when the trial stopped tells the interval nothing about where the difference lies. Splitting the sums of squares by period does not change what kind of quantity they are.

That is also why the rule can be run by somebody who is blind to the treatment effect. A ratio of within-arm spreads is a contrast of variances, with no mean in it; what a two-arm rule may not pool drew the line at pooling across the arm label, because a spread computed without the arm label contains the difference. The per-period rule keeps the arm label and adds a period label, and the period label — block parity — is a fact about the design, known before the first unit is randomised. A data monitoring committee could run it without seeing anything a blinded committee is not allowed to see.

What the measurement adds to the argument is that it holds at the sample sizes the rule actually stops at. The per-period sums of squares in the small blocks rest on one arm-B degree of freedom a block, and at the eightfold small-noisier configuration the trial stops after about eighteen blocks, nine of them small: the small blocks’ arm-B ratio is estimated on nine degrees of freedom. A stopping time that noisy could have bent the coverage through the weights even if not through the means, and it does not — 95.2% at that configuration.

What the pooled weights still do #

Changing the stopping rule does not rescue weights that are wrong for the periods. The interval weighted by one pooled ratio covers 99.5% to 100% with the small blocks noisier and 84.2% to 88.7% with the large blocks noisier under either stopping rule — the earlier essay’s finding, untouched by when the trial stops. That is worth stating because a reader might expect the better stopping rule to help the simpler interval. It cannot: the stopping rule decides how much data there is, and the pooled weights misread all of it the same way. The per-period ratio is needed twice — once to stop, once to weigh — and the two uses are separate repairs.

What the trial reports and what it spends #

The practical reading follows directly. A corner trial whose allocation alternates between two kinds of block should estimate a variance ratio per kind of block and use it in both places. The stopping rule then spends what the per-period interval actually needs, which at an eightfold difference in the worse alignment is under a third of what the pooled rule spends; the interval it reports stays exact; and its width misses the promise only by Student’s correction. None of this needs the analyst to know that the periods differ. The per-period ratio is estimated in every trial, and when the periods agree the two rules coincide — 1,656 and 1,649 observations, the same interval, the same coverage — so reading the periods costs nothing when there is nothing to read.

The one assumption the rule does need is the period structure itself: that the blocks come in two kinds, known in advance, by parity. Here they do by construction, because the allocation alternates. A trial whose periods were not known in advance would have to find them, and a search for where the variance changes is a search, with a charge of its own.

How the numbers were made #

Three thousand trials at each of seven configurations under each stopping rule, each trial running the corner design until its rule stops: blocks alternating between six and forty-eight units, the allocation alternating between A-heavy and B-heavy, arm B’s standard deviation multiplied by $\sqrt{F}$ or $1/\sqrt{F}$ in alternate blocks. The per-period rule reads a block’s precision from the pooled within-arm spreads of the blocks of its own parity, falling back to the trial’s pooled spreads until a kind of block has two degrees of freedom in each arm. At the end, each kind of period’s ratio is the ratio of its pooled within-arm variances, the interval weighted by $h_b(\hat\lambda_t)$ is formed with Student’s $t$ on one fewer degree of freedom than blocks, and coverage of the true difference, zero, is counted.

Checked on the measurement: the per-period interval is within two and a half standard errors of 95% under both stopping rules at every configuration; at eightfold with the large blocks noisier the per-period rule stops on under a third of the pooled rule’s observations; and the per-period rule’s width excess is at least Student’s ratio and within six hundredths of it everywhere.

Still open: a ratio that drifts inside a period #

The earlier essay’s second open question is untouched here. Between a ratio that wanders at random from block to block, which costs width, and one that differs by kind of period, which costs coverage until it is estimated per period, there is a ratio that drifts gradually through the trial — partly aligned with the design and partly not. A per-period estimate then averages a moving quantity over the whole trial, and a stopping rule reading it lags the drift. How much coverage and how much spending that costs, and whether a ratio estimated over a moving window of recent blocks repairs it without breaking the independence the exactness rests on, is a measurement this design can make.

What is claimed here, and what is not #

This essay takes a stopping rule that reads each kind of period’s own within-arm spreads, in the corner design. The claims are that with one ratio it spends what the pooled rule spends, 1,649 observations against 1,656; that with the large blocks noisier it spends 2,614, 3,368 and 3,019 observations at two-, four- and eightfold differences where the pooled rule spends 2,976, 5,642 and 10,989; that the per-period interval covers between 94.4% and 95.2% under it at every configuration, as it does under the pooled rule; that it delivers between 1.02 and 1.11 of its promised width where the pooled rule delivers between 0.53 and 1.04; that the excess is at least Student’s correction on the blocks it ran and within four hundredths of it; and that the pooled-ratio interval’s miscoverage is unchanged by the stopping rule.

What stays out, named as a decision: non-normal outcomes, on which the width rule on skewed outcomes found the blinded rule’s independence failing for a separate reason; periods not known in advance; and a drifting ratio, deferred above.

The boundary against a ratio that changes with the period is that it repaired the interval’s weights and left the stopping rule pooled; this essay repairs the stopping rule and measures what the two repairs do together. Against the rule that cannot see the mean, the independence argument is the same and is here tested on a stopping time with two variance estimates in it instead of one.