A ratio that changes with the period

Weights that need only a ratio found the exact interval for a difference of two arms in the corner where neither earlier condition holds — the arms have different variances, and the allocation between them changes from block to block. Every block’s difference has variance $\sigma_A^2/h_b(\lambda)$ with $h_b(\lambda) = (1/m_{bA} + \lambda/m_{bB})^{-1}$, so weighting each block by $h_b(\lambda)$ is weighting by its inverse variance, and the interval is exactly $t$. The one unknown is $\lambda = \sigma_B^2/\sigma_A^2$, and the next essay showed that a rule forbidden to see the effect may estimate it, on hundreds of within-arm degrees of freedom.

Every step of that rests on there being one $\lambda$. The derivation needs it, the estimate pools across the whole trial because of it, and the refusal that closed the field — a ratio estimated inside each block covers 83% — is the argument for pooling it. What none of the three asked is whether the corner’s own design gives the ratio a reason not to be one number.

It does. The corner alternates between two kinds of block: small ones, six units with four in arm A and two in arm B, and large ones, forty-eight units with forty in arm B. A trial that runs that way has two kinds of period, and anything that differs between periods — a run-in measured on different equipment, a season, a second site that joins for the B-heavy phase — can make arm B noisier in one kind than the other. This essay measures what one pooled ratio does then, and what replaces it.

What one pooled ratio does to the interval #

The calibration factor that the field derived answers the question before anything is simulated. For any fixed weights $a$ and any block variances $V$, the interval’s own estimate of its scale is off by

$$\frac{(\textstyle\sum a)(\sum aV)/(\sum a^2V) - 1}{b - 1},$$

which is 1 when the weights are the inverse variances and when they are all equal, and nothing else. With the ratio $25F$ in one kind of period and $25/F$ in the other, the variances are $1/m_{bA} + \lambda_t/m_{bB}$, and the weights built from one pooled ratio are neither of the two things that make the factor 1.

It moves fast, and which way it moves depends on which kind of block is the noisy one. With the small A-heavy blocks noisier, the factor is 2.07 at a twofold difference, 3.91 at fourfold and 5.44 at eightfold: the interval’s scale is overstated several times and it will over-cover. With the large B-heavy blocks noisier, it is 0.65, 0.55 and 0.53: the scale is understated by nearly half and the interval will under-cover.

The asymmetry is the pooling. A ratio pooled across the trial is the ratio of arm B’s within-arm sum of squares to arm A’s, and arm B’s degrees of freedom are almost all in the large blocks — thirty-nine a block against one in a small block. So the pooled ratio is the large blocks’ ratio, whatever the small blocks are doing. When the small blocks are the noisy ones, the pooled ratio is $25/F$, far too low for them; it gives each small block many times the weight its variance deserves, and those blocks then scatter the block differences far more than they move the weighted estimate. The interval’s spread reads that scatter, and overstates. When the large blocks are the noisy ones the pooled ratio is right for them and too high for the small blocks, which are then under-weighted, and the error runs the other way.

A ratio per period puts each block’s weight back at its own inverse variance, and its factor is exactly one at every separation. That is not a measurement; it is the theorem applied to each period separately.

Counted on the same trials #

The closed form is about weights at fixed compositions. A real trial has a stopping rule, a random number of blocks and a ratio that is estimated rather than known, so the claim has to be counted.

Each configuration is three thousand trials of the corner design, run by its own fixed-width stopping rule, with the same block differences read through five weightings: one pooled ratio, the rule this field derived; a ratio per period, estimated from the within-arm sums of every block of that kind, with both arms’ spreads estimated per period because an analyst is not told which arm drifts; each period’s true ratio, as a reference; one weight per block; and the effective size.

With one ratio throughout, the pooled and per-period weights both cover 94.3%, within two standard errors of 95%, and the per-period estimate’s extra freedom costs nothing visible. At a fourfold difference with the small blocks noisier the pooled ratio covers 100.0%; with the large blocks noisier it covers 85.1%, and 84.4% at eightfold. The ratio per period covers 95.0% and 94.9% at fourfold, and between 94.3% and 95.1% at every configuration measured.

Two of the other columns are worth reading. One weight per block stays at its level everywhere — 95.2% at one ratio, between 94.8% and 96.0% elsewhere — because equal weights are calibrated whatever the variances are, which is the second root of the calibration factor and the reason flat weights have always been the safe fallback. The effective size, which is $h_b(1)$ and assumes the arms share a variance, over-covers at one ratio and fails in both directions at fourfold: 99.9% and 89.1%.

So the period structure puts the corner’s exact weights back where the effective size was before the corner was solved: calibrated only by luck, with the direction of the error set by details of the design nobody chose for that reason.

Precise, and about the wrong blocks #

The pooled estimate is not a poor estimate. That is what makes the failure easy to miss: every diagnostic a trial would run on it says it is fine.

At a fourfold difference with the small blocks noisier, the true ratios are 100 in the small blocks and 6.25 in the large ones, and the pooled estimate averages 8.68 across the three thousand trials. With the large blocks noisier the truths are 6.25 and 100 and it averages 97.80. In both cases it sits within a few per cent of the large blocks’ ratio and nowhere near the small blocks’, and in both cases it rests on several hundred degrees of freedom and would come with a narrow interval of its own.

So the pooled ratio answers a well-posed question very precisely — what is arm B’s variance relative to arm A’s, averaged over the trial’s within-arm information — and the weights need the answer to a different one, which is that ratio in each block. With one kind of block the two questions have one answer. With two kinds they have two, and the trial’s information is distributed so unevenly between the kinds that the average is simply the larger kind’s value.

The same unevenness is why the problem is invisible from the interval. Its width and its stated degrees of freedom look like any other run’s; only its coverage is wrong, and coverage is the one property a single trial cannot observe. What a single trial can observe is the two per-period estimates side by side, and a fourfold difference between two ratios each estimated on dozens to hundreds of degrees of freedom is not subtle. The diagnostic is the repair: compute the ratio per period, and if the two agree nothing has been lost.

A ratio per period, and what it costs #

The repair is the same move the field already made, one level down. A period’s ratio is a contrast of within-arm spreads, so it contains no mean, and a rule blind to the effect may compute it — the argument of the blinded construction applies to each period unchanged. And each period’s ratio is estimated on the degrees of freedom of every block of its kind, not on one block’s, which is the difference between this and the refused per-block estimate.

The cost of estimating two ratios instead of knowing them is small. The per-period interval’s mean half-width is at most 7.0% wider than the one built on the true ratios, at every configuration, and at four of the seven it is within two per cent.

Flat weights are the other honest option and their cost depends on the alignment in a way that makes them a poor default here. With the small blocks noisier they are 10.3 times as wide as the per-period interval at eightfold: they hand the noisiest blocks the same weight as the cleanest. With the large blocks noisier they are only slightly wider, because the large blocks are then both the noisy ones and the ones flat weights under-count relative to their size.

What each period’s ratio rests on #

The objection to a per-period ratio is the one that sank the per-block ratio: fewer degrees of freedom. It does not apply with the same force, and how much force it has is a count.

The small blocks’ ratio is the thin one, since each small block has two units in arm B and so one degree of freedom there. Across a trial it accumulates 56 within-arm degrees of freedom at a fourfold difference with the small blocks noisier — the trial stops early, after 751 observations, because its pooled precision is optimistic — and 418 with the large blocks noisier. The large blocks’ ratio rests on hundreds to thousands. A per-block ratio, by contrast, rests on one or two degrees of freedom in arm B, which is why it became noise and broke the decomposition.

Fifty-six is enough for the weights to be the inverse variances to within what the coverage can detect: the per-period interval covers 95.0% there. The threshold where a per-period ratio stops being enough is somewhere below the smallest trial in this table, and the eightfold configuration with the small blocks noisier — 53 degrees of freedom, 94.9% coverage — is the closest the measurement comes to it.

The stopping rule pools too #

The interval is not the only thing in the trial that reads a pooled spread. The stopping rule adds blocks until the summed precision of the block differences reaches the target the promised width needs, and it computes each block’s precision from arm A’s and arm B’s spreads pooled across the trial. Under a period structure that sum is wrong in the same direction as the weights.

With the large blocks noisier, the pooled spread of arm B is the large blocks’ spread, so the small blocks’ precision is understated and the rule runs long: 5,642 observations at a fourfold difference against 1,656 with one ratio throughout, and 10,989 at eightfold. The per-period interval at the end of those trials has a mean half-width of 0.234 and 0.159 against a promise of 0.3 — a trial delivering an interval far narrower than it set out to buy, which is the stopping rule paying in observations for the error the weights paid in coverage.

With the small blocks noisier the error runs the other way: the rule stops after 751 observations at fourfold and the per-period interval’s half-width, 0.304, is a shade over the promise. A rule whose precision sum used the per-period ratios would stop at the right time in both cases. It has not been run here, and it is the obvious next construction rather than an established one.

A ratio that wanders is a different defect #

The period structure is one way for the ratio to vary. Another is for it to wander — each block’s ratio drawn afresh, independent of whether the block is large or small. That is the natural model for heterogeneity nobody can attribute to anything, and it behaves completely differently.

With the logarithm of each block’s ratio spread by 0.5 the pooled weights’ calibration factor averages 1.002, and at a spread of 1 — each block’s ratio varying by a factor of about e either way — it averages 1.014. The interval still means what it says. What moves is efficiency: the pooled weights’ estimate has 1.18 times the variance of the inverse-variance estimate at a spread of 0.5 and 1.72 at a spread of 1.

The difference from the period case is alignment. The calibration factor compares a weighted sum of the variances with a weighted sum of their products with the weights, and a wandering ratio is independent of the weights, so on average the two move together. A ratio that follows the periods is correlated with the weights by construction, because the periods are what decide the weights. Variation in the ratio is a cost in width; variation that lines up with the design is a cost in coverage, and only the second is the kind an interval cannot report about itself.

What the design invites, and what it does not #

It is worth saying why this is the corner’s problem in particular rather than a general caution about heteroscedastic data.

A design whose allocation is constant has one kind of block, and a ratio that drifted with calendar time would wander relative to the weights rather than follow them. It is the corner’s alternation — the property that made it the corner in the first place, because it breaks the constant-allocation condition — that creates two kinds of period, and then aligns any between-period change in the arms’ spreads with the weights. The design that needed the ratio weights is the design most exposed to the ratio not being one number.

The recommendation that follows is short. When the allocation changes between periods, estimate the ratio per period. It is blind, it is exact again, and its cost is a few per cent of width; and a trial that has only one kind of period loses nothing by doing it, since with one ratio throughout the per-period and pooled intervals cover identically.

Still open: a stopping rule that reads the periods #

The measurement this leaves undone is the stopping rule. The interval above is repaired by per-period ratios; the decision about when to stop still reads pooled spreads, and that is what made the trial spend about three and a half times the observations its promise needed at an eightfold difference in one alignment, and stop fractionally short in the other. A rule that summed per-period precisions would stop when the per-period interval reached its promised width, and the claim that it remains exact — that a stopping time depending only on within-arm spreads, now per period, is still independent of the block differences — is the same argument the blinded rule rests on and has not been checked on this construction.

A second open question is a ratio that changes within a period, gradually, so that neither one ratio nor two is right. Between the wandering case, which costs width, and the period case, which costs coverage, there is a drift correlated with the design only partly, and how much coverage that costs is a measurement this design can make and has not.

The claim, and its limits #

This essay takes what a variance ratio that differs between the corner’s two kinds of period does to the ratio-weighted interval. The claims are that one ratio pooled across the trial is set by the large blocks, so its weights’ calibration factor is 2.07 to 5.44 at twofold to eightfold differences with the small blocks noisier and 0.65 to 0.53 with the large blocks noisier, in closed form; that counted on three thousand trials the pooled interval covers 100.0% and 85.1% at a fourfold difference; that a ratio estimated per period from within-arm spreads covers between 94.3% and 95.1% at every configuration, at most 7.0% wider than an interval built on the true ratios; that flat weights stay calibrated and are up to ten times wider; and that a ratio wandering at random, independent of the design, leaves the calibration factor within 1.5% of one and costs only efficiency.

What stays out, named as a decision: non-normal outcomes, on which the blinded rule’s independence argument already fails for a separate reason; more than two kinds of period, for which the same construction applies with one ratio per kind and nothing here tests it; and a period structure that is not known, where the analyst would have to find the periods before estimating a ratio for each, which is a search and would have to be charged for.

The boundary against the field’s first essay is that it derived the weights for one ratio and this essay finds the one way the corner breaks that assumption systematically. Against the essay on pooling within a block, whose refusal was pooling across the arm label, this one’s is pooling across the period label — the same mistake made along the other axis of the table.