The sums under a moving level

A block of a two-arm trial reports two numbers beside its within-arm spreads: the difference between the arms’ means and their sum. The difference is what the trial is about, and its scatter between blocks gives the interval its variance estimate. The degrees of freedom in the sums noticed that the sums are a second copy of the same variance, free, and that pooling the two doubles the interval’s degrees of freedom. It also noticed the case that spoils it. Let the block’s level move, and let the effect fall where the level rises at a slope of −2 — a ceiling, where the treatment adds little when everybody is doing well. A block’s sum is then flat while its difference carries everything, pooling halves the excess, and the interval that says 95% covers 88.75%.

The sums made independent then found a different problem in the sums and repaired it. On an unbalanced block the sum and the difference are correlated, and taking from each sum the part its own difference predicts, $\tilde s_b = s_b - \rho_b d_b$, removes that correlation exactly. The residualised sum is independent of everything in the honest interval, and at a fixed level that makes the pooled interval exact at every allocation.

What it did not ask was what happens when both are true. The residualised sum holds the level with weight one and the effect with weight $1 - \rho_b$, so on an unbalanced design a ceiling is partly carried into it and partly removed. Whether that softens the six-point failure, worsens it, or leaves it alone would decide whether the sums can ever be pooled safely in a trial whose level drifts. That is most trials long enough to be blocked.

It softens it, and the softening turns out to be a sideways move.

Who carries what #

The arithmetic fits on one line per piece. Let a block’s level be $L$, and let the effect move with it at a slope $\beta$. Then, ignoring noise and the constant effect, a block’s

• difference carries $\beta L$,
• sum carries $(2 + \beta) L$,
• residualised sum carries $(2 + \beta(1 - \rho_b)) L$.

At a balanced block $\rho_b = 0$ and the last two are one line. On a block of ten and two, $\rho_b = -2/3$, and the residualised sum carries $(2 + \tfrac{5}{3}\beta) L$. Each of them enters a pooled sum of squares against its own noise, on the weights the inverse-variance essay worked out, and the noise in the residualised sum is smaller than in the raw one by $1 - \rho_b^2$, so its movement has to be read per unit of that: the residualised sum’s reading of the level is its weight divided by $\sqrt{1 - \rho_b^2}$.

The figure says what a measurement has to look like before one is taken. Pooling works where a sum sees what the difference sees: where the line for the sum crosses the line for the difference. That is the whole of the condition. Above the difference’s line the sums overstate the between-block variation and the interval is too wide. Below it they understate, and the interval is too narrow.

The raw sum crosses at one slope. $|2 + \beta| = |\beta|$ has one solution, $\beta = -1$. There the level enters the sum and the difference with equal weight, and pooling is unbiased. At every other slope it is not, and at the ceiling the sum’s weight is zero, which is the earlier essay’s failure stated in its simplest form.

The residualised sum on ten and two crosses at two. $|2 + \tfrac{5}{3}\beta|\cdot\tfrac{3}{\sqrt 5} = |\beta|$ gives $\beta \approx -0.83$ and $\beta \approx -2.17$. Between those, the sum’s reading is below the difference’s, and the zero of the sum has moved from −2 to −1.2. The range of under-coverage has not closed. It has narrowed, moved and gained a second edge.

So before any simulation runs, the answer to the old essay’s question is already readable: residualising cannot make the sums safe, because it has only changed where the crossing is. What it can do is reduce how far under the line the sums go.

The curve on ten and two #

The figure at the head of this essay runs that design: ten and two in every block, at a requirement of 0.3 on 4,000 runs per slope, with a level spread of 0.5 between blocks. Three intervals. The sums refused sit at 95% throughout, as they must, since that interval never looks at the sums and the block count is set by the within-arm spreads alone.

Pooled raw, the interval covers 88.8% at a slope of −3, 90.0% at a ceiling, and 94.8% at −1, where the closed form says it should be right. By −0.5 it is over-covering at 97.2%, and from 0 on it covers 98.8% to 99.4%.

Pooled residualised, it covers 97.0% at −3, 93.8% at the ceiling, a trough of 92.7% at −1.5, 94.0% at −1, 97.7% at −0.5 and then 99.6% or better. On this design the residualised sums under-cover only inside $(-2.17, -0.83)$, as the closed form said, and at their worst by 2.3 points. The raw sums under-cover everywhere below −1, by as much as 6.2.

So the answer to the question as asked is that residualising softens the failure on this design substantially, from six points down to about two. For a design with a real imbalance and a trialist who fears a ceiling, that is a genuine improvement.

It is also a trade the reader should see the rest of. The curve above zero is where most trials actually are: a level that moves and an effect that either does not move with it or moves slightly with it. Across that part of the range the residualised interval does worse than the raw one, covering 99.6% at no slope against 98.8%, and 99.9% against 99.4% at one.

Four designs #

The size of the softening depends on the design, in a way the weights predict.

At six and six nothing moves, because $\rho_b = 0$ makes the residualised sum the raw sum on the same runs. The numbers are exactly equal in the table, and a design at balance gets none of this help. A balanced trial, the one most protocols would call the most careful, is the one with the full failure.

Eight and four, swapped, gets about a point and a half. The swap is there because the earlier essay found the alternating allocation to fail on its own terms when the sums are raw, and it means $\rho_b$ takes the values $\pm 1/3$ in turn. A correlation of a third is not much of a lever on the level’s weight.

Ten and two, in either arrangement, gets about four points. And ten and two swapped has the shallowest trough of all: under-covering at most by 1.1 points, with its lowest value right at the ceiling. In the swapped blocks the weights alternate, $2 + \tfrac{1}{3}\beta$ and $2 + \tfrac{5}{3}\beta$, the two zeros are far apart and neither block is ever flat when the other is. So the residualised sums there see the level in one kind of block or the other at any slope in range, and the reading averages across the two.

The strange thing in that table is what it recommends. The more unbalanced the block, the safer the sums. An imbalance has nothing to do with any level, and it is costly: ten and two needs about 24.6 blocks where six and six needs 14.6, at the same requirement. Nobody would choose it to protect a variance estimate. It only happens to help because it spreads the zero of the sum away from the slope that a ceiling sits at.

What over-coverage costs #

The word safely in the question usually means “does not under-cover”, and on that reading the upper half of the plot looks harmless. It is not. A level that moves with no slope on the effect is the most ordinary thing a blocked trial can meet, and there the interval pools an estimate carrying four times the level’s variance into one carrying none.

At no slope on ten and two, the honest half-width is 0.335 against a requirement of 0.3. Pooling the raw sums makes it 0.440, and pooling the residualised ones 0.515, half as wide again. At a slope of one, 0.399 becomes 0.571 and 0.811, which is twice as wide. On six and six at no slope, 0.322 goes to 0.488.

So the pooled interval does not merely over-cover. It fails its own promise of a fixed width, in the direction that a fixed-width design was chosen to prevent. A trial stopped to deliver a half-width of 0.3 reports one of 0.5, and at that point the experiment’s whole purpose has gone. Set against what pooling was supposed to buy: the most it ever narrows the interval below the honest one, at the raw sums’ crossing, is a matter of about three or four per cent, 0.399 to 0.386 on ten and two and 0.426 to 0.409 at six and six.

There is a small upside of a different kind, and it is what the earlier essay valued. A block size that changes priced the same kind of gain when block sizes vary. More degrees of freedom shorten the $t$ point, and that is a fixed gain whatever the level does. With about 15 blocks that gain is small, and with 3 or 4 it is real: the earlier essay’s case for pooling at very short trials is unaffected in the case of no level movement. But very short trials are exactly the ones least able to tell whether the level is moving.

The sums cannot be pooled safely, and it is not the residualising’s fault #

So here is the direct answer. In a trial whose level moves, on these designs, with this size of movement:

At balance the raw sums fail at a ceiling by about seven points, and over-cover everywhere above $\beta = -1$. Residualising has no effect.

On a strong imbalance the residualised sums fail by about two points over a narrow range, and over-cover elsewhere by more than the raw sums do. They are better at what they were pressed into doing and worse the rest of the time.

In neither case is there a free lunch at any slope a trialist cannot name in advance. The one slope that makes pooling exact is a particular value of an unknown nuisance. What the sums’ arithmetic has shown is that the sum sees the level through a weight that depends on an unknown parameter, and no linear operation on the block’s two numbers takes that dependence away. Residualising is the operation that takes the correlation away, and that was a separate problem. It is exact for the correlation and has no opinion about the slope.

And there is a stronger statement sitting underneath it. The difference’s share of the level is $\beta$, the sum’s is some other linear function of $\beta$, and the zero of one is not the zero of the other. So any weighted sum of a block’s two numbers, other than the difference itself, is mismatched at almost every slope. A pooled estimate needs its second piece to see exactly what the first sees, and only a copy of the first does that.

The rule for a protocol, then, is a stronger form of what the earlier essay concluded, and it fits what the blindfold costs in being a limit on what a block’s numbers can be asked to do: refuse the sums whenever a level movement between blocks is possible, and treat that as the default. Reach for them only with a short trial, a stable level that can be defended, and a clear statement in the protocol that the interval is conditional on that stability.

How the numbers were made #

The trial is the field’s standard one: twelve units in each block, a requirement of half-width 0.3 with a stopping rule reading within-arm spreads alone, unit noise, and the level between blocks drawn with a standard deviation of 0.5. For each of four designs and nine slopes there are 4,000 runs on common seeds, which is why six and six’s two pooled columns agree to the digit. The sums are pooled on $2(b-1)$ degrees of freedom as they stand, and the residualised ones on $(b-1) + (b-1)$ or, where the allocation alternates and the mean is regressed on $\rho_b$ as well, $(b-1) + (b-2)$, as in the essay that introduced them.

The standard error on a coverage near 95% is 0.34 points at this many runs, so differences of less than about a point between neighbouring columns are not worth reading.

Two routes meet here. The closed-form crossings on ten and two, at about −0.83 and −2.17, predict the two points at which the residualised curve should cross 95%, and it does cross between −0.5 and −1 and between −2 and −2.5. And the earlier essay’s 88.75% for the raw sums at a ceiling on eight in every block matches the same failure’s 87.6% on twelve at a smaller requirement, in size and in direction. The sums refused must stay within four standard errors of 95% at every slope. The residualised sums must exceed the raw ones at the ceiling by more than two points on ten and two. The claim that residualised sums hold their level at every slope is refused on at least three of the four designs at a floor of 93%.

Still open: a test before pooling, and a level with a trend #

A pretest. The obvious practical repair is to compare the two estimates before pooling: if the sums’ sum of squares and the differences’ disagree by more than an $F$ point, do not pool. That has the shape of a pretest, and every pretest measured in these pages has mostly lost. On four to six blocks, where the sums would help, the $F$ has so little power that it passes a real ceiling most of the time, and on thirty blocks, where it has power, the sums are no longer worth having. The analyses that were really run is the general form of that worry. Whether this case is the same or an exception is a table that can be computed from here.

A level with a trend rather than a scatter. Every level here is drawn independently per block. A trial’s level usually drifts: it is period and site and season, and it is autocorrelated. A smooth level would be absorbed differently by a sum of squares about a weighted mean than by a difference, and the blocks’ swap pattern would interact with its period, on which the variance that follows the period has already found evidence of its own. An alternating allocation under a smooth trend is a design with a known defect in a neighbouring field, and it has not been measured here.

What this leaves of the earlier repair #

This essay takes the block sums of a two-arm trial when the level moves between blocks and the effect moves with it. Its claims are that the level’s weight on a difference, a raw sum and a residualised sum are $|\beta|$, $|2 + \beta|$ and $|2 + \beta(1 - \rho_b)|/\sqrt{1 - \rho_b^2}$; that the raw sum matches the difference at $\beta = -1$ and the residualised one on ten and two at about −0.83 and −2.17; that at a ceiling residualising moves coverage on ten and two from 90.0% to 93.8% and does nothing at balance; that the same interval reports half-widths of 0.515 and 0.811 at no slope and at one, against the requirement’s 0.3; and that the most pooling ever narrows the interval across this range is about four per cent.

What stays out, as a decision: a level with a trend; other sizes of level movement, since 0.5 is one setting and the numbers scale with the ratio of the movement to the noise; and effect heterogeneity independent of the level, measured in the earlier essay. The boundary against the sums made independent is that it held the level fixed and repaired a correlation, exactly, and this one moves the level and finds that repair orthogonal to the new problem.