An effect that is not one number

The interval the test inverts to took the randomisation test a covariate-adaptive trial is entitled to and turned it into an interval. To test whether the effect equals $\tau_0$, subtract $\tau_0$ from each treated outcome, re-run the allocation rule on the same units, and ask whether what was observed looks extreme among the allocations that could have happened. Because the rule reads only covariates, one set of re-run allocations answers every value of $\tau_0$, and the set that is not rejected is the interval. Under minimisation at a hundred and twenty units it covers 95.3% at a width of 0.816, a third narrower than the t interval a regression prints for the same unadjusted difference.

All of that rests on one thing. Subtracting $\tau_0$ from a treated outcome recovers what the unit would have shown untreated only if every unit’s effect is exactly $\tau_0$. That is additivity, and it is a much stronger assumption than any trial believes. If the effect grows with the prognosis, the hypothesised counterfactuals are wrong for every $\tau_0$, and the inverted test is testing a hypothesis that is false at every point of its grid.

So what does the interval then cover? The candidates are not the same thing. There is the population’s mean effect, which is what a trial report would say it had estimated. There is the sample’s mean effect, which is the average over the units actually enrolled. Or perhaps nothing in particular. Either an interval built on additivity degrades gracefully into an interval for an average, or its exactness belonged to the additivity and leaves with it.

It degrades gracefully. The measurement also shows something more useful: why it does, and exactly what that gracefulness costs.

An effect with a slope #

The trial is the field’s standard one. A hundred and twenty units with three baseline factors, sex, age band and centre, of which the prognosis is an additive score, are assigned by Pocock and Simon’s minimisation at a probability of 0.8, the rule balancing what is known in advance introduced and the rule that can be guessed priced. The outcome is the prognosis plus the unit’s own effect if treated, plus unit noise, and the effect is $0.45 + \gamma \times$ the unit’s prognosis. So $\gamma$ is how fast the effect moves with the prognosis, as a multiple of how fast the outcome itself moves with it. At $\gamma = 1$ a one-unit better prognosis means one unit more outcome if untreated and two if treated: a strong interaction by clinical standards, and a value the field’s tables never held.

Each of six hundred trials reports its intervals, and each interval is scored twice: did it contain the population’s mean effect, counted across the trials, and did it contain the mean effect of the units in that trial?

The figure at the head of this essay is coverage against $\gamma$. Read the solid blue and orange lines first, the inverted test, unadjusted and adjusted, for the population’s mean.

At no slope, they cover 96.7% and 96.5%, slightly above 95% at the standard error of 0.9. At a slope of a half, 95.7% and 95.0%. At one, 95.7% and 95.3%. At a slope of 1.5, the strongest here, where the effect ranges by several units across the trial and is negative for many patients, they cover 94.8% and 96.0%. Within sampling error, nothing happened. An interval built on an assumption that is now false at every point of its grid still covers the population’s mean effect at about its level. Against the sample’s mean, the dashed lines, it does better than that, 98.3% and 99.5% at the largest slope.

So that is the answer to the question as it was posed. It is not exact any more, in the sense that the hypothesis it is testing is false, and no theorem now says the number is 95%. But the number stays.

Why the number stays #

The theorem that disappeared said the test is exact under additivity. What is replaced it is older and has nothing to do with re-running allocations: it is Neyman’s, for a difference of means in a randomised experiment. Under random assignment, an estimated difference has a variance over the assignments and an expected residual variance, and the usual variance estimate is conservative for the sample’s mean effect and about right for the population’s. The size of that conservatism is the variance of the unit effects divided by the sample size. In other words the same interval is valid for both questions and sharper for one of them.

The inverted test has inherited this because its reference distribution is built from the observed outcomes, which carry the effect’s spread between units. Residuals around a common fitted effect have a variance that includes that spread. So when the effect has a slope, the test’s reference gets wider, in step with the true uncertainty, with no term in it that knows a slope exists.

That accounts for the behaviour, and it can be checked to more than a shape, because it predicts the width.

The adjusted inverted test’s width runs 0.725, 0.738, 0.773, 0.896 and 1.071 across the slopes. The adjusted t interval is nearly the same line: 0.725, 0.736, 0.769, 0.886 and 1.054. Nearly all of it is reading the same residual variance.

Now look at the bottom line.

The regression that gets narrower, and what it answers #

If the problem is an interaction, the obvious fix is to model it. Fit the arm, the prognosis, and the product of the arm with the prognosis centred at the sample’s mean, so the arm’s coefficient is the effect at the sample’s mean prognosis. On a linear effect that is the sample’s mean effect exactly. The interval that comes out has a width of 0.722, 0.723, 0.723, 0.723, 0.722. It is dead flat across every slope, and it is the width the noise alone requires: the interaction has been fitted and is no longer in the residual. At a slope of 1.5 that is nearly a third narrower than either additive interval.

It is what the current best advice would recommend, and for good reason. It covers the sample’s mean effect at 96.2%, 95.8%, 96.2%, 95.7% and 95.2%: on the level, every time, and doing what the additive interval only approximates.

Then read its coverage of the population’s mean, the solid green line in the first figure: 96.2%, 95.5%, 94.0%, 91.0% and 84.0%.

That is the whole point of the essay, and neither column of coverage is wrong. The regression’s interval is a statement about these hundred and twenty people. The additive interval is, to within sampling error, a statement about the population they came from. The difference is the uncertainty in who turned up. A trial whose effect depends on prognosis has a sample mean effect that depends on the prognosis mix it enrolled: at a slope of 1.5 that sample mean has a standard deviation of 0.186 across trials. That variation is real, and it is unrelated to the treatment assignment. A regression conditioned on the sample’s mean prognosis has no way to see it.

It can be checked on the widths. The regression’s interval has a standard error of about 0.182 at that slope, from its width of 0.722. The sample’s mean effect has a standard deviation of 0.186. Together they give $\sqrt{0.182^2 + 0.186^2} = 0.260$. The adjusted estimate’s own spread across trials is 0.262, and the additive interval’s standard error, read off its width of 1.071, is 0.270. At a slope of 1 the same arithmetic gives 0.220 from 0.183 and 0.122, against a counted spread of 0.223.

So the width that the additive interval adds is the composition of the sample, to within three thousandths. What looked like an inefficiency from failing to model the interaction is the uncertainty a population claim actually has. The extra width was not lost. It is the price of a different question.

What a trial is claiming #

This reframes a choice that protocols usually present as a matter of efficiency.

The additive analyses, adjusted or inverted, are close to valid for a population’s mean effect, and they cost the width that claim requires. They are conservative for the sample’s mean.

The regression with an interaction is valid for the sample’s mean effect, and it is narrower by the amount that the sample’s composition contributes. It is not valid for a population’s mean unless a further term is added for the composition, and at a slope of 1.5 its nominal 95% interval covers 84%.

Most trial reports state their result as though it is about the population. A report that uses an interaction-adjusted analysis, reads its narrower interval, and then writes “the treatment increased the outcome by…” has chosen the second row’s interval and written the first row’s sentence. The same split between what a block’s numbers can answer turned up in the sums under a moving level, and it is easy to miss for the same reason: every interval on the page looks like an interval for the effect. It is a very ordinary thing to do, it is not flagged anywhere, and at large interactions it is wrong by a measurable amount. A clean report would say “among the patients in this trial” at that point, or add the composition’s term back in. That term is one number, computable from the same regression and the sample’s own prognosis spread: the fitted interaction times the sample’s standard deviation of prognosis, over root $n$. With it the regression’s standard error comes out at about what the additive interval reports, and at that point the two analyses have the same interval by different routes.

An effect in one tail #

A slope is the most favourable shape for the additive argument, because the mean prognosis picks out the mean effect. An effect confined to the units past a fixed prognosis cut is a harder test: it is what a subgroup that benefits looks like. Let the effect be 0.45 for everyone below the cut and $0.45 + \gamma$ above it, for the units with a prognosis over 0.5.

At $\gamma = 1.5$, the adjusted inverted test still covers the population’s mean at 95.0%, and the regression at 95.0%. The sample’s mean is more stable under this shape, with a standard deviation of 0.063 across trials rather than 0.186, so the two questions are closer together and the regression’s failure mostly vanishes. The unadjusted inverted test is the one interval that does slip, to 93.8% at $\gamma = 1$ and 93.3% at 1.5, about 1.7 points below the level on a standard error of 0.9. A small effect, and in the direction one might worry about, and it does not appear on the adjusted test.

Two standard errors is not a finding. It is worth flagging only because it is the one place on the page that the unadjusted test sits below its level and the sample’s mean coverage does not compensate. An effect in a subgroup is also the most common claim of heterogeneity in practice, and a more precise run at more trials would be needed before anything could be said beyond that.

Other rules, other sizes #

Minimisation is one rule at one size. The same reading at a slope of one, on 500 trials, across three rules and three sizes.

The gentle result holds up. At 320 units the three rules cover between 93.4% and 97.0%, on a standard error of about one point. At a hundred and twenty it is between 93.8% and 96.2%. The only clear shortfall is at forty units: complete randomisation covers 91.6% unadjusted and 92.6% adjusted, and permuted blocks within strata 92.6% adjusted. The earlier essay’s section on small trials had already reported the additive form of that, at about the same size and with no slope.

So none of it depends on the particular rule, and the reference the covariates supply is the reason: the argument above is about randomisation in general. It is the reference distribution reading the outcome’s spread, and every rule’s reference does that.

Whether the result was ever a test #

There is a real cost to what has been established, and it is worth setting beside the good news. The earlier essay’s headline was that the inverted test’s exactness came with no model of the outcome. Under a slope, it is no longer exact, and its good behaviour rests on a large-sample argument that is as model-free as it is but that is not the same kind of guarantee. An interval that covers 96% for a reason nobody can prove at forty units is a different object from one that covers 95% by a counting theorem.

So the right description of the inverted test has shrunk a little. It is exact for a sharp null, the one hypothesis under which the rule’s re-runs mean exactly what the test says. It is approximately valid for an average effect, because of an argument outside the test. And the two statements have been written as one thing in the analysis has to know the rule and in most of what has been written about randomisation inference since Fisher and Neyman first disagreed about exactly this.

The analysis has to know the rule stated the requirement that a reference distribution come from the rule actually used. Nothing here relaxes that. A re-run under the wrong rule is still wrong under a slope, and it is the reference that must match the allocation, whatever is being estimated.

How the numbers were made #

Six hundred trials at each slope and shape, with 199 re-run allocations in each trial for the inverted tests, both of which use the same allocations as the earlier essay. The population’s mean effect is the average of the sample means across the trials rather than a value worked out, which allows for the prognosis score’s mean of −0.35 on these factors. So the scoring is honest about what the population actually is. The cut for the upper-tail effect is fixed at a prognosis of 0.5, since a unit’s effect cannot depend on who else is enrolled.

The standard error on coverage near 95% at this many trials is 0.9 points, and 1.0 at 500. So coverage can be read to about two points, and 84.0% is unmistakable while 93.8% is not. Four checks hold the result in place. At slopes of 0.5, 1 and 1.5 the unadjusted inverted test must cover the population’s mean within four standard errors of 95%. It must cover the sample’s mean at least about as often. Its width must grow by more than a fifth at a slope of one, and the regression must be narrower than the additive interval there. The reading that a test of a false hypothesis must lose its coverage is refused at the largest slope, against a floor of 90%.

Still open: a composition term for a narrower interval, and what a stopping rule does with it #

The third interval. The regression’s narrow interval and the composition term together imply a third analysis: use the interaction for precision and add the sample-composition variance back explicitly when the claim is about the population. Under a known-mean prognosis this could be narrower than the additive interval, since it is estimating the same thing by a less noisy route, with a covariate’s mean known from a registry, or an external sample. That is a transport question and it is the natural next measurement, for a slope and for a tail.

An interval read on the fly. Every trial here is fixed at its size. A trial stopped on the width of the interval would respond to the slope, because the additive interval’s width carries it, and would stop later when the effect is heterogeneous: which is what a trial should do, and something a regression with an interaction would not do, as its width does not move. Whether that is what happens, and whether either one is then still honest, has not been measured.

Two questions, and the interval that answers each #

This essay takes intervals for a treatment effect under minimisation when the effect varies with the prognosis. Its claims are that the randomisation test inverted to an interval, which assumes an additive effect, covers the population’s mean effect between 94.8% and 96.7% at every linear slope measured, and the sample’s between 96.0% and 99.5%; that its width grows from 0.725 to 1.071 where a regression carrying the interaction stays at 0.722; that that regression covers the sample’s mean at its level and the population’s at 84.0% at the largest slope; and that the difference between their variances is, to within a few thousandths, the variance of the sample’s mean effect.

What stays out, as a decision: non-linear shapes other than one upper-tail step; an outcome with a non-normal noise; and any claim about which question a trial should be asking, which belongs to the people running it. The boundary against the interval the test inverts to is that it held the effect constant and measured what the test delivers, and this one lets it vary and finds what it means.