The interval the test inverts to

The reference the covariates supply made the case for testing a covariate-adaptive trial against its own allocation rule. Hold the outcomes fixed, re-run the rule on the same patients, and count how often a re-run allocation produces a statistic at least as large as the observed one. Because the rule reads only what was recorded before anything happened, the re-runs need nothing about the outcome, and the test is exact. Read that way, the plain unadjusted difference in means went from 20.3% power against the t table to 55.0%.

The essay listed what it left out, and one item was an interval: confidence intervals by inverting the test, available and a different amount of work. A p-value is the answer to one question — is the effect zero — and a trial reports an effect with an interval. This essay does the work. It turns out to be less than the essay feared, for a reason that is worth having on its own, and the interval it produces is the clearest statement yet of how much of the t interval’s width was the rule’s.

Testing every effect at once #

An interval is the set of effects a test would not reject. Under the hypothesis that the treatment adds exactly $\tau_0$ to every patient’s outcome, the outcomes each patient would have had on the other arm are known: the treated patients’ outcomes less $\tau_0$, the control patients’ as observed. So the test of $\tau_0$ is the test of no effect on the adjusted outcomes $y - \tau_0 a$, where $a$ is the observed arm indicator, and the interval is every $\tau_0$ whose p-value exceeds 5%.

Done naively that is a fresh randomisation test at every grid point, and on a grid fine enough to place an interval’s ends that is hundreds of tests of two hundred re-run allocations each. It is not necessary, and the reason is the one that made the test cheap in the first place.

The re-run allocations do not depend on $\tau_0$. The rule reads covariates, the covariates do not change when a hypothesis does, and so the same two hundred allocations are the reference set for every $\tau_0$. And both statistics are linear in the outcomes: a difference in means and a least-squares coefficient. Under a re-run allocation $A^*$ the statistic of $y - \tau_0 a$ is

$$u^* - \tau_0 v^*,$$

with $u^*$ the statistic of the observed outcomes and $v^*$ the statistic of the observed arm indicator, both computed under $A^*$. Two numbers per allocation, computed once, answer every hypothesis. The whole interval costs what one test cost, and the grid search over $\tau_0$ is arithmetic on stored pairs.

The statistics are used unstudentised. What the exactness buys measured the test with studentised statistics; a t-statistic is not linear in $\tau_0$, so a studentised test would need its own re-run allocations at every grid point. The earlier essay’s finding is what makes that safe to give up: the choice of statistic and the choice of reference distribution are independent, the reference does the calibrating, and an unstudentised statistic read against the right reference is as exact as a studentised one.

Four intervals on the same trials #

The trials are the earlier essay’s: a hundred and twenty patients with three prognostic factors at unequal level frequencies, a prognosis that depends on all three, a treatment effect of 0.45 added to every treated patient, and minimisation over the factors’ margins taking the preferred arm with probability 0.8. Four intervals are computed on each: the unadjusted and the covariate-adjusted t intervals a regression prints, and the randomisation test of each statistic inverted, on two hundred re-run allocations.

The unadjusted t interval covers 99.7% at an average width of 1.229. That is the conservatism the analysis has to know the rule found as a test that rejects 0.6% of true nulls where it claims 5%: the t interval’s standard error is computed as though the patients had been allocated by a coin, and minimisation has already removed most of the imbalance that standard error is paying for.

The inverted test of the same unadjusted difference covers 95.3% at 0.816 — 33.6% narrower — with no model of how the outcome depends on the factors. The adjusted t interval covers 96.0% at 0.723, and the inverted adjusted test 96.2% at 0.722.

So the inverted unadjusted interval recovers most of what adjustment recovers, from the opposite direction. Adjustment narrows the interval by modelling the outcome and removing the prognosis from it; inversion narrows it by reading the statistic against the distribution the rule actually generates, in which the prognosis was already balanced. The gap that remains — the inverted unadjusted interval is 13% wider than the adjusted ones — is the prognostic imbalance minimisation leaves when it takes the preferred arm only four times in five.

And the inverted adjusted interval is the adjusted t interval to within a thousandth. Once the statistic has removed the prognosis there is nothing left for the reference distribution to correct, and the t table is close enough to right.

What the width does to a decision #

The interval and the test it came from carry the same information, and the widths make the earlier essay’s power numbers legible. The effect in these trials is 0.45. The unadjusted t interval’s half-width averages 0.615, larger than the effect, so on a typical trial it reaches across zero and the test does not reject: the t table found the effect 20.3% of the time. The inverted interval’s half-width averages 0.408, smaller than the effect, so on a typical trial it stops short of zero: the exact test found the effect 55.0% of the time. A third of the width is the difference between a trial that usually misses a real effect and one that usually finds it.

The same arithmetic says what the remaining gap to adjustment is worth. The adjusted intervals’ half-width is 0.36, and against an effect of 0.45 that is a margin of 0.09 rather than 0.04 — the difference between the adjusted analyses’ power and the inverted unadjusted test’s — the arithmetic how many subjects sets out, read off a ruler rather than a table.

Two routes to the adjusted interval #

The two adjusted intervals are worth one more sentence, because their agreement is not automatic. One is model-based: it reads the treatment coefficient against a t distribution on the regression’s residual degrees of freedom, which is right if the outcome is linear in the factors with independent errors. The other is design-based: it reads the same coefficient against the distribution the allocation rule generates on these patients, which is right whatever the outcome model is, provided the effect is additive. They share the statistic and nothing else.

That they agree to within a thousandth at a hundred and twenty patients — 0.723 and 0.722 — is two routes to one number in its most useful form: the model’s assumptions are true here by construction, and the design confirms them without needing to know they are. Where they part, at forty patients, the model-based interval is the one that has moved, which says where its premise has thinned.

The third is the rule’s #

A narrower interval is only interesting if its width comes from where it is claimed to. The control is a rule that balances nothing.

Under complete randomisation — a coin for every patient — the unadjusted t interval covers 95.5% at 1.231 and the inverted test 95.5% at 1.233. They are the same interval. The t table’s assumption that allocations come from a coin is true, the re-run allocations are coins, and reading the statistic against them changes nothing.

Under stratified blocks the t interval is 1.235 wide at 99.7%, the inverted test 0.861 at 94.7%. Under minimisation, 1.229 against 0.816. The two balancing rules order the way balancing what is known in advance said they would: the prognosis here is a sum of the three factors’ effects, so it is balanced by balancing the margins, which minimisation targets directly; blocks inside twenty-four cells spread a hundred and twenty patients thinly enough that many cells end on an incomplete block, and every incomplete block is an imbalance the margins inherit. The width the inverted test removes is the width the rule’s balancing made unnecessary, measured rule by rule, and on the rule that balances nothing it removes none.

That is the cleanest form the earlier field’s point has taken. “The rule is part of the data” was a statement about which reference distribution a test should use. As an interval it is a number: under minimisation, a third of the width a trial reports from a t table is a charge for an imbalance the trial does not have.

When the rule stops being random #

Minimisation with the preferred arm taken every time is the case the field has kept returning to, because a deterministic rule leaves a randomisation test nothing to re-run. Here the rule breaks ties between arms with a coin, so a reference set exists, but a thin one.

The inverted unadjusted test covers 93.8% at 0.775, and the inverted adjusted test 94.3% at 0.740: both a point or so short, on six hundred trials where a coverage’s standard error is nine tenths of a point. The t intervals are unaffected — 100.0% and 95.2% — because they never consult the rule.

The shortfall has the shape the earlier essays predicted. A re-run allocation can differ from the observed one only where the rule met a tie, so the two hundred re-runs are a small number of distinct allocations repeated, the reference distribution is lumpy, and an interval read off its percentiles is placed with less precision than two hundred draws suggest. As the probability of the preferred arm falls from one the reference set fills in. The rule that can be guessed found 87.6% of assignments guessable at full determinism; the randomness that protects a trial from a guesser is the same randomness its analysis is built from.

At forty patients #

With forty patients the unadjusted t interval covers 99.7% at 2.191, and the inverted test 95.3% at 1.672 — 23.7% narrower, less than at a hundred and twenty, because with three factors and nine margin levels a rule balancing forty patients leaves more of the imbalance it was asked to remove. The adjusted t interval covers 94.7% at 1.327 and the inverted adjusted test 94.8% at 1.338.

The adjusted t interval is now the one with the least margin. With forty patients and eight coefficients the residual degrees of freedom are thirty-two, and the model’s standard error rests on them; the inverted adjusted test, which reads the same statistic against the rule, does not, and sits at the same coverage a little wider. At this size the two adjusted intervals are interchangeable and the unadjusted pair are not.

What a trial can report #

The inverted interval is available to any trial whose allocation rule is written down, at the cost of one randomisation test. It needs the rule stated completely — which arm a tie goes to, what probability the preferred arm is taken with, which factors were in the margins — because the reference set is generated from that statement and nothing else. It needs no model of the outcome. And it delivers an interval at its nominal coverage that is a third narrower than the unadjusted t interval under minimisation and the same as it under a coin.

There is a practical asymmetry in that list. A trial that loses its allocation log can still report the adjusted t interval, which needs only the data; it cannot report the inverted one, because the reference set has to be regenerated from the rule and the rule from its statement. So the inverted interval is not only a method but a reason to keep a record: the probability of the preferred arm, the tie-breaking convention and the list of margins are part of what the interval is computed from, in exactly the way the outcomes are. A protocol that states them has made its trial’s analysis reproducible by anybody holding the data. A protocol that says only minimisation has left its reader the t interval, a third too wide, or a reconstruction that is, as the earlier essay found, a coin’s.

And the inverted interval is an honest summary of the trial in a sense the adjusted one is not. The adjusted interval is narrower, but it is narrower by a model that the analyst chose after the outcomes existed and could have chosen differently. The inverted unadjusted interval was fixed, statistic and reference together, before any patient was enrolled — and on these trials it gives up 13% of width for that.

What it assumes is additivity: that the treatment adds the same amount to every patient. That is what makes the hypothesised counterfactuals $y - \tau_0 a$ known, and it is the price of turning a test of no effect — which assumes nothing about how the effect varies — into an interval for one number. A trial whose effect varies with the prognosis has no single $\tau_0$ to be the truth, and what the inverted interval covers then is a different question.

The experiments that could have happened set out the logic this rests on for an exact test; the interval adds one step to it, and the step is free because the allocations never depended on the outcome.

Six hundred trials and two hundred re-run allocations each #

Six hundred trials in each of five configurations, with two hundred re-run allocations per trial for both inverted tests, which share them. The interval is found on a grid of four hundred and one values spanning six reference standard deviations either side of the point estimate, and in no trial did an interval reach the grid’s edge. The unadjusted statistic is the difference in arm means; the adjusted statistic is the treatment coefficient of a least-squares fit on the arm indicator and the factor levels as indicators. The t intervals use the pooled-variance standard error and the regression’s standard error on their residual degrees of freedom.

The p-value of each grid point is the share of re-run allocations whose statistic of the adjusted outcomes is at least as large in absolute value as the observed one, counting the observed allocation, so the interval is the set where that share exceeds 5%.

Two thirds of the width under minimisation, and a point short when nothing is random #

The claims are that inverting the randomisation test for an additive effect needs one set of re-run allocations, because the rule reads only covariates and both statistics are linear in the outcomes; that under minimisation at p = 0.8 the inverted unadjusted interval covers 95.3% at two thirds of the unadjusted t interval’s width; that under complete randomisation the two intervals coincide; that the inverted adjusted interval is the adjusted t interval to within a thousandth; and that under deterministic minimisation the inverted intervals fall about a point short.

Not claimed: anything about effects that are not additive, about outcomes that are not continuous, or about rules with more than two arms, where the margin score generalises in more than one way and the reference set with it.

Still open: an effect that is not one number #

Under a treatment effect that grows with the prognosis, the hypothesised counterfactuals are wrong for every $\tau_0$ and the inverted test is testing a hypothesis that is false everywhere. What it then covers is the question: the average effect, the effect at the average prognosis, or nothing in particular. A trial of minimisation with an effect proportional to the prognosis, scored by how often the inverted interval contains each of those quantities, is the measurement that would say whether an interval built on additivity degrades gracefully into an interval for an average — which is what a trial would report it as — or whether its exactness was the additivity’s and leaves with it.