The test that needs the rule

The randomisation test of the previous essay has an unusually short list of assumptions. No distribution for the outcomes, no large-sample argument, no requirement that the arms be balanced or the patients comparable, no variance estimate that has to be right.

It has one assumption, and this essay is about what happens when it is wrong.

The test has to be told the rule that produced the allocation. Everything it does is re-run that rule; if it re-runs a different one, the reference distribution it builds is the distribution of a trial that never took place.

The mistake is the default #

This would be a minor caveat if getting the rule wrong required an unusual error. It does not. It requires using a permutation test as it is normally packaged.

Every general-purpose permutation routine shuffles labels. Take the group labels, permute them at random among the observations, recompute the statistic, repeat. That procedure is correct — provably exact — under one allocation mechanism: a completely randomised design where every assignment of labels was equally likely.

An adaptive trial does not have that mechanism. Its allocations are wildly unequal in probability: the allocation that sends most patients to the arm that did badly is one the rule would essentially never produce. Shuffling labels generates it with the same probability as any other, so the reference set contains a great many experiments the trial could not have run, and weights them equally with the ones it could.

The rule is not an input to a label-shuffling permutation test, so there is nowhere for it to be wrong. It is simply absent, and its absence is indistinguishable in the output from its being right.

That last property is what makes this worth an essay rather than a warning. A procedure with a missing argument usually announces itself: something throws, or a default is documented, or a column comes back empty. Here the argument was never in the signature. A permutation test called on an adaptive trial’s data runs cleanly, produces a p-value in the usual range, and gives no indication that it has assumed anything at all — because from its own point of view it has not. It shuffled labels, which is what it does.

The gap is between what the routine implements and what the phrase “permutation test” is taken to mean. The routine implements a specific exact test for a specific allocation mechanism. The phrase is read as “an assumption-free test”, and the difference between those two readings is the eight percent below.

What it costs #

Six hundred adaptive trials with no effect in either arm, analysed two ways, on the same seeds:

• told the rule that produced the data, the randomisation test rejects 4.0%;
• told a fair coin instead, it rejects 8.0%.

The second number is the ordinary z’s failure back again — 9.2% on these trials — arrived at by a procedure that looks as though it has fixed the problem.

That is the worst kind of failure this site catalogues. It is not that the repair fails to work; it is that the repair produces output identical in form to a working repair, computed by a procedure whose name and reputation both suggest exactness, and the number it produces is wrong in the same direction and by nearly the same amount as the thing it replaced.

Which direction it fails in, and why #

The direction is not arbitrary and it is worth working out, because it says what kind of error to expect.

An adaptive rule produces unbalanced allocations, and the imbalance is not random: the arm that happened to accumulate successes early is the arm that receives most of the patients. So the rule systematically concentrates good outcomes on one side, and its reference distribution — every allocation that rule could have made from this outcome sequence — contains a great many large statistics.

A fair coin does none of that. Its allocations split the same fixed outcomes roughly evenly, so the successes end up distributed between the arms and the differences in proportions come out small. Its reference distribution is therefore narrower, and an observed statistic that would sit unremarkably inside the adaptive rule’s spread sits out in the coin’s tail.

A narrower reference set means a smaller p-value for the same data, which means rejecting too often. The measured 5% point of the coin-based reference set for this trial is 1.946 — essentially the normal’s 1.96 — against 2.101 for the correct one, and the p-value for the observed statistic falls from 0.2590 to 0.1560.

The two numbers differ by about 8%, and that 8% is the whole of the repair. A test read against 1.946 is the ordinary analysis with extra steps.

Where the rule actually lives #

There is a practical difficulty underneath all of this and it is not statistical.

The rule is a property of how the trial was run, and the analysis is often done by somebody who did not run it, from a dataset that records outcomes and assignments and not the mechanism that produced the assignments. A column saying “arm: A” does not say whether A was chosen by a coin, by permuted blocks of four, by minimisation on baseline covariates, or by a Bayesian adaptive rule with a particular prior and a particular update schedule.

All of those produce different reference distributions. Two of them — the coin and permuted blocks — produce reference distributions close enough that the difference rarely matters, which the measurements below confirm: a coin-allocated trial analysed as permuted blocks comes out at 4.2% against 5.8% analysed correctly, a gap that would not change a decision. The adaptive ones are a different matter entirely.

It is worth noticing that this asymmetry is itself informative about where the risk sits. The rules that are nearly interchangeable are the ones that ignore the outcomes; the rules that are not are the ones that use them. A rule that looks at the data is a rule that has to be reported, and that is a sharper statement than a general call to document the methods, because it says which method and why. A trial that allocated by coin and forgot to say so has lost almost nothing. A trial that adapted and forgot to say so has lost the ability of anyone downstream to analyse it correctly, with no way for them to find out.

So the field’s practical requirement is a documentation requirement: the allocation rule has to travel with the data, in enough detail to be re-executed. Not described — executed. “Response-adaptive” is not a rule; “Thompson sampling from Beta(1 + successes, 1 + failures) on each arm, updated after every patient” is.

This is the same shape as the stopping-rule field’s finding, and the two are worth putting side by side. A p-value is defined relative to a sampling plan, so two experiments with identical data and different stopping rules have different p-values — and the stopping rule is also not in the dataset. In both cases the number that gets reported depends on something the data does not carry, and in both cases the profession’s convention is to omit it.

Every combination, measured #

The direction argument above is a piece of reasoning, and this site’s habit is to count rather than to reason. So here is every pairing of a rule that ran the trial with a rule the analysis assumed, on six hundred true nulls each, at the same seeds. Three rules: a fair coin, permuted blocks of four, and the adaptive rule.

Where the analysis is told the truth:

• adaptive analysed as adaptive — 4.0%
• coin analysed as coin — 5.8%

Where a trial that was adaptive is analysed as something less adaptive:

• adaptive analysed as a coin — 8.0%
• adaptive analysed as permuted blocks — 8.5%

Where a trial that was not adaptive is analysed as something more adaptive:

• coin analysed as adaptive — 4.5%
• coin analysed as permuted blocks — 4.2%

The two directions fail in opposite ways. Assuming less adaptation than there was is anti-conservative — 8.0% and 8.5% at a nominal 5%, which is most of the failure the test existed to repair. Assuming more adaptation than there was is conservative — 4.5% and 4.2%, which is safe and costs power.

Why “assume something conservative” is not a strategy #

The table above suggests an obvious defensive move: if assuming more adaptation is conservative, always assume the most adaptive rule going, and be safe.

It fails for a reason that is worth stating carefully, because the move is tempting and the failure is not about the arithmetic.

“Conservative” is not a property a reference distribution has on its own. It is a property of the relationship between the assumed rule’s allocations and the real ones. Assuming the adaptive rule was conservative in the row above because the real rule was a coin — a less concentrating rule than the assumed one. Against a rule that concentrates harder than Thompson sampling, the same assumption would be anti-conservative, and there is no most-adaptive rule to hide behind: an allocation rule can be arbitrarily aggressive.

So a reader who wants safety without documentation has to bound how adaptive the real rule could have been, and bounding that is knowing it. There is no direction of error that is safe by default here. The only defence is the rule.

The cost of the conservative direction is also not negligible. A test running at 4.2% instead of 5% is giving up about a sixth of its rejection region, and the power that goes with it, in exchange for an assurance that could have been had exactly by writing one sentence into a protocol.

The two critical values predict the eight per cent #

The essay reports two numbers that look like separate evidence — a 5% point of 2.101 for the correct reference set against 1.946 for the coin’s, and a rejection rate of 8.0% against 4.0% — and the first pair very nearly implies the second.

If the correct reference distribution is read as roughly normal, its 5% point of 2.101 puts its own standard deviation at 2.101/1.96 = 1.072. Reading a statistic from that distribution against the coin’s threshold of 1.946 rejects when it exceeds 1.946/1.072 = 1.82 of its own standard deviations, which is 6.9% of the time.

Against a counted 8.0% that is most of the way there, and the residual is the shape: the correct reference set is a discrete distribution over the allocations one rule could make, and it has heavier shoulders than a normal. So the whole of the failure is the threshold, and the number 1.946 is worth carrying as the essay’s second headline — a permutation test on an adaptive trial is reading a statistic against 1.946, which is 1.96, which is the ordinary analysis it was reached for instead of.

That is a sharper statement than the rates alone. The routine did not merely fail to repair the problem; it reconstructed the normal quantile from a simulation, by a route with no normal distribution anywhere in it.

What each direction of error costs in power #

The table’s two directions are called anti-conservative and conservative, and both are worth pricing in the currency an experimenter cares about.

Assuming less adaptation than there was runs at 8.0% instead of 5%, which is a threshold of 1.751 rather than 1.960. Against an alternative a trial is powered at 80% for, that buys 5.3 points of power it did not pay for — 85.3% against 80.0% — at a cost of three points of size. Nobody chose that trade and nothing in the output reports it.

Assuming more adaptation runs at 4.2%, a threshold of 2.034, and costs 2.1 points of power at the same alternative and about 3.0 at a marginal one. That is the price of the defensive move the essay declines, and it is smaller than the phrase “a sixth of the rejection region” suggests — because power is a tail probability of a shifted distribution and loses less than the level does when a threshold moves.

The asymmetry is worth stating because it changes the shape of the advice. The safe direction is cheap — two or three points of power — and the unsafe one is not merely unsafe, it is unsafe in a way that looks like a gain. A trial analysed as less adaptive than it was reports both a lower p-value and a higher apparent power, and every visible quantity moves in the direction an investigator would welcome.

The rule is data, and this site keeps finding that out #

Three fields on this site have now arrived at the same structural fact from three directions, and putting them together is worth more than any of them separately.

Stopping rules. A p-value is defined relative to a sampling plan, so two experiments with identical observations and different stopping rules have different p-values. Testing five times at the nominal level rejects a true null 14% of the time. The stopping rule is not in the dataset.

Analytic choices. Twenty analyses of nothing measures what a handful of individually defensible decisions — which cases to exclude, which covariates to include — do to a rejection rate when the reported analysis is the one that worked. The branching is not in the dataset.

Allocation rules. This essay. The rule that assigned the arms decides the reference distribution, and the rule is not in the dataset.

In all three the pattern is identical: a number’s meaning depends on a procedure, the procedure leaves no trace in the output, and the convention is to omit it. And in all three the omission is not a lapse by careless analysts — it is that the file format has no field for it. A dataset is a table of what happened. A procedure is a description of what else might have.

The practical consequence is the same in all three cases and it is not a statistical recommendation. It is that the protocol has to be a document that can be executed rather than a document that can be read, and that reporting the analysis means reporting the machinery that would have produced a different number.

What the field looks like from here #

Three positions, and the field’s contribution is to price them rather than to recommend one.

Use the ordinary analysis. Costs nothing, needs nothing, and rejects 9.2% of true nulls on this design.

Use a permutation test as packaged. Costs B re-randomisations, needs nothing beyond the data, and rejects 8.0% — most of the failure retained, with the appearance of rigour added.

Use the rule. Costs B re-randomisations and a documentation requirement most trials do not currently meet, and rejects 4.0%.

The gap between the second and third is the whole content of this essay, and it is invisible from inside either analysis. Both print a p-value with four decimal places.

There is a fourth position worth naming, which is to design the trial so the question does not arise. A fair-coin allocation needs no rule to be documented because the packaged permutation test already assumes it correctly, and the ordinary z is right as well. Everything in this field is a consequence of adapting, and adapting bought something real — most of the trial on the better arm, which is what the adaptive field measures at 86.6%. The analysis burden belongs on the same ledger as that benefit, and it is usually left off.

The check, and the refusal that makes it mean something #

The refusal here is the essay, which is unusual for this site: normally the refusal is a check appended to a field’s machinery, and here the field’s whole subject is what the machinery refuses.

The field’s library asserts both halves on identical trials and identical seeds — told the right rule the test holds its level, told a fair coin it does not — with the second requiring the failure to exceed the first by at least two percentage points rather than merely to differ. A version that had drifted to a smaller gap would be reporting a distinction that no longer justified the essay, and the assertion is written to say so.

The second refusal is the one that keeps the first honest. On a fair-coin trial with no drift, where the ordinary analysis is already correct, the randomisation test is required to agree with it. Without that, a test that rejected nothing would satisfy every claim made here, and “told the wrong rule it rejects more” would be a statement about a test that never rejected anything at all.