The plus one and the round number

The randomisation p-value has an oddity in it that every implementation carries and few explain:

p = (1 + number at least as extreme) / (1 + B)

Not the number at least as extreme, over B. Both halves get a 1 added, and the usual explanation is that the observed allocation is one of the possible allocations and should count as one of its own reference draws.

That explanation is correct and it does not say what the correction is worth. This essay measures it, and what the measurement says is stranger than the correction being important or being negligible.

The arithmetic is exact, so it is arithmetic #

There is no simulation needed for the main result here, and doing it in closed form is what makes the finding visible.

Under the sharp null the observed allocation is exchangeable with the B re-randomised ones — that is the whole basis of the test. So the count k of re-randomisations reaching the observed statistic is uniform on {0, 1, …, B}, each value with probability 1/(B+1).

Both rules are thresholds on that integer:

• with the +1, reject when (1 + k)/(1 + B) ≤ α, which is k ≤ ⌊α(B+1)⌋ − 1;
• without it, reject when k/B ≤ α, which is k ≤ ⌊αB⌋.

And since k is uniform on B + 1 values, the true size of each rule is (number of k values that reject) divided by (B + 1).

That is it. No sampling error, no approximation, and every number in this essay’s main figure is integer arithmetic.

The floors are where all the structure lives. Both thresholds are the same linear function of B up to a constant — αB against α(B+1) − 1, which differ by 1 − α, a number just under one — so whether they round to the same integer depends entirely on where αB sits relative to the nearest whole number. Most of the time it sits somewhere in the middle of an interval and the two floors differ by one, which is one extra value of k and therefore one extra 1/(B+1) of size. Occasionally αB lands exactly on an integer, the two floors coincide, and the rules become identical.

What it gives #

At B = 19 and α = 0.05: the corrected rule rejects when k ≤ ⌊0.05 × 20⌋ − 1 = 0, so one value out of twenty, and its size is exactly 5.00%. The uncorrected rule rejects when k ≤ ⌊0.05 × 19⌋ = 0 — the same value, so the same rejection region, and the same 5.00%.

At B = 19 the two rules are the same rule.

At B = 20: the corrected rule rejects when k ≤ ⌊0.05 × 21⌋ − 1 = 0, one value out of twenty-one, size 4.76%. The uncorrected rule rejects when k ≤ ⌊0.05 × 20⌋ = 1, which is two values out of twenty-one, size 9.52%.

At B = 20 the uncorrected rule is a 9.52% test. The size doubles, from adding one re-randomisation.

At B = 39 and B = 59 the two rules coincide again, exactly. At B = 40 they diverge again, to 4.88% and 7.32%.

Both departures are bounded, and they are the same size #

Because k is uniform on B + 1 values, both rules’ sizes are integers over B + 1, and how far each can be from α follows in one line each.

The corrected rule rejects on $\lfloor \alpha(B+1)\rfloor$ of the B + 1 values, so its size is $\lfloor \alpha(B+1)\rfloor/(B+1)$ — never above α, and never more than $1/(B+1)$ below it.

The uncorrected rule rejects on $\lfloor \alpha B\rfloor + 1$ of them. Its size exceeds α whenever the fractional part of αB is below $1 - \alpha$, which at α = 0.05 is nineteen values of B in twenty, and its excess is at most

$$\frac{1-\alpha}{B+1} .$$

So the two rules are not a correct one and a broken one. They are two roundings of the same integer threshold, one always down and one almost always up, each wrong by at most about $1/(B+1)$.

At B = 100 that bound is 0.0094 — the uncorrected rule can be a 5.94% test, which is a nineteen per cent relative excess. At B = 1000 it is 0.00095, and at B = 10,000 it is a ten-thousandth.

Which reframes what the correction buys. It does not remove an error of a fixed size; it changes the direction of an error whose size is set entirely by B. A practitioner who cannot tolerate a conservative test and a practitioner who cannot tolerate an anti-conservative one want different rules, and both of them want a larger B far more than they want either rule.

The condition, and the convention #

Read the staircases and the pattern is plain: the two rules agree exactly when B + 1 is a multiple of 1/α, and disagree everywhere else.

At α = 0.05 that means B + 1 ∈ {20, 40, 60, 80, …}, so B ∈ {19, 39, 59, 79, …}. And the value everybody writes is 999, whose successor is a multiple of twenty. So is 199. So is 99.

The convention is doing the work the correction was supposed to do.

The size of the gap is worth stating exactly, because it is exactly one thing. The two thresholds differ by at most one value of k, so the two sizes differ by either zero or exactly 1/(B + 1) — one step of the p-value’s own lattice, never more and never anything in between.

That gives the correction two properties at once, and the usual telling has one of them.

It shrinks with B, which is the part everybody knows: one lattice step is 4.76 points at B = 20 and 0.10 points at B = 1000, so the uncorrected rule is a 9.52% test at the first and a 5.09% test at the second. At large B, dropping the +1 is genuinely a small error.

And it is exactly zero at the values everybody uses, which is the part that is not usually said. Not small — zero, at B = 19, 39, 59, 99, 199 and 999, at every level anybody tests at. A practitioner who has never included the correction and always written B = 999 has not been making a small error. They have been making none.

Nobody chose 999 for this reason. It is written that way because the p-value comes out in neat thousandths — 1/1000, 2/1000 — which is a presentational preference. The presentational preference happens to be exactly the condition under which the arithmetic error disappears.

This is a shape worth naming, because it is not the usual one. The site’s normal finding is that a convention hides a failure: a p-value read without its stopping rule, a critical value read from the wrong table, an interval whose coverage was never counted. Here a convention prevents a failure, by accident, and has been doing so reliably for as long as anybody has written 999.

The consequence is not that the correction can be dropped. It is that the correction’s absence is untested, and that the size of what it protects against depends entirely on B. A practice whose correctness depends on an unstated numerical coincidence is one step from being wrong: the person who writes B = 1000 because it looks tidier gives up a tenth of a point, which is nothing, and the person who writes B = 20 to save time doubles the size of their test. Neither gets a warning, and the two mistakes look identical from outside — the p-values still land between 0 and 1, still look plausible, and still get compared to 0.05.

A correctness that depends on a habit is a correctness with no defence. Including the +1 makes the habit irrelevant, which is the actual argument for it.

The condition is about 1/α, not about twenty #

The multiples of twenty are a consequence of testing at 5%, not a fact about randomisation tests. Change the level and the meeting points move with it.

At α = 0.10 the two rules coincide at B = 9, 19, 29, 39, 49, 59 — twice as often, because 1/α is half as large. At α = 0.01 they coincide at B = 99, 199, 299 and nowhere in between, so the sweep has to run five times further before it contains a single meeting point at all.

That last observation is the reason the whole problem stays hidden. B = 999 satisfies the condition at α = 0.05, at α = 0.01 and at α = 0.10 simultaneously, because 1000 is a multiple of 20, 100 and 10. The one number everybody writes happens to be safe at every level anybody tests at, and there is no reason to think that was designed.

This figure also has a small history worth recording, because it is the parameterisation rule working. The generator was written with a fixed sweep of B up to sixty, which is fine at 5% and 10%; asked for α = 0.01 by the paragraph above, its own assertion failed, because a picture of a claim about meeting points with no meeting points in it is not a picture of that claim. The range now follows the level. The defect existed because a generator had only ever been run at one point of its own argument space, which is precisely what running figures at the essay’s numbers exists to prevent.

The simulation agrees, which is the point of running it #

The closed form above assumes the count is uniform on {0 … B}, which is the exchangeability the test is built on. That assumption can be checked rather than believed, which is this site’s habit.

Five hundred adaptive trials with no effect in either arm, analysed at each of the two values:

• at B = 19, the corrected rule rejects 4.40% and the uncorrected rule rejects 4.40% — the same trials, the same seeds, and the same answer to the last trial, because they are the same rule;
• at B = 20, the corrected rule rejects 3.80% and the uncorrected rule rejects 8.00%.

The simulated numbers are not the closed form’s 5.00%, 4.76% and 9.52%, and they should not be: five hundred trials has a standard error of about one percentage point, and the reference distribution has ties in it — two allocations can produce exactly the same statistic — which makes the count slightly less than uniform and the test slightly conservative.

What matters is that the structure matches. The two rules are identical at B = 19 to the last trial, and at B = 20 the uncorrected one is roughly twice the size of the corrected one. Both facts are predicted by integer arithmetic that knows nothing about the trials.

Why the p-value’s coarseness is not an error #

The figure above raises the obvious objection: a p-value that can only be 0.05, 0.10, 0.15 … is a crude thing. Does the coarseness not cost something?

It costs resolution and it does not cost validity, and separating those two is the point.

The p-value at B = 19 is genuinely a random variable — run the same trial’s test again with a different stream and it moves. Over a hundred and twenty repeats of one trial its standard deviation is 0.099 at B = 19 and 0.038 at B = 199. That is a real wobble: a result reported as p = 0.10 might have come back as p = 0.05 or p = 0.20 from the same data.

And the size of the test at B = 19 is 5%, exactly, whatever that wobble does.

The two statements sit together because the randomness of the p-value is part of the reference distribution rather than an error in it. The test’s decision rule is “reject if the observed statistic is the most extreme of the twenty”, and that event has probability 1/20 under the null by construction. The p-value’s noise does not leak into the error rate; it shows up as a loss of power, which is what the last essay in this field prices.

This is the same distinction p-values must be flat is built on: the question about a p-value is not whether it is precise but whether its distribution under the null is the one claimed. A coarse p-value uniform on twenty lattice points is a correct p-value. A smooth p-value with the wrong distribution is not.

The exhaustive case, where none of this arises #

It is worth saying where the whole problem comes from, because it is not the +1 and it is not the level.

The design field’s randomisation test enumerates every possible assignment rather than sampling some of them. Twenty units split ten and ten gives 184,756 assignments, all of them are walked, and the p-value is the exact proportion of the reference set at least as extreme as the observed. There is no B, no sampling, no correction, and no lattice — the p-value is a rational number with a denominator of 184,756 and it is the p-value rather than an estimate of one.

Everything in this essay exists because the reference set here cannot be enumerated. Two hundred patients allocated one at a time gives 2²⁰⁰ possible allocations, and the adaptive rule assigns them wildly unequal probabilities besides, so the only available move is to sample the reference set by running the rule.

Sampling is what introduces the count k, the lattice of attainable p-values, and the question of whether the observed allocation belongs in its own reference set. The +1 is the answer to that last question and it is a consequence of sampling rather than a property of randomisation tests. The exhaustive version has no +1 because it needs none: the observed allocation is already in the enumeration.

What to actually do #

Short, and one item is not the obvious one.

Include the +1. It is free, it is right at every B, and it is the only version that does not need the practitioner to have thought about divisibility.

Choose B so that B + 1 is a multiple of 1/α. With the +1 included this changes nothing about validity and it makes the attainable levels land on the level being used: at B = 39 the test’s size is exactly 5%, and at B = 49 the largest attainable size is 4%, because the p-value’s lattice has no point at 0.05. A test claiming 5% that structurally cannot exceed 4% is throwing away a fifth of its rejection region for no reason.

That second item is why the figures in this field use B = 39 rather than a rounder-looking 50.

And do not read a p-value near the threshold at small B. The wobble at B = 19 is a tenth, so a result at 0.05 is one whose repeat could be anywhere. Increase B rather than interpret it — that is exactly what B buys.

There is a fourth item that is really a note about reporting. The number of re-randomisations belongs in the paper, for the same reason the allocation rule does: a p-value of 0.05 from B = 19 and a p-value of 0.05 from B = 999 are different objects, one of which is the coarsest possible non-trivial result and one of which is a fairly precise measurement. Both print identically. B is cheap to state, it is a single integer, and without it a reader cannot tell which of the two they are looking at — nor whether the value chosen happened to be one where the correction mattered.

The check, and the refusal that makes it mean something #

The field’s library asserts both halves of the finding, because either alone is misleading. At B = 20 the uncorrected rule’s size is more than one and a half times the corrected one’s; at B = 19 the two are equal to the last trial — asserted as an exact equality of two simulated rates rather than as a small difference, which is a claim that can only hold if the two rules have literally the same rejection region.

That exact-equality assertion is the one worth pointing at. A tolerance-based version of it would have passed on any small difference and would have missed the actual structure, which is not “the correction is small here” but “the correction does not exist here”. The distinction is the essay.

The refusal is the closed form itself, checked against the simulation on the same settings. Two routes to one number, and the arithmetic route is the one that can say why — a simulation showing 4.40% and 4.40% at B = 19 is a coincidence until integer arithmetic explains that the rejection regions are identical.