Randomisation is not balance

The standard justification for randomising an experiment is that it balances everything, including the things nobody thought to measure. The claim is half right in a way that matters, and the wrong half is checkable by enumeration rather than by argument.

What a single randomisation actually does #

Take sixteen units with a measured covariate — age, baseline severity, plot fertility, anything. Split them into two groups of eight. There are 12,870 ways to do it, and every one of them is equally likely under randomisation.

Enumerating all of them gives the exact distribution of the difference between the two halves’ means, in units of the covariate’s own standard deviation. It is centred on zero, which is the half of the claim that is right: on average, randomisation balances.

It is also spread out. 33.3% of those 12,870 assignments put the two halves more than half a standard deviation apart. One assignment in three, in a study of sixteen, produces exactly the kind of imbalance that a reader shown the baseline table would call a failed randomisation.

Nothing has gone wrong in those cases. They are randomisation working as specified. The average of a distribution is not a description of a draw from it, which is the same confusion as reading 95% coverage as a statement about the one interval in hand.

The spread is 2/√n, exactly, whatever the covariate #

The distribution has a property that is worth stating as an identity rather than as a rule of thumb.

The root-mean-square standardised imbalance across every possible assignment is exactly

2/√n

with no approximation and no dependence on the covariate’s own distribution. Normal, skewed, bimodal, a wildly skewed cube of an exponential: the covariate’s shape cancels against the sample standard deviation it is divided by, and the answer is the same to the last digit the arithmetic carries. The site’s gate checks it at machine precision on a deliberately grotesque covariate for that reason.

At sixteen units that is 0.5, which is why a half-standard-deviation imbalance is a one-sigma event and happens a third of the time. At a hundred units it is 0.2, at four hundred it is 0.1, and the imbalance shrinks as √n like everything else.

The practical reading of the identity: randomisation balances well in large studies and badly in small ones, and how badly is not a matter of luck but a computable number. A trial of twenty is going to have a visibly uneven baseline table and there is no procedure failure to look for.

What a third of assignments becomes at other sizes #

The 33.3% is a statement about sixteen units and a threshold of half a standard deviation, and both halves of that matter. Holding the threshold fixed and moving n says how fast the problem goes away.

The difference’s spread is $2/\sqrt{n}$, so half a standard deviation is $\sqrt{n}/4$ spreads out. The share of assignments beyond it is therefore

• sixteen units: 1.0 spreads out — one assignment in three
• sixty-four: 2.0 out — one in twenty-two
• a hundred: 2.5 out — one in eighty
• four hundred: 5.0 out — about one in two million

So a half-standard-deviation imbalance is the normal case at sixteen units and essentially impossible at four hundred, and the transition happens over a range of sample sizes a single field will contain.

The normal approximation is worth checking against the enumeration while both are available: it gives 31.7% where the exact count over all 12,870 splits gives 33.3%. A point and a half apart, and in the direction that flatters randomisation — which is a small enough discrepancy to make the approximation usable at the larger sizes where enumeration is not.

The imbalance a trial should expect to see #

Turned round, the same spread says what the ninety-fifth percentile of the imbalance is, which is the number a reader looking at one baseline table actually wants.

It is $1.96 \times 2/\sqrt{n} = 3.92/\sqrt{n}$ standard deviations: 0.98 at sixteen units, 0.39 at a hundred, 0.20 at four hundred.

One trial in twenty of sixteen units has its two arms a full standard deviation apart on any given covariate, and that is the design working exactly as specified. Anybody who would call that a failed randomisation should expect to call one trial in twenty failed, on every covariate they look at.

Which is why blocking exists #

If randomisation delivered balance, blocking would be pointless. The whole content of the previous essay is that arranging the allocation to balance a known factor removes the variance that factor contributes, and the figure above is the same fact from the other side: what is left to chance stays uneven, one assignment in three by half a standard deviation.

So the two techniques divide the work. Block on what is known and expected to matter — the imbalance in those factors becomes zero by construction. Randomise the rest, which leaves a known distribution of imbalance in every unmeasured factor rather than a controlled one.

That second half is the part that is genuinely irreplaceable, and it is not about balance at all.

What randomisation buys instead #

Randomisation makes the assignment the only random thing in the experiment, and that turns the analysis into a counting problem.

Under the null hypothesis that the treatment does nothing, each unit’s outcome is what it is regardless of which arm it landed in. So the difference actually observed is one of the differences the assignment could have produced — and the full set of those differences is computable, because the assignment mechanism is known exactly. It was chosen.

At ten units split five and five there are 252 possible assignments and therefore 252 possible differences. That set is the null distribution. Not an approximation to it, not a large-sample limit of it: the distribution itself, requiring no assumption about normality, symmetry, variance homogeneity or sample size.

A test whose size is counted rather than derived #

Rejecting the most extreme 5% of that set gives a test, and its size can be computed by enumeration rather than measured by simulation: for each of the 252 assignments, work out the p-value it would produce, and count how many are at or below 0.05.

The answer is 4.76% on the dataset in the figure, and the important part is what it can never be: above 5%. The size is at or below the nominal level on every dataset, always, by construction — the site’s gate checks two hundred of them and takes the largest.

It comes out below 5% rather than at it because the reference set is discrete. With 252 assignments the achievable p-values are multiples of 1/252, and the largest one not exceeding 0.05 leaves a little of the error rate unspent. That is conservatism of a very specific and benign kind: known in advance, computable, and shrinking as the study grows.

This is the same discreteness that makes coverage for a proportion oscillate rather than settle at its nominal level, and it has the same cause: a procedure with finitely many possible outcomes cannot spend an arbitrary error rate exactly. The difference is the direction. There the discreteness sometimes leaves the interval covering less than it claims; here it can only leave error rate unspent, because the reference set is the exact null distribution rather than an approximation to one.

Against the test everyone runs instead #

The t test on the same design and the same data is not exact, and the size of the gap depends on whether the data looks like what the t test assumes.

On skewed data — exponential, at five units per arm — the t test’s actual size is 3.91% against a nominal 5%. It is not catastrophic, and it is not 5% either, and there is no way to know from the output which direction the error runs in a particular case. The randomisation test on the same data is at or below 5% by construction and needs to know nothing about the shape.

Two things follow, and they point in opposite directions.

The t test is remarkably robust, which is why it survives. Being 3.92% instead of 5% on badly skewed data at five per arm is a good showing for an approximation stretched well past where it should apply.

And robustness is not exactness, so where the assumption is doubtful and the reference set is enumerable there is a test that assumes nothing. At ten units the enumeration takes microseconds; at forty it is sampled rather than enumerated, and the same guarantee holds up to the sampling error.

Where the enumeration stops being possible #

The reference set grows quickly and that changes the arithmetic without changing the argument.

Ten units give 252 assignments and sixteen give 12,870, both instant. Twenty give 184,756, still trivial. Forty units split twenty and twenty give about 138 billion, which is not enumerable, and a hundred units give a number with twenty-nine digits.

The response is not to abandon the method but to sample it: draw ten or a hundred thousand assignments at random from the set, and use those as the reference distribution. The test is then exact up to a simulation error that is itself computable — the same trade this site makes everywhere, with the difference that here the exact answer exists and is merely expensive.

The vocabulary distinguishes them. A permutation test enumerates; a Monte Carlo permutation test samples. Both are reported as randomisation tests, and for any study large enough for the distinction to matter the sampling error is far smaller than anything else in the analysis.

Restricted randomisation, and what it costs #

Between “block on it” and “leave it entirely to chance” there is a family of procedures that constrain the randomisation without fully determining it, and they are worth naming because they are what large trials actually use.

Blocked randomisation allocates within consecutive groups of four or six so the arms stay level in size throughout — which matters when recruitment stops early or is stopped early.

Stratified randomisation does the same within levels of a prognostic factor, which is blocking under another name.

Minimisation allocates each arriving unit to whichever arm currently makes the covariate totals more even, with a random element retained. It balances several factors at once far better than chance and is nearly deterministic in its later allocations.

Each buys balance and spends something for it, and the thing spent is the reference set. A procedure that constrains the assignment makes some assignments impossible, so the null distribution is over the assignments that remain — and an analysis that ignores the constraint is using the wrong reference set. In practice the difference is usually small and it is not zero, and the honest form is the one this whole field keeps arriving at: the analysis has to know how the allocation was done.

The rule is part of the result, again #

The randomisation test makes a point this site has made from the stopping-rule field and from forking paths: the reference distribution is determined by how the data was going to be collected, not by the numbers in the dataset.

Here that dependence is total. Change the allocation rule — block the design, allocate in a three-to-one ratio, use a biased coin — and the set of possible assignments changes, so the null distribution changes, so the p-value for the same observed numbers changes. Nothing about the outcomes moved.

That is uncomfortable stated abstractly and completely reasonable stated concretely: a result is surprising relative to what else could have happened, and what else could have happened is a fact about the design.

It also gives the sharpest available answer to the question of what randomisation is for. Randomisation is what makes “what else could have happened” a known set. In an observational study it is unknown, which is why every inference there requires assumptions that an experiment does not.

What the test estimates, which is not always what was wanted #

One limitation deserves stating, because the exactness above is easy to over-read.

The null hypothesis a randomisation test rejects is sharp: the treatment changed nothing for any unit. That is stronger than the hypothesis most analyses have in mind, which is that the treatment changed nothing on average. The two coincide when the effect is constant across units and come apart when it is not — a treatment that helps half the subjects and harms the other half by the same amount has no average effect and is not a treatment that did nothing.

So a significant randomisation test says the outcomes are not exchangeable between the arms, which may be because the average moved, or because the spread did, or because the treatment helped some units and hurt others. The test statistic decides which of those it is sensitive to, and a difference in means is sensitive mostly to the first.

This is not a defect specific to randomisation tests. It is the general fact that a test rejects a hypothesis rather than establishing an alternative, which is the whole subject of what a p-value does not say. It is worth restating here because exactness is a strong property and invites the reading that the answer is therefore complete.

What to do about an uneven baseline table #

The common practical question, with an answer that follows from the arithmetic above.

Do not test the baseline table for significance. A test of whether the arms differ on a covariate is testing a null hypothesis that is known to be true — the assignment was random, by construction — so any significant result is a false positive by definition, and running twenty such tests produces one about as often as not. This is the multiplicity argument from the corrections field applied to a null that cannot be false.

Do consider adjusting for a covariate that is prognostic, whether or not it looks imbalanced, and say in advance which ones. Adjustment for a strongly prognostic covariate increases precision in much the same way blocking does, and choosing which to adjust for after seeing the imbalance is a forking path.

Do prefer blocking next time. Every imbalance visible in a baseline table is one the design could have removed at no cost, and the reason it is there is that nobody blocked on it.

The one thing not to do is re-randomise until the table looks even without saying so. That is a constrained allocation procedure, it changes the set of assignments that were possible, and an analysis using the unconstrained reference set is then using the wrong one. Re-randomising is defensible when it is declared, specified in advance and carried into the analysis; it is a silent change to the design when it is not.

What this establishes #

Randomisation does not deliver balance. It leaves a known distribution of imbalance whose spread is exactly 2/√n, which at sixteen units means a third of assignments differ by more than half a standard deviation, and at small sizes means that an uneven baseline table is the expected outcome rather than a fault.

What it delivers instead is stronger and less often stated: the set of results the experiment could have produced, known exactly because the mechanism producing them was chosen. That set is a test with no distributional assumption in it, whose size is at or below its nominal level on every dataset, and which can be computed by counting rather than derived.

The two halves of the claim are worth separating one last time, because the popular version fuses them. Randomisation balances the covariates is false in any individual experiment and true only of the average across experiments nobody ran. Randomisation makes the analysis valid is true, and it is true for a reason that has nothing to do with balance: it is what makes the set of possible results knowable, and a p-value is a statement about that set.

An experiment can therefore be badly balanced and perfectly analysable, which is the case a third of the time at sixteen units. And an observational study can be beautifully balanced on everything measured and remain unanalysable in this sense, because nobody chose the mechanism and the set of things that could have happened is not available at any price.

The next essay turns from how units are allocated to what is varied, where the arithmetic is even more one-sided: changing every factor at once estimates each of them from every run, and changing them one at a time recommends a combination the experiment never tried.