Normal Distribution
nothing is called 95% until it has been counted
Concept

Covariate-adaptive randomisation — where it appears

An allocation rule that decides each arrival's arm from the baseline covariates already enrolled, so as to keep the arms alike on them. Because the assignment then depends on the data, the analysis has to condition on the rule that produced it, which is a randomisation test rather than a t test.

Named by 20 essays across 5 fields — each of them below, with the objects they name alongside it.

What a median split can see. A standard normal covariate with its median marked, and the mean of each category as a vertical rule: -0.7979, 0.7979. A rule that balances the categories is balancing those numbers and nothing else, so the part of the covariate it can act on is the variance between them — 0.6366 of the total, which at two categories is exactly 2/π because the two half-normal means are ±√(2/π). The rest, 0.3634, is variation inside the categories that the rule cannot see and does not touch: the assignment within a category is still a coin. Everything the next figure measures is a consequence of this one, and it is available before any unit has arrived.

A covariate with no levels

Every balancing rule on this site reads a level. Age and blood pressure have none, so somebody cuts them into categories — and a median split can see exactly 2/π of a normal covariate, whatever the rule does with the halves.

continuous · Assignment What each rule leaves behind, at 120 patients. Four allocation rules over the same cohorts and the same seeds, each scored on three imbalances: the number of patients in each arm, the worst of the nine factor levels, and the worst of the 24 cells of the cross-classification. No rule holds all three. Permuted blocks hold the totals exactly and leave the margins near a coin's. Blocks inside every cell hold the cells and let the totals drift, because 24 part-filled blocks do not have to end level. Minimisation holds the margins and the totals and is at 83% of a coin's cell imbalance. Each of the three columns is somebody's definition of a balanced trial.

Balancing what is known in advance

Four allocation rules, three definitions of balance, and no rule that holds more than one of them. Minimisation keeps the worst factor margin near three patients whether the trial has forty or six hundred and forty — and lets the imbalance in the cross-classified cells climb to 86% of a coin's, because the cells are not what it is watching.

covadapt · Assignment Where one rule becomes three. Every arrival in 200 simulated trials is put to all three scores, and the picture is how often they would send that patient to different arms. The range and the pairwise sum are the same rule at two arms and at three — for sorted counts the pairwise sum is twice the range, so the arm that minimises one minimises the other — and they part company at four, where the pairwise sum is 3(d − a) + (c − b) and the range still sees only d − a. The variance disagrees with both from two arms onwards, on 5.4% of arrivals at two and 27.0% at five, because the scores are summed over 3 factors and a sum of squares does not order the candidates the way a sum of absolute values does. All three are called minimisation.

Three arms and three scores

Minimisation balances a trial by keeping the arms' counts even inside every prognostic factor. With two arms there is one way to measure how uneven two counts are. With three there are several, they are all called minimisation, and they send different patients to different arms.

multiarm · Assignment A trial designed 2:1:1, and what two scores deliver. 500 trials of 180 patients, three arms, a target of 2:1:1. The shaded bars are a minimisation score that divides each arm's count by the share that arm is supposed to receive before measuring the spread; it delivers 49.9% : 25.1% : 25.1%. The others are the same rule with the counts left raw, which delivers 33.4% : 33.3% : 33.3% — the balance it enforces inside every factor level is equality, and equality is what it gets. The marks are the shares that were asked for.

Balancing towards unequal targets

A three-arm trial allocating two to one to one is the ordinary case, and a balancing rule built from raw counts does not know it. It balances the arms towards equality inside every factor level, delivers a third to each arm, and reports that it minimised imbalance.

multiarm · Allocation What a rule gives away by being predictable. Minimisation run at every probability from a coin to fully deterministic, over 500 cohorts of 120 at each. The upper line is the share of assignments an investigator who knows the rule and the enrolled patients can name in advance: 49.7% at p = 0.5, which is a coin and cannot be beaten, and 87.6% at p = 1 — short of everything only where the two arms tie and the rule falls back on a coin. The lower line is what that is worth: an investigator who enrols a patient 0.5 of a standard deviation better than average whenever they predict their favoured arm produces a treatment effect of 0.75 where the truth is zero. Nothing about the randomisation was broken; the bias entered through who was enrolled, which is the one thing an allocation rule cannot control. The dashed line is the closed form 2δ(2g − 1).

The rule that can be guessed

A balancing rule improves as it becomes more deterministic, and a deterministic rule can be worked out in advance from information the person enrolling the patient already has. At full determinism 87.6% of assignments are guessable, and an investigator who acts on the guess produces a treatment effect of three quarters of a standard deviation where the truth is zero.

covadapt · Assignment Two rates, not a factor. The standard deviation of the covariate imbalance under three rules, at five trial sizes, 260 trials each, on log axes. The upper line is a coin: its slope is -0.489, against a closed form of exactly −½. The middle line is minimisation on a median split; its slope is -0.519 — the same rate — because inside a category the assignment is still a coin, and what it buys is the constant, 0.654 of a coin's at n = 200. The lower line is the rule that reads x and maximises the information about the treatment effect: slope -0.987, nearly twice as steep. Its advantage is therefore not a number that can be quoted — it is 0.258 of a coin's at n = 50 and 0.065 at n = 800, and it keeps going.

The rule that reads the number

Stop categorising and let the rule read the covariate itself. What it should minimise is not an invented distance but the variance of the effect being estimated — and what comes back is not a better constant but a different rate.

continuous · Assignment Four analyses of the same 3-arm trials, under a true null. 250 trials of 150 patients, 3 arms, minimisation with p = 0.85, 99 re-randomisations for each exact test. Two statistics — an F on the arms alone and an F on the arms after the balanced factors — against two reference distributions: the table the statistic is named for, and the distribution the allocation rule itself generates when the outcomes are held fixed and the rule is re-run. Only the first cell is wrong, and it is wrong in the direction that costs power rather than the one that manufactures findings: 0.0% where 5% is claimed. Either repair works — adjusting for what the rule balanced, or asking the rule what it would have done.

The analysis after three arms

An unadjusted analysis after a two-arm balancing rule rejects 0.6% of true nulls where it claims 5%. With three arms and a deterministic rule it rejects none at all — and the repair is the same repair, which is a sentence and a column in the model.

multiarm · Assignment Four analyses of the same trials, with no treatment effect at all. 700 trials of 120 patients allocated by minimisation at p = 0.8, with the prognostic factors carrying a real effect on the outcome and no treatment effect — every rejection below is a false one. Two statistics, the plain difference and the same after adjusting for the balanced factors, each read against two reference distributions: a t table, and the set of allocations the rule could have produced from these covariates. The unadjusted comparison rejects 0.6% where it claims 5% — conservative, which is a loss of power rather than an error, and nothing on the output says so. Adjusting puts it back at 5.4%. Both re-randomised versions are at their nominal level by construction, whatever statistic goes into them.

The analysis has to know the rule

A trial balanced by minimisation and analysed by comparing the two arms' means rejects a true null 0.6% of the time where it claims 5%, and at full determinism 0.0%. That is not an error anybody complains about — it is a test that has stopped working, paid for by a balance the analysis then refused to use.

covadapt · Assignment Three analyses of the same trials, none of them wrong about the data. 320 trials at n = 60 with no treatment effect at all, so every rejection counted is a false one, and a covariate that drives the outcome with coefficient 1. The unadjusted comparison is at 5.94% after a coin — its level — and at 0.00% after the rule that reads the covariate: the design removed the imbalance and the analysis is still pricing it. Adjusting for the covariate gives 4.06%, and the rule's own reference distribution — hold the outcomes, re-run the rule 199 times, count — gives 3.13% against the 4.5% that 199 draws can deliver. The last of the three has to be told the assignment rule and nothing else, which is the one thing the experimenter certainly knows.

What the balanced trial is worth

A rule that reads the covariate removes three quarters of the imbalance. An analysis that does not know it happened prices the imbalance anyway, rejects one true null in two hundred instead of one in twenty, and finds a real effect less often than a coin-tossed trial does.

continuous · Randomisation What balancing several numbers at once costs each of them. The criterion generalises without a word changing — the covariate imbalance becomes a vector and the correction a quadratic form — so the question is what it is worth rather than whether it can be done. At n = 200 with 200 trials per point, a rule balancing one covariate leaves 12.7% of a coin's imbalance in it; balancing eight leaves 23.2% in each. The assignment has a fixed amount of freedom and every covariate added takes a share of it. The rule degrades rather than failing: at eight covariates it is still four times better balanced than a coin, and the eight are being held simultaneously rather than in turn.

Balancing more than one number

The criterion generalises to several covariates without a word changing, which makes the question what it is worth rather than whether it can be done. Each one added takes a share of the assignment's freedom, and the imbalance left in every one of them rises.

continuous · Blocking What a guesser gets, and what a guesser gets for nothing. 600 trials of 150 patients under a fully deterministic rule, with an investigator who knows the rule, the factors and every assignment so far. The guess rate falls with the number of arms — 87.8% at 2, 86.2% at 3, 81.0% at 4 — which reads like a trial getting safer and is not: what a guesser can trade on is the excess over the 50%, 33%, 25% they would get by naming an arm at random, and that goes the other way, from 1.76× chance at two arms to 3.24× at 4. The gap a guesser manufactures between the best and worst arm under a true null is 0.759, 0.733, 0.711 standard deviations — nearly unchanged.

Guessing one arm in three

A balancing rule is guessable because it is balancing. With three arms the next assignment is worked out less often than with two — and by more, relative to what a guesser gets for nothing, and the damage they can do is almost unchanged.

multiarm · Assignment The allocations this trial could have made, and the ones it could not. One 120-patient trial allocated by minimisation at p = 1, re-randomised 399 times. No outcome is redrawn anywhere in this figure: each re-randomisation runs the rule again over the same patients in the same order with the same recorded factors, so what is drawn is the set of experiments that could have happened. The bars are that set; the outline is what shuffling the labels gives, which is the reference distribution of a coin and is what every off-the-shelf permutation routine assumes. The coin's is wider — its 5% point is 1.95 against the rule's 1.09 — because a coin's allocations are less balanced and a less balanced allocation gives a larger statistic. Reading this trial against it makes the test conservative rather than anti-conservative, which is the opposite error from the outcome-adaptive case and for the same structural reason.

The reference the covariates supply

Hold the outcomes fixed, re-run the rule that assigned them, count. The same construction cost nineteen points of power in the adaptive field, because its rule chased outcomes and its critical value depended on a rate nobody has. Here the rule reads only what was recorded before anything happened, and the same unadjusted statistic goes from 20.3% power to 55.0% by being read against the right distribution.

covadapt · Assignment What balancing a recorded number does to the true one. The imbalance three rules leave in the true covariate X when what they read is a recording W of it, at six reliabilities and n = 200, 600 trials each, as a share of a coin's exact 2/√n. The lower curve is √(1 − λ), the closed-form floor for any rule that reads W: X = λW plus a residual of variance 1 − λ that W says nothing about, and a coin is all any rule can be on that residual. The rule that reads the number sits on it — 0.125 at 1, 0.345 at 0.9, 0.452 at 0.8, 0.612 at 0.64, 0.718 at 0.5, 0.834 at 0.3 — while balancing W itself to 0.125–0.133 of a coin's throughout. The upper curve is √(1 − 2λ/π), the same floor for minimisation on a median split of W, and minimisation sits on that one. At λ = 2/π the rule that reads the number is exactly as good as a median split of an error-free covariate: a cut is a measurement with reliability 0.637.

A number read with error

A rule that balances the recorded covariate balances the true one only as far as the recording reaches. At a reliability of 0.8 it leaves 45% of a coin's imbalance in the covariate that drives the outcome, back on the coin's 1/√n rate — and a median split turns out to be a recording with reliability 2/π.

continuous · Assignment What a guesser gets from less than everything. 1,000 three-arm trials of 150 patients under minimisation on three factors with the preferred arm taken with probability 1, and five investigators guessing each next assignment from different information. Knowing the rule, the factors and every assignment so far: 86.3%. Knowing everything about the patients at their own centre and nothing about the others, and applying the rule's own score to that: 52.8%. Knowing only how many of their own centre's patients went to each arm: 55.5%. Knowing only the trial's arm totals, which a drug-supply system shows anyone who reorders: 54.5%. A guess at random gets 33.3%. The cruder the information, the less the guesser gets — except that the rule's own score, applied to a partial record, does worse than simply counting.

A guesser who sees the totals

The bound on how guessable a three-arm minimisation is assumes an investigator who sees everything. One who sees only the arm totals — what a drug-supply screen shows — names the next arm 54.5% of the time against a chance 33.3%, and manufactures 45% of the full guesser's bias.

multiarm · Assignment Four intervals for the effect, minimisation at p = 0.8, 120 patients. Under minimisation at p = 0.8, 120 patients, over 600 trials with an additive effect of 0.45 and 199 re-run allocations per trial, the average width of four 95% intervals for the effect, and how often each covers it: unadjusted t interval 1.229 wide, covering 99.7%; unadjusted, test inverted 0.816 wide, covering 95.3%; adjusted t interval 0.723 wide, covering 96.0%; adjusted, test inverted 0.722 wide, covering 96.2%.

The interval the test inverts to

A randomisation test of no effect becomes an interval for the effect by testing every value at once: subtract the hypothesised effect from the treated outcomes and ask the same question again. Because the rule reads only covariates, one set of re-run allocations answers every value. Under minimisation the unadjusted t interval covers 99.7% at a width of 1.229; the inverted test of the same unadjusted difference covers 95.3% at 0.816, a third narrower, with no model of the outcome. Under complete randomisation the two are the same interval — which is what says the third was the rule's.

covadapt · Assignment Sixty-three subsets of six functions. The largest share of a coin's imbalance left in any of four shapes, measured on fresh trials, for each of the 63 subsets of six functions at 100 units. Subsets of one size spread widely: at four functions from 12.1% to 72.9%. The designer's order of addition gives 1: 94.4%, 2: 41.2%, 3: 39.0%, 4: 31.5%, 5: 27.0%, 6: 22.7%, against the best chosen subset of each size at 1: 83.6%, 2: 41.2%, 3: 25.4%, 4: 12.1%, 5: 16.2%, 6: 22.7%.

The list it was chosen on

Six functions of one covariate looked like 720 designs, one per order of addition. The rule reads a span and not a sequence, so they are 63, and at a hundred units the best four leave 12.1% of a coin's imbalance against the designer's first four at 31.5%. Those best four contain no cube and are, function for function, the four shapes the worst case was scored on. Read on six shapes it was not told about, the same design leaves the cube 77.0% of a coin's imbalance, and the designer's four, which lost by a factor of two and a half, leave it 28.1%.

shape · Criterion Four rules and what each leaves. The share of a coin's imbalance each rule leaves in the true value of each covariate, at 200 units and a reliability of 0.8. one reading of X: 21.4% in X and 100.1% in Z; two readings of X: 12.3% in X and 105.0% in Z; one reading each of X and Z: 21.7% in X and 21.5% in Z; two of X and one of Z: 13.4% in X and 22.0% in Z. The floors in closed form are 20.0% for one reading and 11.1% for two.

A second reading or a second covariate

A rule that balances a recording balances the truth only as far as the recording reaches, and a second reading of the covariate raises that reach. So does a first reading of something else. At a reliability of 0.8 the first reading of a second covariate does more once that covariate matters a third as much as the first, and the threshold is exact: the square root of (1 − λ)/(1 + λ), with no trial size in it. A protocol's list of measurements turns out to be a small knapsack, and it is rarely filled by reading anything twice.

continuous · Assignment What the interval covers as the effect stops being one number. Coverage at 120 units under minimisation, 600 trials and 199 re-run allocations each, as the effect's slope on the prognosis rises from 0 to 1.5. the inverted test, unadjusted, for the population's mean: 96.7%, 96.2%, 95.7%, 95.7%, 94.8%. the same, for the sample's mean: 96.7%, 96.0%, 96.2%, 97.3%, 98.3%. the inverted test, adjusted, for the population's mean: 96.5%, 95.5%, 95.0%, 95.3%, 96.0%. the same, for the sample's mean: 96.5%, 95.7%, 96.5%, 98.0%, 99.5%. the regression with an interaction, for the population's mean: 96.2%, 95.5%, 94.0%, 91.0%, 84.0%. the same, for the sample's mean: 96.2%, 95.8%, 96.2%, 95.7%, 95.2%.

An effect that is not one number

A randomisation test becomes an interval for the effect by testing every constant effect in turn, so when the effect grows with the prognosis every hypothesis it tests is false. It does not fall apart. It covers the population's mean effect within a point of its level at every slope measured, and its width grows by exactly what the sample's own composition adds. A regression that models the interaction stays narrow at 0.72, covers the sample's mean at 95%, and covers the population's at 84%. The extra width was never a defect. It was the difference between two questions.

covadapt · Assignment The guess rate through a trial. Guess rates in each tenth of a trial of 150 patients on three arms with the rule taking its preferred arm with probability 0.8, over 1500 trials. knowing everything: 65.9%, 72.2%, 72.0%, 72.8%, 72.4%, 72.2%, 72.5%, 72.7%, 72.7%, 72.4%. a joint table learned in earlier trials: 53.4%, 55.4%, 54.8%, 55.1%, 54.3%, 53.8%, 54.9%, 54.8%, 54.1%, 54.8%. the centre's totals: 45.4%, 50.3%, 50.2%, 50.8%, 49.8%, 50.2%, 50.9%, 50.6%, 50.4%, 50.7%. following whichever has been right: 46.5%, 50.2%, 49.3%, 49.6%, 48.9%, 48.7%, 49.3%, 49.0%, 49.0%, 48.7%. a joint table learned in this trial: 36.4%, 43.8%, 45.0%, 46.2%, 46.7%, 46.8%, 48.1%, 48.5%, 47.9%, 48.9%. the centre's own factors, scored: 46.8%, 51.4%, 49.7%, 49.9%, 48.4%, 48.6%, 48.5%, 47.7%, 47.6%, 46.7%. Chance is 33.3%.

The guesser who learns

An investigator who watches a three-arm minimisation can tally which of several partial readings has been right so far and follow the leader. It does not help. Following the leader names 48.9% of assignments where the best single reading names 49.9%, and a table of what the three readings say together, learned inside the trial, starts at 36% and has not caught up by a hundred and fifty patients. Learned over earlier trials, that table reaches 54.5%. The full guesser's 71.8% stays out of reach of all of them, because the gap to it is information rather than practice.

multiarm · Assignment The two shapes a scale decides between. The share of a coin's imbalance left at 400 units, by marginal and by what the rule reads. normal: linear 1.1% on values and 5.7% on ranks; split 37.4% on values and 25.4% on ranks. skew 2.26: linear 1.5% on values and 28.6% on ranks; split 61.9% on values and 25.0% on ranks. skew 4.75: linear 2.5% on values and 52.2% on ranks; split 95.5% on values and 25.1% on ranks. exponential: linear 1.4% on values and 28.1% on ranks; split 58.7% on values and 25.0% on ranks. A rank's hold on the split does not move with the marginal.

The covariate on its ranks

A skewed covariate spoils every parity argument a balancing rule relies on, and a rule that balances ranks instead of values looks like the way out, since a rank has the same uniform law on any marginal. The measurement says the skew really does belong to the scale for one kind of shape: a rank holds a median split at exactly a quarter of a coin's imbalance whatever the covariate. But it pays with a cap on any shape linear in the value, at nearly half a coin's imbalance on the strongest skew. And a rule holding values and median splits together holds both, beating the rank on the shape a rank gives up and matching it on the one it gains.

skew · Criterion

Named alongside it

The objects these essays reach for when they reach for this one.

MinimisationMonte CarloRandomisationCovariate adjustmentCovariate balanceExperimental designPermuted blocksContinuous covariateAllocationRandomisation testReference distributionSelection bias

All concepts