Assignment — the series · Normal Distribution

What each rule leaves behind, at 120 patients. Four allocation rules over the same cohorts and the same seeds, each scored on three imbalances: the number of patients in each arm, the worst of the nine factor levels, and the worst of the 24 cells of the cross-classification. No rule holds all three. Permuted blocks hold the totals exactly and leave the margins near a coin's. Blocks inside every cell hold the cells and let the totals drift, because 24 part-filled blocks do not have to end level. Minimisation holds the margins and the totals and is at 83% of a coin's cell imbalance. Each of the three columns is somebody's definition of a balanced trial.

Balancing what is known in advance

Four allocation rules, three definitions of balance, and no rule that holds more than one of them. Minimisation keeps the worst factor margin near three patients whether the trial has forty or six hundred and forty — and lets the imbalance in the cross-classified cells climb to 86% of a coin's, because the cells are not what it is watching.

part 1 · covadapt

The rule that can be guessed

A balancing rule improves as it becomes more deterministic, and a deterministic rule can be worked out in advance from information the person enrolling the patient already has. At full determinism 87.6% of assignments are guessable, and an investigator who acts on the guess produces a treatment effect of three quarters of a standard deviation where the truth is zero.

part 2 · covadapt

Four analyses of the same trials, with no treatment effect at all. 700 trials of 120 patients allocated by minimisation at p = 0.8, with the prognostic factors carrying a real effect on the outcome and no treatment effect — every rejection below is a false one. Two statistics, the plain difference and the same after adjusting for the balanced factors, each read against two reference distributions: a t table, and the set of allocations the rule could have produced from these covariates. The unadjusted comparison rejects 0.6% where it claims 5% — conservative, which is a loss of power rather than an error, and nothing on the output says so. Adjusting puts it back at 5.4%. Both re-randomised versions are at their nominal level by construction, whatever statistic goes into them.

The analysis has to know the rule

A trial balanced by minimisation and analysed by comparing the two arms' means rejects a true null 0.6% of the time where it claims 5%, and at full determinism 0.0%. That is not an error anybody complains about — it is a test that has stopped working, paid for by a balance the analysis then refused to use.

part 3 · covadapt

The allocations this trial could have made, and the ones it could not. One 120-patient trial allocated by minimisation at p = 1, re-randomised 399 times. No outcome is redrawn anywhere in this figure: each re-randomisation runs the rule again over the same patients in the same order with the same recorded factors, so what is drawn is the set of experiments that could have happened. The bars are that set; the outline is what shuffling the labels gives, which is the reference distribution of a coin and is what every off-the-shelf permutation routine assumes. The coin's is wider — its 5% point is 1.95 against the rule's 1.09 — because a coin's allocations are less balanced and a less balanced allocation gives a larger statistic. Reading this trial against it makes the test conservative rather than anti-conservative, which is the opposite error from the outcome-adaptive case and for the same structural reason.

The reference the covariates supply

Hold the outcomes fixed, re-run the rule that assigned them, count. The same construction cost nineteen points of power in the adaptive field, because its rule chased outcomes and its critical value depended on a rate nobody has. Here the rule reads only what was recorded before anything happened, and the same unadjusted statistic goes from 20.3% power to 55.0% by being read against the right distribution.

part 4 · covadapt

Where one rule becomes three. Every arrival in 200 simulated trials is put to all three scores, and the picture is how often they would send that patient to different arms. The range and the pairwise sum are the same rule at two arms and at three — for sorted counts the pairwise sum is twice the range, so the arm that minimises one minimises the other — and they part company at four, where the pairwise sum is 3(d − a) + (c − b) and the range still sees only d − a. The variance disagrees with both from two arms onwards, on 5.4% of arrivals at two and 27.0% at five, because the scores are summed over 3 factors and a sum of squares does not order the candidates the way a sum of absolute values does. All three are called minimisation.

Three arms and three scores

Minimisation balances a trial by keeping the arms' counts even inside every prognostic factor. With two arms there is one way to measure how uneven two counts are. With three there are several, they are all called minimisation, and they send different patients to different arms.

part 5 · multiarm

Four intervals for the effect, minimisation at p = 0.8, 120 patients. Under minimisation at p = 0.8, 120 patients, over 600 trials with an additive effect of 0.45 and 199 re-run allocations per trial, the average width of four 95% intervals for the effect, and how often each covers it: unadjusted t interval 1.229 wide, covering 99.7%; unadjusted, test inverted 0.816 wide, covering 95.3%; adjusted t interval 0.723 wide, covering 96.0%; adjusted, test inverted 0.722 wide, covering 96.2%.

The interval the test inverts to

A randomisation test of no effect becomes an interval for the effect by testing every value at once: subtract the hypothesised effect from the treated outcomes and ask the same question again. Because the rule reads only covariates, one set of re-run allocations answers every value. Under minimisation the unadjusted t interval covers 99.7% at a width of 1.229; the inverted test of the same unadjusted difference covers 95.3% at 0.816, a third narrower, with no model of the outcome. Under complete randomisation the two are the same interval — which is what says the third was the rule's.

part 5 · covadapt

Four analyses of the same 3-arm trials, under a true null. 250 trials of 150 patients, 3 arms, minimisation with p = 0.85, 99 re-randomisations for each exact test. Two statistics — an F on the arms alone and an F on the arms after the balanced factors — against two reference distributions: the table the statistic is named for, and the distribution the allocation rule itself generates when the outcomes are held fixed and the rule is re-run. Only the first cell is wrong, and it is wrong in the direction that costs power rather than the one that manufactures findings: 0.0% where 5% is claimed. Either repair works — adjusting for what the rule balanced, or asking the rule what it would have done.

The analysis after three arms

An unadjusted analysis after a two-arm balancing rule rejects 0.6% of true nulls where it claims 5%. With three arms and a deterministic rule it rejects none at all — and the repair is the same repair, which is a sentence and a column in the model.

part 6 · multiarm

What the interval covers as the effect stops being one number. Coverage at 120 units under minimisation, 600 trials and 199 re-run allocations each, as the effect's slope on the prognosis rises from 0 to 1.5. the inverted test, unadjusted, for the population's mean: 96.7%, 96.2%, 95.7%, 95.7%, 94.8%. the same, for the sample's mean: 96.7%, 96.0%, 96.2%, 97.3%, 98.3%. the inverted test, adjusted, for the population's mean: 96.5%, 95.5%, 95.0%, 95.3%, 96.0%. the same, for the sample's mean: 96.5%, 95.7%, 96.5%, 98.0%, 99.5%. the regression with an interaction, for the population's mean: 96.2%, 95.5%, 94.0%, 91.0%, 84.0%. the same, for the sample's mean: 96.2%, 95.8%, 96.2%, 95.7%, 95.2%.

An effect that is not one number

A randomisation test becomes an interval for the effect by testing every constant effect in turn, so when the effect grows with the prognosis every hypothesis it tests is false. It does not fall apart. It covers the population's mean effect within a point of its level at every slope measured, and its width grows by exactly what the sample's own composition adds. A regression that models the interaction stays narrow at 0.72, covers the sample's mean at 95%, and covers the population's at 84%. The extra width was never a defect. It was the difference between two questions.

part 6 · covadapt

What a guesser gets, and what a guesser gets for nothing. 600 trials of 150 patients under a fully deterministic rule, with an investigator who knows the rule, the factors and every assignment so far. The guess rate falls with the number of arms — 87.8% at 2, 86.2% at 3, 81.0% at 4 — which reads like a trial getting safer and is not: what a guesser can trade on is the excess over the 50%, 33%, 25% they would get by naming an arm at random, and that goes the other way, from 1.76× chance at two arms to 3.24× at 4. The gap a guesser manufactures between the best and worst arm under a true null is 0.759, 0.733, 0.711 standard deviations — nearly unchanged.

Guessing one arm in three

A balancing rule is guessable because it is balancing. With three arms the next assignment is worked out less often than with two — and by more, relative to what a guesser gets for nothing, and the damage they can do is almost unchanged.

part 7 · multiarm

What a median split can see. A standard normal covariate with its median marked, and the mean of each category as a vertical rule: -0.7979, 0.7979. A rule that balances the categories is balancing those numbers and nothing else, so the part of the covariate it can act on is the variance between them — 0.6366 of the total, which at two categories is exactly 2/π because the two half-normal means are ±√(2/π). The rest, 0.3634, is variation inside the categories that the rule cannot see and does not touch: the assignment within a category is still a coin. Everything the next figure measures is a consequence of this one, and it is available before any unit has arrived.

A covariate with no levels

Every balancing rule on this site reads a level. Age and blood pressure have none, so somebody cuts them into categories — and a median split can see exactly 2/π of a normal covariate, whatever the rule does with the halves.

part 8 · continuous

What a guesser gets from less than everything. 1,000 three-arm trials of 150 patients under minimisation on three factors with the preferred arm taken with probability 1, and five investigators guessing each next assignment from different information. Knowing the rule, the factors and every assignment so far: 86.3%. Knowing everything about the patients at their own centre and nothing about the others, and applying the rule's own score to that: 52.8%. Knowing only how many of their own centre's patients went to each arm: 55.5%. Knowing only the trial's arm totals, which a drug-supply system shows anyone who reorders: 54.5%. A guess at random gets 33.3%. The cruder the information, the less the guesser gets — except that the rule's own score, applied to a partial record, does worse than simply counting.

A guesser who sees the totals

The bound on how guessable a three-arm minimisation is assumes an investigator who sees everything. One who sees only the arm totals — what a drug-supply screen shows — names the next arm 54.5% of the time against a chance 33.3%, and manufactures 45% of the full guesser's bias.

part 8 · multiarm

Two rates, not a factor. The standard deviation of the covariate imbalance under three rules, at five trial sizes, 260 trials each, on log axes. The upper line is a coin: its slope is -0.489, against a closed form of exactly −½. The middle line is minimisation on a median split; its slope is -0.519 — the same rate — because inside a category the assignment is still a coin, and what it buys is the constant, 0.654 of a coin's at n = 200. The lower line is the rule that reads x and maximises the information about the treatment effect: slope -0.987, nearly twice as steep. Its advantage is therefore not a number that can be quoted — it is 0.258 of a coin's at n = 50 and 0.065 at n = 800, and it keeps going.

The rule that reads the number

Stop categorising and let the rule read the covariate itself. What it should minimise is not an invented distance but the variance of the effect being estimated — and what comes back is not a better constant but a different rate.

part 9 · continuous

The guess rate through a trial. Guess rates in each tenth of a trial of 150 patients on three arms with the rule taking its preferred arm with probability 0.8, over 1500 trials. knowing everything: 65.9%, 72.2%, 72.0%, 72.8%, 72.4%, 72.2%, 72.5%, 72.7%, 72.7%, 72.4%. a joint table learned in earlier trials: 53.4%, 55.4%, 54.8%, 55.1%, 54.3%, 53.8%, 54.9%, 54.8%, 54.1%, 54.8%. the centre's totals: 45.4%, 50.3%, 50.2%, 50.8%, 49.8%, 50.2%, 50.9%, 50.6%, 50.4%, 50.7%. following whichever has been right: 46.5%, 50.2%, 49.3%, 49.6%, 48.9%, 48.7%, 49.3%, 49.0%, 49.0%, 48.7%. a joint table learned in this trial: 36.4%, 43.8%, 45.0%, 46.2%, 46.7%, 46.8%, 48.1%, 48.5%, 47.9%, 48.9%. the centre's own factors, scored: 46.8%, 51.4%, 49.7%, 49.9%, 48.4%, 48.6%, 48.5%, 47.7%, 47.6%, 46.7%. Chance is 33.3%.

The guesser who learns

An investigator who watches a three-arm minimisation can tally which of several partial readings has been right so far and follow the leader. It does not help. Following the leader names 48.9% of assignments where the best single reading names 49.9%, and a table of what the three readings say together, learned inside the trial, starts at 36% and has not caught up by a hundred and fifty patients. Learned over earlier trials, that table reaches 54.5%. The full guesser's 71.8% stays out of reach of all of them, because the gap to it is information rather than practice.

part 9 · multiarm

The crossing barely moves. Both methods' costs in one unit — assignments evaluated per usable draw — as the tolerance tightens. A hunt costs 1/p and rises without limit: from 2.22 at a tolerance of 1.2 to 357.14 at 0.18. A walk costs its autocorrelation time and barely moves. The two cross at a tolerance of 0.190 at one swap and 0.195 at eight — the whole family of proposal sizes crosses inside a band of about two hundredths, because where the crossing is, the large proposal has already lost its advantage. A multi-swap proposal is worth a factor of 5.65 in the regime where the walk should not be used at all.

Where the gain is, and where the decision is

A bigger proposal is worth a factor of six at a loose tolerance and nothing at a tight one. The tolerances where it helps are the ones where a hunt costs two evaluations a draw, and the crossing barely moves.

part 10 · blocks

Where the set stops being one set. How many of the 3,432 equal splits of fourteen units a balancing rule admits, as the tolerance tightens, with the number of components single swaps leave it in. The set falls from 886 to 84 assignments, and somewhere in that fall it stops being connected: at 0.8 it is in 2 pieces and every assignment's complement is in the other one. Nothing about the rule changes at that point and nothing a chain reports changes either, which is the whole difficulty — the acceptance rate, the stationary distribution and the detailed balance are all in order on both sides of it.

Before the trial and after

The same diagnostic run at two moments answers two different questions. Before, a positive verdict changes the design. After, it changes which number gets reported — and only for the numbers the defect can reach.

part 10 · after

A number read with error

A rule that balances the recorded covariate balances the true one only as far as the recording reaches. At a reliability of 0.8 it leaves 45% of a coin's imbalance in the covariate that drives the outcome, back on the coin's 1/√n rate — and a median split turns out to be a recording with reliability 2/π.

part 10 · continuous

Whichever dial made the set thin, the crossing is at the same thinness. Each curve is one dictionary, swept over eight tolerances at two hundred units; a point above the line is a set thin enough that walking beats hunting. The curves lie nearly on top of one another, which is the answer to whether the crossing is a fact about the tolerance or about the thinness it produces: the crossings sit between one admissible assignment in 176 and one in 268 for dictionaries of 3 to 6 functions. The mechanism is that a hunt costs exactly 1/p and a walk costs almost the same everywhere — between 82 and 394 evaluations per usable draw across the whole table — so the crossing is wherever 1/p reaches a number that does not move.

The set a dictionary leaves

A rule constrained on six functions at a loose tolerance leaves a set as thin as one constrained on three at a tight one. Both sampling methods cross over at the same thinness, and the tolerance where that happens moves by a factor of three.

part 11 · dict

A second reading or a second covariate

A rule that balances a recording balances the truth only as far as the recording reaches, and a second reading of the covariate raises that reach. So does a first reading of something else. At a reliability of 0.8 the first reading of a second covariate does more once that covariate matters a third as much as the first, and the threshold is exact: the square root of (1 − λ)/(1 + λ), with no trial size in it. A protocol's list of measurements turns out to be a small knapsack, and it is rarely filled by reading anything twice.

part 11 · continuous