Normal Distribution
nothing is called 95% until it has been counted
Series

Allocation — the series

12 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. Every split of 100 units, σ = 1 against 3. Each point is one integer split, with its variance computed exactly rather than simulated. The minimum is at 25:75, which is the ratio of the spreads 25:75, and equal allocation costs 25% more variance — the same as throwing away 20 of the 100 units. The shaded band is every split within 5% of the best, and it runs from 17% to 35%: sharp to state, flat to sit on.

    Not half and half

    The same units, the same measurements, the same analysis — and a different variance, decided before anything is measured. When the two arms have different spreads the best split is σ₁ : σ₂, equal allocation costs 2(σ₁²+σ₂²)/(σ₁+σ₂)², and at three to one that is a quarter of the experiment.

    part 1 · allocation
  2. A budget of 4,000, at 1 and 20 a unit. Every affordable pair, enumerated. The best is 280 cheap units and 186 expensive ones — a ratio of 1.51, against the σᵢ/√cᵢ rule's 1.49. The unit rule, which says buy in the ratio of the spreads, lands at 66:197 and costs 17% more variance for the same money. Both rules are right about their own constraint; only one of them was asked.

    The cost of a unit

    Change the constraint from units to money and the allocation rule changes with it — from σᵢ to σᵢ/√cᵢ, which can point the other way. An arm that is noisy and expensive gets fewer units than the same arm would if the money were not the thing running out.

    part 2 · allocation
  3. What a pilot buys, σ = 1 against 3. Each point is 6,000 two-stage experiments of 100 units: a pilot of m per arm, then the rest split by the pilot's own estimate of the two spreads. Above the line the pilot has made the experiment worse than not bothering. The best pilot here is 8 per arm at 0.809, against 0.800 for a designer who knew the spreads — so the rule recovers 96% of what knowing them is worth. A larger pilot estimates the ratio better and has less left to apply it to, which is why the curve turns.

    Allocating on a guess

    Every allocation rule in this field is a function of quantities the experiment is being run to find out. Fed a pilot's estimate of them, the rule that minimises the variance makes the experiment worse than not bothering — until the arms differ by about a factor of two, which is further than anyone would guess.

    part 3 · allocation
  4. A trial designed 2:1:1, and what two scores deliver. 500 trials of 180 patients, three arms, a target of 2:1:1. The shaded bars are a minimisation score that divides each arm's count by the share that arm is supposed to receive before measuring the spread; it delivers 49.9% : 25.1% : 25.1%. The others are the same rule with the counts left raw, which delivers 33.4% : 33.3% : 33.3% — the balance it enforces inside every factor level is equality, and equality is what it gets. The marks are the shares that were asked for.

    Balancing towards unequal targets

    A three-arm trial allocating two to one to one is the ordinary case, and a balancing rule built from raw counts does not know it. It balances the arms towards equality inside every factor level, delivers a third to each arm, and reports that it minimised imbalance.

    part 4 · multiarm
  5. Where the constraints exhaust the randomisation. At 16 units there are 12,870 equal splits, so the ones meeting a stated tolerance can be counted rather than estimated. With each of the first k standardised imbalances required to be within 0.4 of a coin's own spread, the admissible count runs 3874 → 1006 → 314 → 0 → 0 → 0 — and at 4 functions there is no admissible assignment at all. The count is the number of distinct answers a randomisation test can give: at 3 functions its finest attainable p-value is 1 in 314. Balance improves with every constraint and the reference distribution shrinks with it, and the two run out at different rates.

    When the constraints run out

    Every function added to a basis is a constraint the assignment has to satisfy with the same units. At sixteen units and a stated tolerance the admissible assignments run 3,874, then 1,006, then 314, then none — and the count is exact, because the assignment space is finite.

    part 5 · basis
  6. Free until the sums stop seeing what the differences see. Coverage with and without the block sums pooled into the interval's variance estimate. With one effect and one level they are free. With an effect that varies between blocks they are still free, because a block's sum picks that variation up exactly as its difference does. With a level that varies they make the interval 37% wider and conservative. And where the effect falls as the level rises — a ceiling, and not an exotic thing to suppose — the sums carry none of the between-block variation while the differences carry all of it, the pooled estimate is short, and the interval that uses it covers 88.75% on a width 20% narrower than the honest one.

    What a two-arm rule may not pool

    A spread computed "within the block" without the arm label carries a share of the effect, so the trial runs 173 observations at a null and 282 at an effect of 1.5. The stopping rule is reading the thing it exists to measure, and the phrase that produced it is one word long.

    part 6 · contrast
  7. The same number of assignments, held less tightly. For each basis size the box admits at a tolerance of 0.4, the Mahalanobis rule is given exactly the same number of admissible assignments — the ones with the smallest Mahalanobis distance — so the two differ only in the shape of the region. Each bar is the worst named function's remaining imbalance variance as a share of a coin's, with the mean over the named functions beside it. At one function the regions are the same interval: 0.0523 both ways. At 3 functions and 314 assignments, the box leaves 0.0771 in its worst function and the ellipse 0.1545; on average 0.0633 against 0.0927. The ellipse spends its assignments on a direction no function in the basis names.

    What the ellipse spends its assignments on

    Rerandomising on Mahalanobis distance never runs out of assignments, where a box of per-function tolerances does at four functions. But at the same number of admissible assignments it holds the worst named function half as tightly, because it spends its constraint on a combination of the basis nobody named.

    part 6 · basis
  8. The weights may not read the block they weight. A weighted least squares decomposition needs weights that are constants, or at least independent of the differences they multiply. One λ̂ pooled across the trial is estimated on hundreds of degrees of freedom and is effectively a constant; a λ̂ estimated inside each block is estimated on that block's own two or three, and is correlated with the difference it weights. Coverage falls from 94.68% to 82.76% — and the interval gets wider while doing it, 0.5163 against 0.3024, which is the signature of weights that are noise.

    The condition that cannot be dropped

    The weights may not read the block they weight. Estimate the variance ratio inside each block rather than across the trial and the coverage falls to 83% — on an interval that is at the same time seventy per cent wider.

    part 7 · corner
  9. Three regions at the box's own count. Across 24 sets of 16 units, on the sets where a box at a tolerance of 0.4 has any admissible assignment, the share of its own variance left in the worst of the k named functions at the box's count. the ellipse: 0.070, 0.134, 0.196, 0.194, 0.182. a sphere on the standardised functions: 0.062, 0.076, 0.091, 0.086, 0.080. the box: 0.062, 0.072, 0.081, 0.073, 0.069, at 2, 3, 4, 5, 6 functions, on 24, 20, 9, 4, 4 of the sets. The box's mean admissible count is 1045, 354, 115, 32, 13.

    A box with a count

    The ellipse never runs out of assignments and holds the named functions loosely; the box holds them tightly and can come up empty. Dropping the ellipse's thin directions was meant to give both, and it does not: no cutoff comes within a third of the box. But the box's emptiness was never a property of the box. It belongs to a tolerance fixed in advance. Set the tolerance by a count instead, and at that count the region is the box, on every set of units, with nothing left to run out.

    part 7 · basis
  10. What the guess is worth, when it is worth anything. The variance cost of an even split relative to the variance-minimising one for a risk difference, against the first arm's proportion, with the second at 0.3. The cost is a pure number: it does not depend on the trial's size. It is exactly zero at 0.3 and at 0.70, where the two arms have the same p(1 − p); it is 0.19% at a half and 4.36% at a tenth. Across the whole range from a tenth to nine tenths it never exceeds 4.36%, which is what the variance-minimising rule is worth here — and what it is worth is the reason it is safe to use with a guess.

    The arm whose variance is its answer

    With a binary outcome the allocation rule is a function of the proportions the trial exists to estimate. It costs at most 4.36% of variance to ignore it anywhere between a tenth and nine tenths, because √(p(1−p)) stays within a factor of two of its peak across 98% of the unit interval.

    part 8 · allocation
  11. Three contrasts on one dataset, three different splits. The variance-minimising allocation for each of three ways of reporting the same two-arm comparison, against the first arm's proportion, with the second at 0.1. A risk difference wants the arm with the larger p(1 − p) to get more units; a log odds ratio wants it to get fewer, and the two curves are exact reflections of each other in the half line. A log risk ratio wants something else again. At a first-arm proportion of 0.6 they ask for 62.0%, 21.4% and 38.0% of the units. A trial reporting more than one of them cannot be optimal for either.

    Two contrasts, one split

    A risk difference wants 62.0% of the units in the first arm, a log risk ratio wants 21.4% and a log odds ratio wants 38.0% — on one dataset, with one pair of proportions. The difference's rule and the odds ratio's are exact reflections of each other, so no split can be near-optimal for both.

    part 9 · allocation
  12. How often reporting whichever contrast is most significant rejects when the arms do not differ, by the split. 20,000 trials a point. With both arms at 0.05, 0.1, 0.3: at shares 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7 the chosen contrast rejects 6.5%, 5.6%, 5.0%, 5.0%, 5.1%, 5.1%, 5.2%, 5.2%, 6.3%; 6.0%, 5.4%, 5.3%, 4.9%, 4.8%, 4.9%, 5.2%, 5.4%, 5.8%; 5.8%, 5.6%, 5.2%, 5.1%, 5.2%, 5.1%, 5.3%, 5.4%, 5.7%.

    A contrast chosen after the data

    A two-arm trial with a binary outcome that reports whichever of the risk difference, the log risk ratio and the log odds ratio comes out most significant has tested three statistics that ask one question. At an equal split the choice is no choice at all: the difference has the largest z in every one of twenty thousand trials, so the selected test is the difference's, at its 5%. At a 30–70 split the choice is real and costs a point and a half of false positives, 6.5% at the rarest outcome. Pre-registering the difference and the odds ratio as a pair with Bonferroni's correction — the cautious alternative — costs eight to twelve points of power at every split.

    part 10 · allocation

All series