One control, many arms

Add a third arm to a two-arm experiment and two things change at once, and they are usually treated as belonging to different subjects. The allocation question — how many units on the control, how many on each treatment — is a design problem. The multiplicity question — what critical value to use for three comparisons instead of one — is an analysis problem.

They are the same problem. Both are decided by how much of the experiment sits on the shared control, and neither can be settled without the other.

The control is worth √k arms #

With n₀ on the control and n on each of k treatment arms, each comparison has variance σ²(1/n + 1/n₀). The control’s error appears in every one of them; each treatment arm’s error appears in one. So the control is worth more per unit, and the amount has a clean answer:

n₀ = √k · n.

Enumerated over every integer control size at 360 units:

The rule is one of the few pieces of design advice that is both widely repeated and exactly right, and the penalty column is why it is worth following: at eight arms, splitting the units evenly across the nine groups costs 22.8% of the precision of every comparison, which is 22.8% more units to get it back.

The same decision sets the correlation #

Now the second half, and it is a consequence of the first rather than a separate topic.

The k comparisons share the control’s error. Cov(x̄ᵢ − x̄₀, x̄ⱼ − x̄₀) = σ²/n₀, and each has variance σ²(1/n + 1/n₀), so their correlation is

ρ = n / (n + n₀)

which is a fact about the design. At equal groups, ρ = ½. At the √k optimum, ρ = 1/(1 + √k): 0.41 at two arms, 0.33 at four, 0.26 at eight.

So enlarging the control does two things at once. It shrinks every comparison’s variance, and it makes the comparisons less alike — which turns out to decide which multiplicity correction is right.

What Bonferroni does not look at #

Bonferroni’s correction is α/k. It depends on how many comparisons there are and on nothing else: not on their correlation, not on the design, not on anything the experimenter could have chosen differently.

At eight arms against a control of the same size, twenty thousand trials with no effect anywhere:

Uncorrected, a quarter of these experiments report a finding when nothing is real. Bonferroni holds 3.95% — below its own claim, because it treats eight comparisons that share a control as eight unrelated ones — and Dunnett, which integrates over the control’s error, holds 4.90%.

The conservatism grows with the correlation, which is to say with the smallness of the control. Squeeze the control to a fifth of the experiment — four arms of sixty against a control of twenty, ρ = 0.75 — and Bonferroni holds 3.36% where Dunnett holds 4.94%. Enlarge it the other way, four arms of forty against a control of eighty, ρ = 0.33, and Bonferroni reaches 4.92% and the two corrections are nearly the same thing.

That is the sentence this essay exists for: the design decides how much the correction matters. An experiment allocated by the square-root rule has a low correlation and can use Bonferroni with almost no loss; an experiment that squeezed its control has a high one and cannot.

The correlation is also making the problem smaller #

Before the corrections, the uncorrected rate is worth looking at on its own, because it says how much multiplicity there actually is — and it is less than the standard arithmetic predicts.

If the k comparisons were independent, the chance that at least one of them fires would be 1 − 0.95ᵏ. They are not independent, they share a control, and at equal groups ρ = ½:

At eight arms the sharing is worth eight and a half percentage points of familywise error that never happens. The reason is direct: a control mean that came out low makes every comparison look positive at once, so the events pile up on the same experiments instead of spreading over more of them. The number of experiments that report something falls; the number of false claims within those experiments does not.

That is the same quantity Dunnett’s value is computed from, seen from the other side, and it is why the correction can be smaller than Bonferroni’s without controlling anything less.

Where Dunnett’s value comes from #

There is no closed form, and this site does not quote tables, so the value is solved for.

The k statistics are equicorrelated at ρ, and equicorrelation has a one-dimensional representation: with Z₀ the control’s error and Zᵢ each arm’s,

tᵢ = √ρ·(−Z₀) + √(1−ρ)·Zᵢ

so the probability that none of the k exceeds c is an integral over Z₀ of a product of k identical normal probabilities. One quadrature, one bisection on c, and the answer is deterministic — the figures do not move between builds and nothing was simulated.

Two cases pin it. With k = 1 there is no multiplicity and the value must be the ordinary 1.96: it comes out at 1.959964. With ρ → 0 the comparisons are independent and it must be the Šidák value 1 − (1 − α)^(1/k): at four arms that is 2.490915, and the solver gives it.

Those two checks are what make the third number believable, and the third number is the one that matters: at the correlation a real design produces, the counted familywise rate is 5.29% at ρ = 0.33 and 4.90% at ρ = 0.5, against a nominal 5% and a simulation error of about 0.15 points.

What the difference buys #

A smaller critical value is not an end in itself, and the case for Dunnett is the power it returns.

With one of eight arms carrying a real effect of half a standard deviation, the same twenty thousand trials find it 34.2% of the time under Dunnett and 31.0% under Bonferroni. Three points of power for nothing but using the correction the design earned.

Uncorrected, the same arm is found 60.8% of the time — and 25.2% of the experiments where nothing is real also report something, which is the trade the multiplicity field is about and is not a trade this essay is reopening.

What is worth noticing is the shape of the accounting. The design has three levers — the total, the control’s share, and the correction — and they interact:

More units on the control improves every comparison and lowers the correlation, which makes Bonferroni nearly right and Dunnett’s advantage small.

Fewer units on the control raises the correlation, which costs precision and makes the choice of correction worth more.

So an experiment that has allocated well has less to gain from the sophisticated correction, and an experiment that has allocated badly needs it most. That is not a coincidence: both are consequences of how much of the design is being shared.

What an extra arm costs the arms already there #

The allocation question above holds k fixed. The prior question — how many arms to run at all — has an answer in the same enumeration, and it is worth reading off because it is usually decided on other grounds entirely.

At 360 units, optimally allocated at each k, the variance of a single arm-versus-control comparison:

Going from a two-arm trial to an eight-arm one, at the same total, multiplies the variance of every comparison by 2.5 — before any multiplicity correction is applied. The correction then raises the critical value from 1.96 to 2.65 on top of that.

Both costs are real and only the second is usually discussed. An experiment that adds arms because “they are only extra arms” has paid twice: once in precision, which the design cannot avoid, and once in the boundary, which is the price of controlling the rate.

What it bought is the chance to find an effect in any of eight places rather than one, and whether that is a good trade depends on how many of the eight were worth trying. The arithmetic here does not answer that, and it does put a number on the cost side of it.

The whole table is one expression #

The enumeration was run over every integer control size, and the two columns it produced — the variance of a comparison, and what equal groups cost — are both a single closed form. Substituting n₀ = √k·n into a total of N = n₀ + kn gives n = N/(k + √k), and the variance of one arm-versus-control comparison becomes

σ²(1 + √k)² / N

At 360 units that is 4/360 = 0.01111 for one arm, 9/360 = 0.02500 for four and (1 + 2.828)²/360 = 0.04071 for eight — the enumerated column to every digit it prints, with the small residue at k = 8 being the rounding to whole units that the enumeration is obliged to do and the algebra is not.

Read as a rate, it says something the table only implies. The variance grows as (1 + √k)², which for large k is linear in the number of arms rather than quadratic: doubling the arms of a large experiment costs about a doubling of every comparison’s variance, and no more. The relative column is ((1 + √k)/2)², which is 3.66 at eight arms against the 3.67 counted.

The same substitution settles the penalty column. Equal groups put N/(k + 1) everywhere, for a variance of 2(k + 1)σ²/N, so the cost of ignoring the rule is

2(k + 1)/(1 + √k)² − 1

which gives 2.9% at two arms, 7.2% at three, 11.1% at four, 14.6% at five and 22.8% at eight. Five enumerated numbers, reproduced from an expression that never counted anything.

And the expression is bounded, which the table is too short to show. As k grows, 2(k + 1)/(1 + √k)² tends to 2: equal groups never cost more than a doubling of the variance, however many arms are added. That is a reassuring ceiling and a slow approach to it — the penalty is still only 22.8% at eight arms and reaches half its limit somewhere past thirty — so the rule is worth following and is never the difference between a study that works and one that does not.

The optimum lowers the correlation as it raises the arm count #

Two of this essay’s quantities move with k in opposite directions at the optimum, and putting them together says when the sophisticated correction is worth its trouble.

More arms means more comparisons, and Bonferroni’s conservatism grows with the number of them. But at the square-root allocation the correlation is ρ = 1/(1 + √k), which falls with k: 0.41 at two arms, 0.33 at four, 0.26 at eight. Bonferroni’s error is that it treats correlated comparisons as unrelated, and a well-allocated experiment is one whose comparisons become progressively less related as it grows.

So a design that follows the rule is partly repairing, by allocation, the defect the correction would otherwise have to handle. That is why the counted numbers land where they do: eight arms at equal groups sit at ρ = ½ and Bonferroni holds 3.95% against Dunnett’s 4.90%, while four arms at the optimum sit at ρ = 0.33 and the two rules are within a few hundredths of each other.

The correction to reach for is decided by the allocation, not by the number of arms. An experiment with eight arms and a properly sized control has less correlation to exploit than one with four arms and a squeezed one, and the second is where Dunnett earns its quadrature.

The all-pairs design is a different design #

The square-root rule optimises the k treatment-versus-control comparisons, and the moment an experiment also wants treatments compared with each other, the optimum moves.

The algebra is quick. A comparison between two treatment arms has variance σ²(1/nᵢ + 1/nⱼ), which does not involve the control at all — so a design optimising the average over all pairs puts nothing extra on the control and lands on equal groups, the design the square-root rule rejects.

An experiment that wants both gets an optimum in between, and where it lands depends on how the two families of comparisons are weighted. That weighting is a statement about what the experiment is for, which means the design cannot be settled by arithmetic until the purpose has been.

The practical form is blunter than it sounds. If the trial exists to find out whether any of these treatments beats the standard, the control is the pivot and deserves its √k. If it exists to rank the treatments against each other, the control is one arm among many and should be sized like one. Trials regularly claim both, allocate for neither, and then apply a correction chosen for a third structure — which is three decisions taken separately that are one decision.

Where the correlation is not a nuisance #

One consequence of the sharing is worth separating from the error-rate arithmetic, because it changes how a result should be read rather than how it should be tested.

Two arms that both beat the control are not two independent pieces of evidence. At ρ = ½ a low control mean lifts both comparisons together, so “two of the four arms were significant” is a weaker statement than it sounds — the second significance is partly the first one repeated.

The clean way to see it is that a shared control makes the pattern of results informative in a way independent comparisons would not be. All arms significant is the signature a low control produces. One arm significant and the rest flat is what a real effect in one arm produces. The two are distinguishable, and neither the familywise rate nor any per-comparison p-value says which happened.

Nothing here recommends a procedure for that. What it recommends is not throwing the information away: an experiment with a shared control should report the control mean and its standard error alongside the comparisons, because a reader who can see it can tell the two patterns apart and a reader with only p-values cannot.

What this does not settle #

Two things, and the first is the one that decides whether any of the above is the right calculation.

Which error rate is being controlled. Everything here is a familywise rate: the chance that any comparison is a false positive. A screening experiment with eight arms may want the false discovery rate instead, in which case the whole table is answering the wrong question — and Dunnett’s advantage over Bonferroni, three points of power, is small against the difference between the two promises.

And whether the comparisons are the ones that matter. The √k rule optimises the k treatment-versus-control comparisons and nothing else. An experiment that also wants to compare treatments with each other has k(k−1)/2 more contrasts, none of which involve the control, and the allocation that is optimal for one set is not optimal for the other — the all-pairs problem’s answer is equal groups, exactly the design the square-root rule rejects.

Which of those an experiment is doing is a question about its purpose rather than its arithmetic, and it has to be answered first. The design that follows is then decided, and the correction with it.

And the quantity nobody has, again #

Every number in this essay assumed the arms share one σ. They do not, in general — the first essay in this field is about what to do when they differ — and the two rules combine in the obvious way: each arm’s claim is its own σ, and the control’s is σ₀√k, because it appears in k comparisons rather than one.

Which means this design needs k + 1 spreads to be allocated properly, all of them unknown before the experiment. That is the standing problem of the field with more terms in it, and the same consolation applies: the loss is second order near the optimum, so the ratio has to be badly wrong before the design is.

There is one asymmetry worth carrying, though, and it favours doing the arithmetic. Under-sizing the control hurts every comparison at once and raises the correlation that makes the correction expensive; under-sizing one treatment arm hurts one comparison. The errors are not symmetric, so where the spreads are genuinely unknown, the control is the place to be generous.