What the correction corrects

Run twenty tests where nothing is happening. The chance that at least one comes out significant at 5% is 64%, by the closed form 1 − 0.95²⁰ and by counting six thousand families of twenty.

The quantity being controlled #

Before any procedure, the thing it promises to hold has to be named, because this field’s central confusion is that different procedures promise different things.

The familywise error rate is the probability that a family of tests produces at least one false positive. Not the expected number of them, and not the proportion of the findings that are false: the chance of any at all.

That is a demanding quantity. It treats one false positive among twenty findings as exactly as bad as one among one, and it is the right quantity when a single false claim is what does the damage — a safety signal, a regulatory conclusion, a claim that will be acted on individually.

At twenty tests and no correction it stands at 64%, which is the number this field exists to fix.

Bonferroni, and why it works #

Compare every p-value against α/m instead of α. With twenty tests, the threshold is 0.0025.

The justification is one line and requires nothing about the tests’ relationship to one another. The probability of a union of events is at most the sum of their probabilities, so with m tests each rejecting a true null with probability α/m, the chance of any rejection is at most m × α/m = α.

That the bound holds regardless of dependence is the procedure’s real virtue. Correlated tests, independent tests, tests with a structure nobody has worked out: the guarantee is the same. Very few results in this subject are that robust.

Measured across four thousand families of twenty true nulls, the familywise rate is 5.27%, against the promised 5%. The bound holds, and it holds with almost nothing to spare in this case — which is worth noting, because Bonferroni’s reputation is for being crushingly conservative and with independent tests at this size it is barely conservative at all.

How conservative Bonferroni actually is, exactly #

“Barely conservative at all” can be made exact here, because with independent tests the familywise rate has a closed form.

Twenty tests each rejecting a true null with probability 0.0025 produce at least one rejection with probability

$$1 - (1 - 0.0025)^{20} = 0.04881 .$$

So the true rate is 4.881% against a promised 5%, and the measured 5.27% on four thousand families sits 1.1 standard errors above it — the standard error at this rate being 0.34 points.

Bonferroni leaves 0.119 points of its 5% budget unused: 2.4% of the level.

And that does not get worse with more tests. As m grows, $1 - (1-\alpha/m)^m \to 1 - e^{-\alpha}$, which at α = 0.05 is 4.877%. So the unused budget is 0.123 points at any number of independent tests whatever — a quarter of one per cent of the level, forever.

Where the reputation comes from, then #

The same expression at the other extreme says why Bonferroni is nonetheless remembered as crushing.

If the twenty tests were perfectly correlated they would all reject together or not at all, so the familywise rate under Bonferroni would be 0.25% — a twentieth of the budget, with nineteen twentieths thrown away.

So Bonferroni’s conservatism runs from 2.4% of the level under independence to 95% of it under perfect dependence, and every criticism of it is a criticism of the second case. With independent tests, at any family size, there is essentially nothing on the table.

Which places what the alternatives can be worth. The most that can be recovered by assuming independence is the gap between Bonferroni’s threshold and Šidák’s: holding the familywise rate at exactly 5% needs a per-test threshold of $1 - 0.95^{1/20} = 0.002564$ against Bonferroni’s 0.0025 — 2.6% larger.

That is the whole of the independence gain, and it is why the improvement worth having is not a sharper bound on the same idea. It is Holm, which improves on Bonferroni without assuming anything about dependence at all, and therefore takes its gains from a different place entirely.

And what the 64% is #

For completeness, the number the field exists to fix is the same formula with no correction in it: $1 - 0.95^{20} = 0.6415$.

Twenty independent tests at five per cent produce at least one false positive 64% of the time, which is not a subtle effect and is not a statement about anybody’s judgement. It is what twenty independent chances at one in twenty comes to.

Holm, which is strictly better #

The improvement is small to describe and it dominates Bonferroni everywhere, which makes the continued use of plain Bonferroni difficult to defend.

Sort the p-values ascending. Compare the smallest with α/m. If it passes, compare the next with α/(m − 1), then α/(m − 2), and so on, stopping at the first failure.

The first comparison is identical to Bonferroni’s. Every subsequent one is less strict, because the denominator shrinks. So Holm rejects everything Bonferroni rejects and sometimes more, and it controls the same familywise rate — measured at 5.27% on the same families, indistinguishable from Bonferroni’s.

The measured power, on twenty tests of which some are real:

Never worse, and better whenever more than one effect is real — by six percentage points when most of them are. The site’s gate asserts the domination directly at four configurations rather than describing it, because “uniformly at least as powerful” is the kind of claim that should be checked rather than repeated.

Why the closed form matters here #

A methodological aside, because this field could easily have been built entirely on simulation.

The 64% is available exactly: 1 − 0.95²⁰ = 64.15%, on the assumption that the tests are independent. The counted value across six thousand families is 64.28%. They agree, and the agreement is asserted at every number of tests from one to eighty rather than at one point.

That check is doing more work than it appears. It verifies that the simulated families really are independent, that the p-values really are uniform under the null, and that the counting is correct — three things that would each produce a plausible wrong answer on their own. A simulation matching a closed form at ten different family sizes is evidence about the machinery, not just about the number.

It also sets up the more interesting case. When the closed form and the count disagree, the tests are not independent, and the size of the disagreement measures how much. That is exactly what happens in twenty analyses of nothing, where correlated analyses give 57% against the independent 64% — and the gap is informative rather than an error.

What “a family” means, which is the unsolved part #

Every procedure here corrects for m tests, and m is not determined by the data. It is a judgement about which tests belong together, and the judgement changes the answer.

A study measuring twenty outcomes could treat all twenty as one family, or treat the primary outcome separately from nineteen secondary ones, or group them by domain into four families of five. Each is defensible, each gives a different threshold, and the choice can be made after seeing which grouping produces the desired conclusion.

Nothing in the mathematics constrains this. The procedures take m as an input and control the rate for whatever family is declared.

Two things follow, and only the first is comfortable.

The family has to be declared in advance, in a protocol, before the p-values are known. This is what pre-registration is for, and it is the only mechanism that makes the correction meaningful rather than decorative.

And a correction applied to a family chosen afterwards controls nothing. It looks rigorous — a Bonferroni-adjusted threshold appears in the methods section — and the rate it guarantees is the rate for a family that was selected with the answers visible.

That is the forking-paths argument arriving inside the fix rather than the problem, and it is why this field’s procedures are necessary and not sufficient.

The case against correcting at all #

Worth stating fairly, because there is a real argument and it is not simply resistance to rigour.

The argument is that the familywise rate is the wrong quantity to control in exploratory work. A study measuring twenty outcomes to find out which are worth studying properly is not making twenty claims; it is generating candidates for follow-up. Holding the chance of any false candidate to 5% means missing most of the real ones — the power table above shows Holm finding about half the genuine effects — and in a screening context that trade is backwards.

The counter-argument is that this is fine as long as the exploratory status is carried forward, and it never is. A candidate generated by an uncorrected screen appears in the literature as a finding.

Both positions are right about something, and the resolution is not a compromise threshold. It is that the two situations want different quantities controlled — which is the subject of the next essay, and the reason “correcting for multiple comparisons” is too vague a phrase to be actionable.

What to take from this essay alone #

Three things, before the field complicates them.

Twenty uncorrected tests of true nulls produce a false positive 64% of the time, and this is arithmetic rather than a warning. It is not a claim about bad practice; it is what the procedure does when followed correctly.

Bonferroni holds the rate at its stated level and requires no assumption about dependence, which is unusual and valuable.

And Holm holds the same rate while finding more, at every configuration tested, so the choice between them has one answer. Bonferroni survives because it is one line to describe and because it is the name everyone knows, not because there is a situation where it is preferable.

Why dependence makes the correction conservative #

The closed form assumes independence and the tests in real studies are not independent, so it is worth working out which direction that pushes and by how much.

Correlated tests give fewer independent chances to get lucky. In the limit where all twenty analyses are perfectly correlated they are one analysis and the familywise rate is 5% with no correction at all. In the limit where they share nothing it is 64%. Real families sit between.

So the uncorrected rate is lower than 64% whenever the tests are related — measured at 57% for the correlated family in twenty analyses of nothing — and Bonferroni, which assumes the worst case, is correspondingly conservative.

The size of that conservatism tracks how alike the tests are. Twenty slight variations of one model are nearly one test, and dividing the threshold by twenty is a severe over-correction. Twenty genuinely distinct outcomes are close to independent, and the correction is about right.

Two practical consequences.

Bonferroni’s conservatism is largest exactly where families are most often declared — a set of related outcomes on the same subjects — which is why it has a reputation for being punishing.

And there is no easy fix, because the correlation is rarely estimable from what gets reported. Procedures exist that exploit a known dependence structure and they need the structure. Where it is unknown, the worst-case bound is what is available, and being conservative in an unknown direction is the safe error.

This is the same trade as everywhere else on this site: a guarantee that holds without assumptions costs something, and the cost is paid in power.

The one-in-twenty framing, and why it misleads #

A common way of dismissing the whole problem deserves an answer, because it sounds like a version of the correct argument.

The dismissal: “one test in twenty comes out significant by chance, everybody knows that, and a competent reader discounts accordingly.”

What is right about it: the per-test rate really is 5%, and the expected number of false positives among twenty true nulls really is one.

What is wrong about it: the expected number is not the quantity a reader needs. A study reports its significant findings, not its twenty tests. Seeing a paper with one significant result out of twenty performed is entirely different from seeing a paper reporting one significant result — and the second is what appears, because the twenty are usually not described.

The 64% is the number that matters for reading a report, and it is not the same as “about one in twenty”. It says: a study like this, with nothing going on, produces something to write about two times in three.

That reframing is the value of computing the familywise rate rather than the expected count. Both are correct; only one answers the question a reader has.

Where this fits against the forking-paths essay #

The two are the same phenomenon and they are worth distinguishing carefully, because the available remedies differ.

Twenty analyses of nothing is about analyses that were available and not reported. The number of them is unknown, the correlation among them is unknown, and no correction can be applied because the essential input is missing.

This essay is about tests that were declared. The number is known, it is in the protocol, and the correction is a division.

So the fixable case is the one where somebody wrote down in advance what they were going to test. That is not a coincidence and it is the strongest practical argument for pre-registration in this whole area: it converts an unfixable problem into an arithmetical one.

Where nothing was declared, the procedures here cannot be rescued by applying them afterwards. A correction for the twenty tests that were run does not address the two hundred that could have been.

What the field measures next #

This essay has treated the familywise rate as the quantity to control, which is one choice among several, and the choice has consequences the power table already hinted at: Holm finds about half the real effects when half are real.

That is a large price, and whether it is worth paying depends on what the study is for. A confirmatory trial with one primary outcome should pay it. A screen of ten thousand genes cannot — holding the chance of any false positive to 5% across ten thousand tests leaves essentially no power for anything.

So the field’s next question is whether a different promise is available: one that permits more discoveries by accepting a controlled proportion of false ones rather than a controlled chance of any. That promise exists, it is a different quantity, and both are routinely called “correcting for multiple comparisons”.

A note on the arithmetic that makes Holm valid #

Holm’s step-down procedure controls the familywise rate, and the argument is short enough to give, because it explains why the thresholds rise in exactly the way they do.

Suppose there are m₀ true nulls among the m tests. For the procedure to make a false rejection, it must reject at some step where a true null is the smallest remaining p-value. At that step at most m₀ true nulls remain untested, so the threshold in force is at least α/m₀ — the denominator has come down by however many were rejected before, and those were all rejections of hypotheses ordered ahead of this one.

The chance that the smallest of m₀ true-null p-values falls below α/m₀ is at most m₀ × α/m₀ = α, by the same union bound Bonferroni uses.

So Holm gets its extra power from the observation that once k hypotheses have been rejected, at most m − k true nulls can remain, and the correction only needs to guard against those. Bonferroni guards against all m at every step, which is what it is throwing away.

The reason to give the argument is that it shows the improvement is free rather than a trade. Holm is not accepting a slightly worse guarantee for more power; the guarantee is identical, and Bonferroni was simply being stricter than its own logic required.

That is unusual. Almost every other choice in this field is a trade, and the next two essays are about trades. This one is not, which is why the recommendation can be unconditional.

Everything else in this field involves giving something up, and the essays that follow are about deciding what.

The summary #

Twenty tests of true nulls give a false positive 64% of the time, by closed form and by count. Bonferroni holds that at 5% under any dependence structure. Holm holds the same rate with strictly more power, so Bonferroni has no remaining use case.

And the input that makes any of it meaningful — the number of tests in the family — is a declaration rather than a measurement, which means the correction is only as honest as the protocol it was written into. A division applied to a family chosen after the p-values were seen guarantees nothing, and looks identical in the methods section to one that guarantees everything it claims.