The familywise error rate with no correction, α = 0.05
Two routes: the curve is 1 − (1 − α)^m and the points are counted over 6,000 simulated families of true nulls. With twenty tests the chance of at least one false positive is 64.1%.
Corrections, and what each controlsslider: nominal level, 4 positionswide8 views
What else it draws
The same object, drawn to answer the other questions the essays put to it.
no correction: familywise 40.8%, false discovery 5.3%, power 85%. Bonferroni: familywise 2.8%, false discovery 0.5%, power 49%. Holm: familywise 3.6%, false discovery 0.6%, power 53%. Benjamini–Hochberg: familywise 20.0%, false discovery 2.6%, power 75%.
no correction finds 85.1%, Bonferroni finds 49.1%, Holm finds 52.5%, Benjamini–Hochberg finds 74.9%. The uncorrected procedure finds the most and controls nothing.
Bonferroni is a flat line at α/m = 0.0025. Holm starts there and rises. Benjamini–Hochberg is the steepest line, iα/m. On this family they reject 5, 5 and 6 hypotheses respectively.
BH, every null true: 5.08% at 0, 4.86% at 0.3, 3.70% at 0.6, 2.34% at 0.9. BH, 10 of 20 real: 2.55% at 0, 2.53% at 0.3, 2.26% at 0.6, 1.66% at 0.9. BY, every null true: 1.46% at 0, 1.31% at 0.3, 1.03% at 0.6, 0.69% at 0.9. BY, 10 of 20 real: 0.72% at 0, 0.75% at 0.3, 0.64% at 0.6, 0.50% at 0.9. 20,000 families at each correlation.
The true share is 0.5. Independent tests: mean 0.610, spread 0.160, below half the truth in 0.92% of families. Correlated at 0.6: mean 0.609, spread 0.240, below half the truth in 9.33%.
Effects of three standard errors, ten real, familywise 5%. fixed sequence: 85.3% at position 1, 45.0% at 5, 20.4% at 10; overall power 46.10%; fallback: 49.1% at position 1, 56.4% at 5, 57.9% at 10; overall power 55.87%; Holm: 52.5% at position 1, 52.2% at 5, 52.5% at 10; overall power 52.53%.
85 findings, sorted by their estimate. 12 of their ordinary 95% intervals miss the true effect, every one of them on the far side; 6 of the wider false-coverage-rate intervals miss.
Where it is used
11 essays draw this figure, each at the numbers its own argument is about, so the same picture answers 11 different questions.
- What the correction corrects Corrections, and what each controls
- How many analyses there really were The analyses that were available and not run
- Two different promises Corrections, and what each controls
- What a p-value does not say Tests, and the second number
- One control, many arms Splitting the units
- The price of control Corrections, and what each controls
- Twenty analyses of nothing Tests, and the second number
- False discoveries that arrive together Corrections, and what each controls
- Estimating how many nulls are true Corrections, and what each controls
- An order that spends the error rate Corrections, and what each controls
- Intervals for the findings Corrections, and what each controls