The garden of forking paths — where it appears
Named by 10 essays across 4 fields — each of them below, with the objects they name alongside it.
How many analyses there really were
Bonferroni divides by twenty because twenty analyses were run. Twenty analyses of one dataset are worth 11.37 independent ones at a correlation of 0.6 and 2.58 at 0.95, and the threshold that controls exactly the same error rate is measurable rather than assumed.
What naming it in advance costs
Preregistration is argued for as free. Against an effect of two standard errors hiding in one of twenty analyses, naming the right one detects it 51.5% of the time and naming the wrong one detects it 4.7% of the time; correcting all twenty detects it 22.5% wherever it is. The two are worth the same when the chance of having named correctly is 38%.
An outcome cut in two
Replacing a measured outcome with whether it crossed a threshold keeps 63.7% of the information when the cut is at the mean, 34.2% at the top tenth and 13.1% two standard deviations out. A trial that needs 63 patients per arm on the measured outcome needs 102 cut at the mean and 185 cut at one and a half standard deviations. The responder rates that result read as a share of patients who respond — 50.0% against 69.1% — when every patient moved by the same amount; and a cut chosen after looking turns a 5% test into a 17.7% one.
Twenty analyses of nothing
Twenty honest, correct analyses of data with no effect in it find something significant 57% of the time. Nobody p-hacked, every individual p-value is right, and the reported one is the smallest of twenty.
Naming a handful in advance
A preregistration that names three of twenty analyses sits between naming one and naming none, and whether it beats both depends on one thing: how much of the analyst's belief the three hold. An analyst who is 50% sure of a first choice, and whose belief halves down the list, gets 34.8% expected power from naming three, 28.3% from naming one and 22.7% from exploring all twenty. Spread the same 50% evenly over the rest and naming three is the worst of the three plans. The rule for adding one more to the list needs no belief at all: the next analysis must be at least 20.8% as likely as the first to be worth naming.
An interval for the winner
The largest of twenty correlated analyses, reported because it cleared the family-wise threshold, carries an ordinary 95% interval that covers its true mean 75.96% of the time at an effect of two standard errors and never at all when there is no effect. An interval built on the fact that it won covers 94.90% — and it is 2.48 times as wide, excludes zero for only 14.10% of the winners, and has no lower limit at all for 14.32% of them. It restores the promise by declining to say most of what the ordinary interval said.
Five places to cut one variable
A continuous baseline variable has to be cut before it makes a subgroup table, and the cut is chosen. In a randomised trial of eighty where the treatment helps every patient equally, the strongest baseline variable cut at its median shows a Simpson reversal in 4.4% of trials; tried at five candidate cuts, some cut reverses in 10.8%, and at nine, 13.2%. Nine correlated cuts are worth about five independent chances, not nine. Across twenty continuous variables cut at five points each, some table reverses in 32.2% of trials — twice the rate with each variable cut once, and about thirty-one independent chances where the table holds a hundred.
An interval for the analyses admitted to
An analyst runs twenty correlated analyses, admits to one, and reports the winner with an interval conditioned on having cleared 1.96 — the selection as written down rather than as it happened. The interval covers 95.3% of the time, against 95.6% for one conditioned on all twenty, because a winner's truncation point is set by the threshold far more than by its unseen rivals: hiding nineteen analyses moves it by 0.061 on average. What the hiding changes is how often there is a winner at all. With no effect anywhere, a family of twenty produces a reported winner 35.5% of the time against 4.8% when all twenty are admitted, and one whose interval excludes zero 1.48% of the time against 0.13% — the interval keeps its promise and the report does not.
A cut that slides
Slide the cut on a trial's strongest baseline variable along every gap between observed values in its middle eighty per cent, and a trial of eighty whose treatment helps everyone equally shows a Simpson reversal somewhere in 23.4% of trials — against 13.2% for the nine deciles and 4.4% for the median. Sixty-five cut positions are worth about 9.5 independent cuts, and at 640 patients, with 513 positions, about 7.2: a table that moves one patient at a time is a continuous scan, and its count is set by the range scanned rather than by the number of cuts. The reversing cuts spread along the whole range, in 2.62 separate runs at eighty patients and 8.36 at 640.
A winner chosen by its diagnostic
An analyst runs twenty correlated analyses and reports the one whose diagnostics look cleanest, if it clears 1.96 — and a diagnostic is rarely independent of the statistic it checks. With no effect anywhere, a diagnostic correlated 0.9 with its statistic lets a winner through in 13.07% of families against 2.52% for an unrelated one. The interval conditioned on clearing 1.96 still covers 93.27%, but its misses have moved: it excludes zero in 4.69% of reported winners, where a 95% interval should do so in 2.5%. Conditioning on the diagnostic as well repairs it, and so does something a reader can do alone — weight the reported statistic by the chance an analysis like it wins the diagnostic contest, from three numbers a report can state.
Named alongside it
The objects these essays reach for when they reach for this one.
Multiple comparisonsFamilywise error rateFalse-positiveConfidence intervalCorrelationCoverageDichotomisationResearcher degrees of freedomSelective inferenceStatistical powerTruncated normalBonferroni