Field

The analyses that were available and not run

A correction divides by the number of analyses, and that number is the one quantity nobody measures. Twenty analyses of one dataset are worth 11.37 independent ones at a correlation of 0.6 and 2.58 at 0.95, so the threshold controlling a stated error rate can be counted rather than assumed. What the correction costs is charged in the estimate rather than the error rate: a larger statistic is a more selected one, and what survives averages 1.69 times the truth after correction against 1.35 times before it. Naming the analysis in advance is the alternative, and it is worth exactly what correcting all twenty is worth when the chance of having named the right one is 38%.

What 20 analyses of one dataset are worth. The threshold giving a 5% family-wise error rate, read back as a number of independent analyses. At no correlation it is 20.05; at 0.6 it is 11.37; at 0.95 it is 2.58. Bonferroni divides by 20 throughout.

How many analyses there really were

Bonferroni divides by twenty because twenty analyses were run. Twenty analyses of one dataset are worth 11.37 independent ones at a correlation of 0.6 and 2.58 at 0.95, and the threshold that controls exactly the same error rate is measurable rather than assumed.

Naming the analysis in advance, against correcting for all 20 of them. The prespecified analysis detects an effect that is in it 52% of the time at two standard errors and an effect elsewhere 5% of the time. The corrected slate detects it 23% of the time wherever it is. The two are worth the same when the chance of having named the right analysis is 38% — and that figure rises to 91% at four standard errors.

What naming it in advance costs

Preregistration is argued for as free. Against an effect of two standard errors hiding in one of twenty analyses, naming the right one detects it 51.5% of the time and naming the wrong one detects it 4.7% of the time; correcting all twenty detects it 22.5% wherever it is. The two are worth the same when the chance of having named correctly is 38%.

What the family-wise correction does to the effect it lets through. At two standard errors the estimate that clears an uncorrected 5% threshold averages 1.35 times the truth, and the one that clears the family-wise threshold averages 1.69 times it. The correction fixes the error rate by demanding a larger estimate, and a larger estimate is a more selected one.

The correction that makes the estimate worse

Correcting for twenty analyses repairs the p-value by demanding a larger statistic, and a larger statistic is a more selected one. At two standard errors the surviving estimate averages 1.35 times the truth before the correction and 1.69 times it after — so the honest error rate is bought with a more inflated effect.

How much naming more analyses in advance is worth, at an effect of 2 standard errors, for four analysts' beliefs. Twenty analyses correlated at 0.6, one of which holds the effect. Each curve is an analyst whose belief falls by a fixed ratio from one choice to the next: 0.2, 0.5, 0.7, 0.9. The best number to name is 1, 3, 5, 15 respectively, and each best set holds between 80% and 90% of the analyst's belief. Every curve ends at 22.7%, the power of correcting for all twenty.

Naming a handful in advance

A preregistration that names three of twenty analyses sits between naming one and naming none, and whether it beats both depends on one thing: how much of the analyst's belief the three hold. An analyst who is 50% sure of a first choice, and whose belief halves down the list, gets 34.8% expected power from naming three, 28.3% from naming one and 22.7% from exploring all twenty. Spread the same 50% evenly over the rest and naming three is the worst of the three plans. The rule for adding one more to the list needs no belief at all: the next analysis must be at least 20.8% as likely as the first to be worth naming.

Thirty winners of twenty analyses, effect 2 standard errors in one of them, with two intervals each. Each row is the largest of twenty statistics correlated at 0.6, reported because it cleared the family-wise threshold of 2.846. The thin grey bar is the ordinary interval, z ± 1.96; the heavy bar is the interval conditional on the statistic having won; the tick is the true mean of the analysis that won. Of these thirty the ordinary interval misses 1 and the conditional one 2; over 4,468 reported winners they cover 76.0% and 94.9%.

An interval for the winner

The largest of twenty correlated analyses, reported because it cleared the family-wise threshold, carries an ordinary 95% interval that covers its true mean 75.96% of the time at an effect of two standard errors and never at all when there is no effect. An interval built on the fact that it won covers 94.90% — and it is 2.48 times as wide, excludes zero for only 14.10% of the winners, and has no lower limit at all for 14.32% of them. It restores the promise by declining to say most of what the ordinary interval said.

How often a winner's interval covers when it is conditioned on the analyses admitted to rather than the twenty run, effect 2. The largest of twenty analyses reported when it clears the family-wise threshold for the k admitted. At k = 1, 2, 5, 10, 20: correlation 0.6, conditioned on the admitted family 95.3%, 94.9%, 94.9%, 95.1%, 95.2%; the ordinary interval 67.7%, 71.1%, 75.2%, 76.5%, 76.1%; correlation 0, conditioned on the admitted family 94.4%, 95.6%, 95.3%, 94.8%, 95.6%; the ordinary interval 47.2%, 52.0%, 58.9%, 62.4%, 64.7%.

An interval for the analyses admitted to

An analyst runs twenty correlated analyses, admits to one, and reports the winner with an interval conditioned on having cleared 1.96 — the selection as written down rather than as it happened. The interval covers 95.3% of the time, against 95.6% for one conditioned on all twenty, because a winner's truncation point is set by the threshold far more than by its unseen rivals: hiding nineteen analyses moves it by 0.061 on average. What the hiding changes is how often there is a winner at all. With no effect anywhere, a family of twenty produces a reported winner 35.5% of the time against 4.8% when all twenty are admitted, and one whose interval excludes zero 1.48% of the time against 0.13% — the interval keeps its promise and the report does not.

How often a diagnostically chosen winner's interval excludes zero, against the diagnostic's correlation with the statistic, no effect anywhere. Twenty analyses correlated 0.6; the one with the best diagnostic is reported if it clears 1.96. A winner is reported in 2.52%, 4.16%, 6.81%, 13.07% of families at γ = 0, 0.3, 0.6, 0.9. Its interval excludes zero: conditioned on clearing 1.96, 1.69%, 2.28%, 3.30%, 4.69%; conditioned on the diagnostic as well, 1.69%, 1.86%, 1.83%, 2.31%; rebuilt from the report's three numbers, 1.69%, 1.80%, 2.13%, 2.52%.

A winner chosen by its diagnostic

An analyst runs twenty correlated analyses and reports the one whose diagnostics look cleanest, if it clears 1.96 — and a diagnostic is rarely independent of the statistic it checks. With no effect anywhere, a diagnostic correlated 0.9 with its statistic lets a winner through in 13.07% of families against 2.52% for an unrelated one. The interval conditioned on clearing 1.96 still covers 93.27%, but its misses have moved: it excludes zero in 4.69% of reported winners, where a 95% interval should do so in 2.5%. Conditioning on the diagnostic as well repairs it, and so does something a reader can do alone — weight the reported statistic by the chance an analysis like it wins the diagnostic contest, from three numbers a report can state.

How often two orders of selection report a winner from twenty analyses, no effect anywhere. Twenty analyses correlated 0.6, each with a diagnostic correlated γ with its own statistic, over 20,000 families a point. Clean then significant — the best diagnostic, reported if it clears 1.96 — reports a winner in 2.55%, 4.25%, 6.91%, 13.15% of families at γ = 0, 0.3, 0.6, 0.9. Significant then clean — the best diagnostic among those that cleared 1.96 — reports one in 18.04%, 18.04%, 18.04%, 18.05%: in every family where any analysis cleared 1.96, whatever γ is.

The cleanest of the significant

An analyst with twenty correlated analyses can choose in two orders: take the one whose diagnostics look cleanest and report it if it clears 1.96, or take the analyses that cleared 1.96 and report the cleanest of them. The second is the one described as prudent, because it never reports an analysis whose diagnostics look poor — and with no effect anywhere it reports a significant analysis in about 18% of families whatever the diagnostic is, against 2.55% to 13.15% for the first. The diagnostic decides which significant analysis is shown and never whether one is. Each report's own interval can still be made exact, by conditioning on a selection set that is a union of pieces rather than a half-line; the family's claim count cannot.

All essays