20,000 p-values from a true null, n = 12

Flat, as it must be: under the null a p-value is uniform on (0,1). The Kolmogorov–Smirnov distance from uniform is 0.0090 (p = 0.81). That flatness is the check that catches an error a single rejection rate would miss.

The level, read off 400,000 counted null sets of ten, at which the smallest of the three repaired combined p-values falls below it 5% of the time. Every study sharing: 2.449%, 2.723%, 2.904%, 3.094%, 3.229%, 3.449%, 3.681%. Studies sharing in pairs: 2.457%, 2.471%, 2.527%, 2.526%, 2.543%, 2.572%, 2.614% — at correlations 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6. For independent studies the level is 2.449%; the further the studies share, the closer it moves to 5%, because the three repaired combinations agree more.

For a common effect, Stouffer's repaired statistic has the precision of a number of independent studies equal to the square of the count over the count plus twice the sum of the correlations between pairs. Ten studies all sharing at a correlation of 0.5 are worth 1.82; in pairs they are worth 6.67. At 0.2 the two are 3.57 and 8.33.

Twenty analyses correlated 0.6; the one with the best diagnostic is reported if it clears 1.96. A winner is reported in 2.52%, 4.16%, 6.81%, 13.07% of families at γ = 0, 0.3, 0.6, 0.9. Its interval excludes zero: conditioned on clearing 1.96, 1.69%, 2.28%, 3.30%, 4.69%; conditioned on the diagnostic as well, 1.69%, 1.86%, 1.83%, 2.31%; rebuilt from the report's three numbers, 1.69%, 1.80%, 2.13%, 2.52%.

Twenty analyses correlated 0.6, each with a diagnostic correlated γ with its own statistic, over 20,000 families a point. Clean then significant — the best diagnostic, reported if it clears 1.96 — reports a winner in 2.55%, 4.25%, 6.91%, 13.15% of families at γ = 0, 0.3, 0.6, 0.9. Significant then clean — the best diagnostic among those that cleared 1.96 — reports one in 18.04%, 18.04%, 18.04%, 18.05%: in every family where any analysis cleared 1.96, whatever γ is.

One family of twenty analyses correlated 0.6, diagnostics correlated 0.6 with their statistics. Under significant then clean the reported analysis's statistic is 2.31, and given everything else about the family it would have been the one reported for any value in [1.96, 2.49] ∪ [3.65, ∞): above the gap another analysis becomes significant and has the cleaner diagnostic. Its exact interval, conditioned on that set, is [−7.83, 4.16]; conditioned only on clearing 1.96, [−8.30, 4.01]. Under clean then significant the analysis with the best diagnostic has a statistic of 1.85 and nothing is reported.

Significant then clean, twenty analyses correlated 0.6, 20,000 families a point. The reported winner's interval excludes zero — wrongly, since every analysis is null: conditioned only on clearing 1.96, in 0.64%, 1.05%, 1.69%, 3.35% of reports at γ = 0, 0.3, 0.6, 0.9; conditioned on the whole selection set, 2.05%, 2.30%, 2.13%, 2.19%; rebuilt from the number of analyses, their correlation and the diagnostic's correlation, 2.19%, 2.36%, 2.52%, 2.63%. A 95% interval should exclude it 2.5% of the time.

The threshold-only interval misses below in 1.69%, 2.28%, 3.30%, 4.69% of reported winners at γ = 0, 0.3, 0.6, 0.9, and above in 3.28%, 3.06%, 2.61%, 2.05%. The rebuilt interval misses below in 1.69%, 1.80%, 2.13%, 2.52% and above in 3.28%, 2.88%, 2.42%, 1.91%.

Twenty analyses correlated 0.6, the others null. At z = 2 the chance is 0.074 for γ = 0.3, 0.111 for γ = 0.6, 0.194 for γ = 0.9; at z = 4, 0.114, 0.242, 0.576. With no correlation it would be one in twenty at every z.

Of 5,229 reported winners, 76.0% have no truncation beyond 1.96. The rest are moved by up to 2.30; 6.4% of all winners by more than half a standard error and 1.6% by more than one. The mean move is 0.090.

Flat, as it must be: under the null a p-value is uniform on (0,1). The Kolmogorov–Smirnov distance from uniform is 0.0090 (p = 0.81). That flatness is the check that catches an error a single rejection rate would miss.

Power is 21%. The studies that reached significance report a mean effect of 0.621 — 2.07 times the truth. Every one of them is honest; the selection did the inflating.

The data has no effect in it. Each analysis is correct and each p-value is honest. With 20 correlated outcomes available, something reaches p below 0.05 58% of the time.

A p of 0.04 at ten observations needs an effect of 0.76 standard deviations; at two thousand it needs 0.046. The p-value alone does not say which of these happened, which is why it should never be reported alone.

Twenty thousand two-sided z-tests, each on 25 observations whose true mean is 0.5603 standard deviations from the null, a noncentrality of 2.802. The bars are the counted share of p-values in bins a quarter of a power of ten wide, with the leftmost bin holding everything smaller; the line is the closed form. The middle eighty per cent of the p-values runs from 4.4×10⁻⁵ to 0.13, 3.46 orders of magnitude, the median is 0.0051, and 80.0% fall below 0.05, which is what the power means.

Quantiles of a two-sided z-test's p-value in closed form, at powers from 5% to 99%, on a scale of powers of ten. The pale band is the middle ninety-five per cent, the darker one the middle eighty per cent, and the line the median. At 5% power the null is true and the middle eighty per cent is 0.1 to 0.9. At 80% power it is 4.4×10⁻⁵ to 0.13, 3.46 orders of magnitude, and at 99% it is 5.01: more power moves a p-value down and widens the range it lands in.

Two models for the effect behind a first two-sided p-value. Taking the estimate as the truth gives a replication significant in the same direction 73.1% of the time after p = 0.01 and 90.8% after 0.001. A flat prior, which carries the estimate's own uncertainty into the prediction, gives 66.8% and 82.7%. At p = 0.05 both are exactly one half. The points are counted from 600,000 simulated pairs of studies with effects spread flat, keeping the pairs whose first p-value fell near 0.05, 0.01 and 0.001, and they agree with the curve within their error bars of two standard errors.

The predictive distribution of a replication's two-sided p-value after a first p of 0.05, under three models for the effect, on a scale of powers of ten. Each bar is the middle eighty per cent, the thin line the middle ninety-five, the tick the median. Taking the estimate as the truth: 0.0012 to 0.48. A flat prior: 1.6×10⁻⁴ to 0.65. A sceptical prior with τ = 2: 0.001 to 0.74, median 0.11. The open circles on the flat row are the counted tenth, fiftieth and ninetieth percentiles from 7,223 simulated pairs whose first p fell near 0.05.

Two one-sided studies, each summarised by its z statistic. Fisher's combination rejects outside a curve that runs parallel to both axes, so one study past z = 2.378 decides it alone; Stouffer's rejects above the straight line z₁ + z₂ = 2.326; Tippett's rejects when either z passes 1.955. Each region holds exactly 5% of the standard bivariate normal — Fisher's in closed form, e^(−c/2)(1 + c/2) at c = 9.488 — and of 100,000 counted null pairs they catch 4.95%, 4.88% and 5.04%. Two alternatives carry the same Stouffer evidence: one study at 2.326 and the other at nothing, where the powers are 62.7%, 50.0% and 65.4%; and both at 1.163, where they are 47.7%, 50.0% and 38.3%.

Each trial draws 10 uniform p-values, as true nulls produce, and combines them by Fisher's method against a chi-square on 20 degrees of freedom and by Stouffer's against a standard normal. Left half of each bin Fisher, right half Stouffer. Both are flat: Kolmogorov–Smirnov distances 0.0045 and 0.0044, rejection rates 4.92% and 4.90%. Each is exact because its inputs are uniform, and for no other reason.

The same total evidence — enough to give Stouffer's combination 50% power at every spread — placed in m of 10 one-sided studies: a shift of 5.201 standard errors in one study, down to 0.520 in each of all 10. Fisher's power, computed on a lattice whose error is enclosed rather than estimated, falls from 96.1% to 45.0%; Tippett's minimum-p test falls from 99.6% to 18.5%. Fisher beats Stouffer while the signal sits in 6 or fewer of the 10 studies and loses from 7 on. Open circles are Fisher's power counted on 20,000 trials a point.

For each number of studies k, the largest number m of them the same total signal can be spread over while Fisher's combination is still more powerful than Stouffer's, with Stouffer held at 50% power. Every point is located by a bracketed power computation that does not straddle the comparison. It is 1 of 2, 6 of 10 and 18 of 40: a falling share of the studies — 75% at four, 60% at ten, 45.0% at 40 — and a count growing faster than the square root of k. Neither a fixed fraction nor √k describes it.

The exact one-sided binomial test on ten trials at a null probability of one half can return only eleven p-values, and alone it rejects 1.07% of the time at a nominal 5%; its mid-p version rejects 5.47%. Combining k of them by Fisher's method, with every multiset of outcomes enumerated, the true size is 1.07%, 1.59%, 1.67%, 1.33%, 1.12%, 0.94%, 0.71%, 0.55% at k = 1, 2, 3, 4, 5, 6, 8, 10 from exact p-values, and 5.47%, 3.36%, 3.76%, 4.26%, 4.04%, 4.01%, 3.98%, 3.89% from mid-p values. At ten studies: 0.55% and 3.89%. Open circles are sizes counted on 40,000 simulated sets, with two standard errors.

Each combination alone rejects exactly 5% of null sets. Counted on 1,000,000 sets of ten null studies: Fisher or Stouffer 6.63%, Fisher or Tippett 8.05%, Stouffer or Tippett 8.96%, any of the three 9.66% — enclosed on a two-dimensional lattice between 9.18% and 10.09% — and all three together 1.05%. The three sizes add to 15%.

The threshold giving a 5% family-wise error rate, read back as a number of independent analyses. At no correlation it is 20.05; at 0.6 it is 11.37; at 0.95 it is 2.58. Bonferroni divides by 20 throughout.

The prespecified analysis detects an effect that is in it 52% of the time at two standard errors and an effect elsewhere 5% of the time. The corrected slate detects it 23% of the time wherever it is. The two are worth the same when the chance of having named the right analysis is 38% — and that figure rises to 91% at four standard errors.

At two standard errors the estimate that clears an uncorrected 5% threshold averages 1.35 times the truth, and the one that clears the family-wise threshold averages 1.69 times it. The correction fixes the error rate by demanding a larger estimate, and a larger estimate is a more selected one.

Each bar is the reported effect as a multiple of the truth; each note is the share of families that clear that threshold at all. The uncorrected threshold lets 68.5% through at 1.35 times the truth; Bonferroni lets 17.0% through at 1.76 times it.

Twenty analyses correlated at 0.6, one of which holds the effect. Each curve is an analyst whose belief falls by a fixed ratio from one choice to the next: 0.2, 0.5, 0.7, 0.9. The best number to name is 1, 3, 5, 15 respectively, and each best set holds between 80% and 90% of the analyst's belief. Every curve ends at 22.7%, the power of correcting for all twenty.

At an effect of two standard errors, naming one analysis gives 51.6% power if it is the right one, three give 39.1%, and all twenty give 22.7%. The certainty that the named set contains the effect needed to beat exploring rises from 37.9% at one to 51.9% at three and 77.8% at ten.

Adding a second named analysis pays when it is at least 20.8% as likely to hold the effect as the first, at an effect of two standard errors; a third, when it is 13.2% as likely as the first two together; a fourth, 9.7%. Against an effect of three standard errors the shares are 6.8%, 4.9% and 3.8%.

Each row is the largest of twenty statistics correlated at 0.6, reported because it cleared the family-wise threshold of 2.846. The thin grey bar is the ordinary interval, z ± 1.96; the heavy bar is the interval conditional on the statistic having won; the tick is the true mean of the analysis that won. Of these thirty the ordinary interval misses 1 and the conditional one 2; over 4,468 reported winners they cover 76.0% and 94.9%.

Twenty analyses correlated at 0.6, threshold 2.846. The ordinary interval is the winner ± 1.96 whatever the margin. The conditional interval has no lower limit until the winner clears the threshold by 0.10, excludes zero only beyond a margin of 1.03, and approaches the ordinary interval as the margin grows.

Among winners that clear the family-wise threshold, the conditional interval covers 95.1%, 94.4%, 95.1%, 94.6%, 95.2% at effects of 0, 1, 2, 3, 4; the ordinary one covers 0.0%, 9.7%, 76.4%, 92.2%, 95.2%. The conditional interval excludes zero for 2.6%, 4.2%, 14.2%, 32.7%, 62.0% of them and has no lower limit for 28.4%, 24.3%, 13.9%, 7.5%, 2.6%.

The largest of twenty analyses reported when it clears the family-wise threshold for the k admitted. At k = 1, 2, 5, 10, 20: correlation 0.6, conditioned on the admitted family 95.3%, 94.9%, 94.9%, 95.1%, 95.2%; the ordinary interval 67.7%, 71.1%, 75.2%, 76.5%, 76.1%; correlation 0, conditioned on the admitted family 94.4%, 95.6%, 95.3%, 94.8%, 95.6%; the ordinary interval 47.2%, 52.0%, 58.9%, 62.4%, 64.7%.

For winners observed at z = 3, 4 and 5: the lower limit with truncation at 1.96 is -0.80, 1.65, 2.97; the truncation point at which it reaches zero is 1.96, 3.02, 4.23.

Correlation 0.6, no effect anywhere, 12,000 families. A winner is reported in 35.5%, 22.9%, 11.9%, 7.4%, 4.8% of families at k = 1, 2, 5, 10, 20; one whose admitted-family interval excludes zero in 1.48%, 0.83%, 0.43%, 0.19%, 0.13%.

Assuming a correlation of 0.6, the number of unreported null analyses at which the median truncation point reaches the point where the interval's lower limit is zero: z = 3.25, 7; z = 3.5, 10; z = 3.75, 16; z = 4, 25; z = 4.25, 37; z = 4.5, 60; z = 4.75, 100; z = 5, 171.