Normal Distribution
nothing is called 95% until it has been counted
Concept

Fisher's method — where it appears

A combination of independent p-values that sums minus twice their logarithms and reads the total against a chi-squared on twice as many degrees of freedom. The logarithm is dominated by the smallest p-values, so the combination can be convinced by one strong study as readily as by several moderate ones.

Named by 3 essays across one field — each of them below, with the objects they name alongside it.

Also named here as global null, stouffer's method, tippett's method — the same set of essays touches all of them, so they are one junction rather than several.

Three ways to reject with two studies, drawn where the two z statistics live. Two one-sided studies, each summarised by its z statistic. Fisher's combination rejects outside a curve that runs parallel to both axes, so one study past z = 2.378 decides it alone; Stouffer's rejects above the straight line z₁ + z₂ = 2.326; Tippett's rejects when either z passes 1.955. Each region holds exactly 5% of the standard bivariate normal — Fisher's in closed form, e^(−c/2)(1 + c/2) at c = 9.488 — and of 100,000 counted null pairs they catch 4.95%, 4.88% and 5.04%. Two alternatives carry the same Stouffer evidence: one study at 2.326 and the other at nothing, where the powers are 62.7%, 50.0% and 65.4%; and both at 1.163, where they are 47.7%, 50.0% and 38.3%.

Two ways to combine p-values

Fisher's and Stouffer's combinations are both exactly right when every null is true, for the single reason that each p-value is flat. Under a real effect they disagree about which evidence counts: with Stouffer held at 50% power across ten studies, Fisher is the more powerful while the signal sits in six or fewer of them and the less powerful from seven.

testing · Uniformity How often each combination, and each union of them, rejects ten studies of nothing. Each combination alone rejects exactly 5% of null sets. Counted on 1,000,000 sets of ten null studies: Fisher or Stouffer 6.63%, Fisher or Tippett 8.05%, Stouffer or Tippett 8.96%, any of the three 9.66% — enclosed on a two-dimensional lattice between 9.18% and 10.09% — and all three together 1.05%. The three sizes add to 15%.

The smallest of three combinations

Reporting whichever of Fisher's, Stouffer's and Tippett's combinations is smallest is a test of its own, and on ten studies of nothing it rejects 9.66% of the time — not 5%, and nowhere near the 15% the three sizes add to, because the statistics are correlated at up to 0.903. Read at 2.448% each it is exact, and then it trails the best single combination by at most 7.45 points and leads the worst by at least 10.30.

testing · Uniformity Ten studies every study shares the control: the three combinations read as though independent, by the correlation. Counted on 400,000 null sets of ten one-sided studies at each correlation, every study shares the control. Fisher: 5.00%, 9.54%, 12.83%, 15.01%, 16.63%, 17.88%, 18.80%. Stouffer: 5.00%, 11.51%, 16.29%, 19.54%, 22.20%, 24.23%, 25.86%. Tippett: 4.98%, 4.86%, 4.68%, 4.45%, 4.17%, 3.72%, 3.27%. Smallest of three: 9.66%, 14.44%, 17.94%, 20.41%, 22.55%, 24.34%, 25.88% — at correlations 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6. Each combination is read at its critical value for independent studies.

Studies that share a control

Ten comparisons against one control arm of the same size are correlated at exactly one half, and combined as though they were independent, Stouffer's sum rejects 24.23% of sets in which nothing works, Fisher's 17.88%, while Tippett's smallest p-value falls to 3.72%. Repaired with the correlation the design states, each is exact again — and the smallest of the three is then inflated less than for independent studies, 7.05% against 9.66%, because sharing makes the three combinations agree. The two inflations the dependence was expected to stack do not stack.

testing · Uniformity

Named alongside it

The objects these essays reach for when they reach for this one.

Global nullMonte CarloMultiple comparisonsp-valueStouffer's methodTippett's methodUniformityCorrelationStatistical powerChi squaredClosed formConservative test

All concepts