Normal Distribution
nothing is called 95% until it has been counted
Series

Uniformity — the series

5 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. 20,000 p-values from a true null, n = 12. Flat, as it must be: under the null a p-value is uniform on (0,1). The Kolmogorov–Smirnov distance from uniform is 0.0090 (p = 0.81). That flatness is the check that catches an error a single rejection rate would miss.

    A p-value that is not flat is not a p-value

    Under a true null, p-values are uniform. That is stronger than saying the test rejects 5% of the time, it constrains the whole distribution rather than one point of it, and it catches implementation errors that a rejection rate sails past.

    part 1 · testing
  2. The p-value of a study with 80% power, twenty thousand times. Twenty thousand two-sided z-tests, each on 25 observations whose true mean is 0.5603 standard deviations from the null, a noncentrality of 2.802. The bars are the counted share of p-values in bins a quarter of a power of ten wide, with the leftmost bin holding everything smaller; the line is the closed form. The middle eighty per cent of the p-values runs from 4.4×10⁻⁵ to 0.13, 3.46 orders of magnitude, the median is 0.0051, and 80.0% fall below 0.05, which is what the power means.

    The p-value a replication gets

    Under a true null a p-value is flat. Under a real effect its distribution is closed form and wide — a study with 80% power returns anything from 4.4×10⁻⁵ to 0.13 in eight runs of ten — and the chance that an exact replication of a p = 0.05 result is significant again is exactly one half, under both of the models people use without naming them.

    part 2 · testing
  3. Three ways to reject with two studies, drawn where the two z statistics live. Two one-sided studies, each summarised by its z statistic. Fisher's combination rejects outside a curve that runs parallel to both axes, so one study past z = 2.378 decides it alone; Stouffer's rejects above the straight line z₁ + z₂ = 2.326; Tippett's rejects when either z passes 1.955. Each region holds exactly 5% of the standard bivariate normal — Fisher's in closed form, e^(−c/2)(1 + c/2) at c = 9.488 — and of 100,000 counted null pairs they catch 4.95%, 4.88% and 5.04%. Two alternatives carry the same Stouffer evidence: one study at 2.326 and the other at nothing, where the powers are 62.7%, 50.0% and 65.4%; and both at 1.163, where they are 47.7%, 50.0% and 38.3%.

    Two ways to combine p-values

    Fisher's and Stouffer's combinations are both exactly right when every null is true, for the single reason that each p-value is flat. Under a real effect they disagree about which evidence counts: with Stouffer held at 50% power across ten studies, Fisher is the more powerful while the signal sits in six or fewer of them and the less powerful from seven.

    part 3 · testing
  4. How often each combination, and each union of them, rejects ten studies of nothing. Each combination alone rejects exactly 5% of null sets. Counted on 1,000,000 sets of ten null studies: Fisher or Stouffer 6.63%, Fisher or Tippett 8.05%, Stouffer or Tippett 8.96%, any of the three 9.66% — enclosed on a two-dimensional lattice between 9.18% and 10.09% — and all three together 1.05%. The three sizes add to 15%.

    The smallest of three combinations

    Reporting whichever of Fisher's, Stouffer's and Tippett's combinations is smallest is a test of its own, and on ten studies of nothing it rejects 9.66% of the time — not 5%, and nowhere near the 15% the three sizes add to, because the statistics are correlated at up to 0.903. Read at 2.448% each it is exact, and then it trails the best single combination by at most 7.45 points and leads the worst by at least 10.30.

    part 4 · testing
  5. Ten studies every study shares the control: the three combinations read as though independent, by the correlation. Counted on 400,000 null sets of ten one-sided studies at each correlation, every study shares the control. Fisher: 5.00%, 9.54%, 12.83%, 15.01%, 16.63%, 17.88%, 18.80%. Stouffer: 5.00%, 11.51%, 16.29%, 19.54%, 22.20%, 24.23%, 25.86%. Tippett: 4.98%, 4.86%, 4.68%, 4.45%, 4.17%, 3.72%, 3.27%. Smallest of three: 9.66%, 14.44%, 17.94%, 20.41%, 22.55%, 24.34%, 25.88% — at correlations 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6. Each combination is read at its critical value for independent studies.

    Studies that share a control

    Ten comparisons against one control arm of the same size are correlated at exactly one half, and combined as though they were independent, Stouffer's sum rejects 24.23% of sets in which nothing works, Fisher's 17.88%, while Tippett's smallest p-value falls to 3.72%. Repaired with the correlation the design states, each is exact again — and the smallest of the three is then inflated less than for independent studies, 7.05% against 9.66%, because sharing makes the three combinations agree. The two inflations the dependence was expected to stack do not stack.

    part 5 · testing

All series