A p-value that is not flat is not a p-value

A p-value is the probability, if the null hypothesis is true, of a statistic at least as extreme as the one observed. Read that definition carefully and a consequence falls out immediately: under the null, a p-value is the value of its own distribution function, and is therefore uniform on (0, 1).

Why flatness is the better check #

The obvious check on a test is its rejection rate: run it under the null and confirm that about 5% of runs give p < 0.05. That is worth doing and it is weak.

A rejection rate checks the distribution at one point. A test can have exactly the right rate at 0.05 and be wrong everywhere else — too many small p-values compensated by too few moderate ones, a distribution that is right in the far tail and wrong in the body, or a systematic distortion that happens to cross the correct value at the threshold being checked.

Uniformity checks the whole distribution. Every quantile has to be right simultaneously, which is a far more demanding condition and correspondingly harder to satisfy by accident.

The practical difference is that flatness catches the errors people actually make:

• a wrong degrees of freedom, which shifts the whole distribution
• a one-sided p reported as two-sided, or the reverse
• a variance computed with n instead of n − 1
• a test statistic missing a scaling factor

Each of those breaks flatness immediately. Several of them can leave the rejection rate at 0.05 looking roughly acceptable.

The place to look for them is a small sample, and that is worth being explicit about because it is the opposite of where a simulation is usually pointed. Most of the errors in that list are errors about a correction — a degrees-of-freedom count, a divisor of n against n − 1, a factor of two — and every one of those corrections shrinks as the sample grows. At two thousand observations a variance divided by n instead of n − 1 is wrong by a twentieth of a percent, the resulting distribution is flat to within the Monte Carlo error of any reasonable number of trials, and the check passes a test that is genuinely broken. At four observations the same mistake is a 33% error in the variance, the whole distribution shifts, and it is unmissable.

So the sample size to check at is the smallest one the test is ever going to be used at, not the largest one the simulation can afford. A flat histogram at n = 4 is a strong statement about the arithmetic; a flat histogram at n = 2000 is close to a statement that the central limit theorem exists, which was not in doubt.

The check, checked #

An assertion that has never been shown to reject anything proves nothing, so the site’s gate feeds this check something it must refuse.

It takes the same t test and halves every p-value — the classic one-sided/two-sided slip, and a mistake that produces plausible-looking output. The rejection rate roughly doubles, which a careful reviewer might notice. The uniformity check does not need care: the distribution is crushed into the lower half of the range and the Kolmogorov–Smirnov statistic detects it overwhelmingly.

The gate requires that refusal to happen. If halving the p-values stopped being detected, the check would have quietly stopped working and the build fails.

What twenty thousand draws can see #

A flatness check is a claim about a histogram, so it is worth saying what size of departure the histogram can resolve.

Twenty thousand p-values in twenty bins of width 0.05 put an expected 1,000 in each, with a standard deviation of $\sqrt{1000 \times 0.95}$, which is 31.

So a bin is 1,000 ± 31, and a departure of two standard errors is 62 — about six per cent of the bin.

Twenty thousand draws therefore detect a six per cent distortion in any one bin and nothing smaller.

Which is the right resolution for the claim being made. A test whose true size is 5.5% rather than 5% has a ten per cent excess in the first bin — comfortably visible. A test whose size is 5.1% has a two per cent excess and would pass this check every time.

The check is a refutation instrument, not a proof: it can show a test is not flat and it cannot show one is, and twenty thousand draws is the number that fixes where the line between those falls.

Reading the flatness correctly #

Two things the figure does not say.

A flat histogram does not mean the test is right for a given dataset. It means the implementation is consistent with its own assumptions. Here the data is genuinely normal and independent, which is exactly what the t test assumes. Feed the same correct implementation correlated data and the p-values stop being uniform — which is useful, and is a check on the assumptions rather than the code.

The flatness is a property of the null, not of the world. Under an alternative, p-values pile up near zero, and how fast they pile up is the power — while how widely they spread under a real effect is a second number, and a larger one than intuition allows. The uniform case is the baseline against which that is measured, which is why the power curve is the same measurement taken at a different truth.

What this implies about a single p-value #

If p-values are uniform under the null, then under the null every value is as likely as every other. A p of 0.04 and a p of 0.96 are equally probable when nothing is going on.

That has a consequence worth sitting with: a single p-value just below a threshold is weak evidence. The likelihood ratio between “there is an effect” and “there is not” at p = 0.04 is modest under most reasonable alternatives — far more modest than the word “significant” suggests.

It also explains why a distribution of p-values across many studies is so much more informative than any one of them. A field where the null is mostly true should produce a flat distribution of published p-values; a lump just below 0.05 is a signature of selection, and the size of the lump is measurable. That is not a hypothetical technique — it is how the replication literature diagnosed the problem it went on to document.

The habit #

The transferable point is not about p-values. It is that the strongest checks constrain a whole distribution rather than a single summary.

The same idea appears everywhere on this site. An interval is checked by its coverage across the whole range of the parameter rather than at one value. A random generator is checked in two dimensions rather than one, because a bad one passes the one-dimensional test. A distribution function is checked against its own quantile at a dozen probabilities rather than at the median.

In each case the cheap check passes things the demanding check catches, and the demanding check costs almost nothing extra.

What flatness looks like when it fails #

The check is only useful if the failures are recognisable, and the four common ones have distinct signatures.

A one-sided p reported as two-sided, or the reverse, compresses the distribution into half the interval. The histogram is flat over (0, 0.5) and empty above, or the mirror image. Unmistakable once seen and easy to miss if only the rejection rate is checked.

A wrong degrees of freedom tilts the distribution: too many small p-values if the degrees of freedom are overstated, too few if understated. The tilt is gentle and the rejection rate can still look plausible.

A variance computed with n rather than n − 1 inflates the statistic slightly, which piles up mass at the low end. At large n it is invisible; at small n it is obvious, which is the opposite of most errors.

A discrete test statistic produces a genuinely lumpy distribution, and this one is not an error — it is the same discreteness that makes interval coverage oscillate. A test on counts cannot have uniform p-values, and expecting it to is the mistake.

That last case is why the check reports a KS distance and a p-value rather than a verdict. It is a measurement, and interpreting it requires knowing whether the statistic was continuous.

Conservative and anti-conservative #

The two directions of failure have very different consequences and deserve their own names.

A test whose p-values pile up above uniform is conservative: it rejects less often than its stated level, so its false positives are under control and it has less power than advertised. The costs are invisible — missed effects and inflated estimates among those it does detect.

A test whose p-values pile up below uniform is anti-conservative: it rejects more often than stated. Its false positives exceed the advertised rate, and every claim made with it is weaker than it appears.

The second is the dangerous one and the harder to notice, because a method that finds more results looks better by every casual measure. Flatness is the check that distinguishes them and it does so in one picture.

Why this is the site’s first check #

The gate runs this before anything else, and the ordering is deliberate.

Every p-value on this site comes from the same t-test implementation. Every rejection rate, every power figure, the forking-paths measurement and the winner’s curse are counts of how often that implementation crosses a threshold.

If the implementation were wrong, all of those numbers would be wrong together, in a correlated way, and each would look individually plausible. Nothing downstream would catch it, because nothing downstream has an independent route to the same answer.

Uniformity is that independent route. It constrains the implementation using a fact — p-values are uniform under the null — that follows from the definition and not from the code. That is why it goes first.

Beyond testing #

The pattern transfers, and naming it is more useful than the instance.

Where a quantity has a known distribution under a stated condition, checking the whole distribution is far stronger than checking one summary of it — and usually costs the same.

A residual should be normal if the model is right: check the whole set, not the mean. A random generator should be uniform: check every bin, and in two dimensions, not just the overall average. An interval procedure should cover at 95%: check across the whole parameter range, not at one convenient value.

In every case the cheap version passes things the demanding version catches, and the demanding version is a few more lines.

The check on real data #

The uniformity check above uses simulated data where the null is true by construction. On real data the null is not known to be true, so the check changes character — and becomes useful in a different way.

If a set of p-values from a real study is far from uniform, one of two things holds: there are real effects, or the test’s assumptions are violated. Those are not distinguishable from the distribution alone.

But the shape carries information. Real effects produce a pile-up near zero with a roughly flat remainder. Violated assumptions — dependence, most often — produce distortions across the whole range, including too many values near one.

So the plot is worth making for a real analysis with many tests, and worth reading as a diagnostic rather than a verdict. A distribution that is flat except for a spike at zero is the picture of a well-behaved analysis with some real findings in it.

What a p-value would need to be uniform for #

The condition is more specific than it is usually stated, and the qualifications are where the practical failures live.

Uniformity requires the test statistic to be continuous and its distribution under the null to be known exactly. Both fail routinely.

A statistic on counts is discrete, so its p-values take finitely many values and cannot be uniform. They are conservative — the distribution sits above uniform — which is why exact tests on small counts have less power than their nominal level suggests. That is the same discreteness that makes interval coverage oscillate.

A statistic whose null distribution is approximated rather than known — most large-sample tests — has p-values that are only approximately uniform, and the approximation degrades in the same place every approximation on this site degrades: the tail, which is where the small p-values are.

So a plot that is flat in the body and wrong below 0.01 is the expected picture for an approximate test, and it is exactly the region that matters. Checking uniformity across the whole range, rather than eyeballing the histogram, is what catches it.

The permutation alternative #

Where the null distribution is not known, it can often be constructed, and the construction gives uniformity by design.

A permutation test computes the statistic under every relabelling of the data consistent with the null. The resulting distribution is the null distribution — not an approximation to it — so the p-value is exact up to the discreteness of the permutation set.

It costs computation and it buys the property this essay is about: the p-value means what it says, without relying on a distributional assumption that a small sample cannot check.

That makes it the natural default for small samples, where the assumptions are least checkable and the approximations least reliable. It is not a niche technique; it is the honest answer to the question of how the null distribution is known, and the answer is that it is constructed.

The test that cannot be flat, however correct it is #

Everything above assumes a test whose p-value can take any value in the unit interval. A large class of perfectly correct tests cannot, and the discipline has to be stated differently for them.

The exact binomial test is the clearest case. With a sample of twenty there are twenty-one possible counts, so the test can produce at most twenty-one distinct p-values. A distribution supported on twenty-one points cannot be uniform on a continuum. The histogram is a picket fence, and no amount of correctness will smooth it.

The consequence is measurable, and it is not small. Summing over the sample space rather than simulating, the exact binomial test at a nominal 5% has a true rejection rate of:

• 2.15% at n = 10
• 4.14% at n = 20
• 3.28% at n = 50

Not 5%, and not converging tidily either — the value at fifty is further from nominal than the value at twenty, for the same discreteness reason the coverage curve oscillates.

The test is not wrong. It is conservative: it rejects less often than advertised, so its error rate is below the level rather than above it, which is the safe direction. What it is not is flat, and a flatness check applied to it naively would report a failure that is actually a property of the sample space.

What the discipline becomes for a discrete test #

The right generalisation is that the p-value must be stochastically at least uniform: the probability of a p-value at or below any threshold must not exceed that threshold. For a continuous test that condition is equality and gives flatness. For a discrete test it is an inequality, satisfied with room to spare, and the picket fence is what satisfying it looks like.

Stated that way the check still has teeth, and it still catches the errors it was written for. A wrong tail, a doubled p-value, a mistaken degrees of freedom all push the distribution above the diagonal somewhere, and that is a failure under the inequality just as it is under equality.

What changes is the reading of the gap. For a continuous test, a distribution below the diagonal means something is wrong — a test rejecting at 3% when it claims 5% has a bug or a mis-specified null. For a discrete test the same picture is the expected and correct behaviour, and the gap is the price of exactness.

There is a device that restores flatness exactly, and it is worth knowing about even though almost nobody uses it. The randomised p-value breaks the ties by drawing a uniform within each atom, which makes the distribution exactly uniform and the test exactly the stated size. It is not used because a test whose conclusion depends on an auxiliary coin flip is unacceptable to report, which is a good reason. But its existence pins down what is being given up: the conservatism is not an approximation error, it is the cost of refusing to randomise.

The practical rule that follows is narrow and useful. Where a test is discrete, its nominal level is an upper bound rather than a description, the true level should be computed by summing the sample space rather than assumed, and any comparison of methods should be made at the true level rather than the claimed one — otherwise a conservative method is being credited with an error rate it never spends.

Where this check sits among the others #

Flatness is the first check this site runs, and its position is not sentimental. It is the check with the widest blast radius: if the p-values are not right, then every rejection rate is meaningless, every power curve is measuring the wrong thing, and every essay that quotes a proportion of significant results is quoting a number about a broken test rather than about the world. Nothing downstream can be trusted ahead of it.

That is the argument for running it first, and it is also the argument for running it against something known to be wrong, which is why the gate feeds it a p-value halved by a one-sided-versus-two-sided slip and requires the refusal.