A p-value that is not flat is not a p-value
A p-value is the probability, if the null hypothesis is true, of a statistic at least as extreme as the one observed. Read that definition carefully and a consequence falls out immediately: under the null, a p-value is the value of its own distribution function, and is therefore uniform on (0, 1).
Why flatness is the better check
The obvious check on a test is its rejection rate: run it under the null and confirm that about 5% of runs give p < 0.05. That is worth doing and it is weak.
A rejection rate checks the distribution at one point. A test can have exactly the right rate at 0.05 and be wrong everywhere else — too many small p-values compensated by too few moderate ones, a distribution that is right in the far tail and wrong in the body, or a systematic distortion that happens to cross the correct value at the threshold being checked.
Uniformity checks the whole distribution. Every quantile has to be right simultaneously, which is a far more demanding condition and correspondingly harder to satisfy by accident.
The practical difference is that flatness catches the errors people actually make:
- a wrong degrees of freedom, which shifts the whole distribution
- a one-sided p reported as two-sided, or the reverse
- a variance computed with n instead of n − 1
- a test statistic missing a scaling factor
Each of those breaks flatness immediately. Several of them can leave the rejection rate at 0.05 looking roughly acceptable.
The check, checked
An assertion that has never been shown to reject anything proves nothing, so the site’s gate feeds this check something it must refuse.
It takes the same t test and halves every p-value — the classic one-sided/two-sided slip, and a mistake that produces plausible-looking output. The rejection rate roughly doubles, which a careful reviewer might notice. The uniformity check does not need care: the distribution is crushed into the lower half of the range and the Kolmogorov–Smirnov statistic detects it overwhelmingly.
The gate requires that refusal to happen. If halving the p-values stopped being detected, the check would have quietly stopped working and the build fails.
Reading the flatness correctly
Two things the figure does not say.
A flat histogram does not mean the test is right for a given dataset. It means the implementation is consistent with its own assumptions. Here the data is genuinely normal and independent, which is exactly what the t test assumes. Feed the same correct implementation correlated data and the p-values stop being uniform — which is useful, and is a check on the assumptions rather than the code.
The flatness is a property of the null, not of the world. Under an alternative, p-values pile up near zero, and how fast they pile up is the power. The uniform case is the baseline against which that is measured, which is why the power curve is the same measurement taken at a different truth.
What this implies about a single p-value
If p-values are uniform under the null, then under the null every value is as likely as every other. A p of 0.04 and a p of 0.96 are equally probable when nothing is going on.
That has a consequence worth sitting with: a single p-value just below a threshold is weak evidence. The likelihood ratio between “there is an effect” and “there is not” at p = 0.04 is modest under most reasonable alternatives — far more modest than the word “significant” suggests.
It also explains why a distribution of p-values across many studies is so much more informative than any one of them. A field where the null is mostly true should produce a flat distribution of published p-values; a lump just below 0.05 is a signature of selection, and the size of the lump is measurable. That is not a hypothetical technique — it is how the replication literature diagnosed the problem it went on to document.
The habit
The transferable point is not about p-values. It is that the strongest checks constrain a whole distribution rather than a single summary.
The same idea appears everywhere on this site. An interval is checked by its coverage across the whole range of the parameter rather than at one value. A random generator is checked in two dimensions rather than one, because a bad one passes the one-dimensional test. A distribution function is checked against its own quantile at a dozen probabilities rather than at the median.
In each case the cheap check passes things the demanding check catches, and the demanding check costs almost nothing extra.
What flatness looks like when it fails
The check is only useful if the failures are recognisable, and the four common ones have distinct signatures.
A one-sided p reported as two-sided, or the reverse, compresses the distribution into half the interval. The histogram is flat over (0, 0.5) and empty above, or the mirror image. Unmistakable once seen and easy to miss if only the rejection rate is checked.
A wrong degrees of freedom tilts the distribution: too many small p-values if the degrees of freedom are overstated, too few if understated. The tilt is gentle and the rejection rate can still look plausible.
A variance computed with n rather than n − 1 inflates the statistic slightly, which piles up mass at the low end. At large n it is invisible; at small n it is obvious, which is the opposite of most errors.
A discrete test statistic produces a genuinely lumpy distribution, and this one is not an error — it is the same discreteness that makes interval coverage oscillate. A test on counts cannot have uniform p-values, and expecting it to is the mistake.
That last case is why the check reports a KS distance and a p-value rather than a verdict. It is a measurement, and interpreting it requires knowing whether the statistic was continuous.
Conservative and anti-conservative
The two directions of failure have very different consequences and deserve their own names.
A test whose p-values pile up above uniform is conservative: it rejects less often than its stated level, so its false positives are under control and it has less power than advertised. The costs are invisible — missed effects and inflated estimates among those it does detect.
A test whose p-values pile up below uniform is anti-conservative: it rejects more often than stated. Its false positives exceed the advertised rate, and every claim made with it is weaker than it appears.
The second is the dangerous one and the harder to notice, because a method that finds more results looks better by every casual measure. Flatness is the check that distinguishes them and it does so in one picture.
Why this is the site’s first check
The gate runs this before anything else, and the ordering is deliberate.
Every p-value on this site comes from the same t-test implementation. Every rejection rate, every power figure, the forking-paths measurement and the winner’s curse are counts of how often that implementation crosses a threshold.
If the implementation were wrong, all of those numbers would be wrong together, in a correlated way, and each would look individually plausible. Nothing downstream would catch it, because nothing downstream has an independent route to the same answer.
Uniformity is that independent route. It constrains the implementation using a fact — p-values are uniform under the null — that follows from the definition and not from the code. That is why it goes first.
Beyond testing
The pattern transfers, and naming it is more useful than the instance.
Where a quantity has a known distribution under a stated condition, checking the whole distribution is far stronger than checking one summary of it — and usually costs the same.
A residual should be normal if the model is right: check the whole set, not the mean. A random generator should be uniform: check every bin, and in two dimensions, not just the overall average. An interval procedure should cover at 95%: check across the whole parameter range, not at one convenient value.
In every case the cheap version passes things the demanding version catches, and the demanding version is a few more lines.
The check on real data
The uniformity check above uses simulated data where the null is true by construction. On real data the null is not known to be true, so the check changes character — and becomes useful in a different way.
If a set of p-values from a real study is far from uniform, one of two things holds: there are real effects, or the test’s assumptions are violated. Those are not distinguishable from the distribution alone.
But the shape carries information. Real effects produce a pile-up near zero with a roughly flat remainder. Violated assumptions — dependence, most often — produce distortions across the whole range, including too many values near one.
So the plot is worth making for a real analysis with many tests, and worth reading as a diagnostic rather than a verdict. A distribution that is flat except for a spike at zero is the picture of a well-behaved analysis with some real findings in it.
What a p-value would need to be uniform for
The condition is more specific than it is usually stated, and the qualifications are where the practical failures live.
Uniformity requires the test statistic to be continuous and its distribution under the null to be known exactly. Both fail routinely.
A statistic on counts is discrete, so its p-values take finitely many values and cannot be uniform. They are conservative — the distribution sits above uniform — which is why exact tests on small counts have less power than their nominal level suggests. That is the same discreteness that makes interval coverage oscillate.
A statistic whose null distribution is approximated rather than known — most large-sample tests — has p-values that are only approximately uniform, and the approximation degrades in the same place every approximation on this site degrades: the tail, which is where the small p-values are.
So a plot that is flat in the body and wrong below 0.01 is the expected picture for an approximate test, and it is exactly the region that matters. Checking uniformity across the whole range, rather than eyeballing the histogram, is what catches it.
The permutation alternative
Where the null distribution is not known, it can often be constructed, and the construction gives uniformity by design.
A permutation test computes the statistic under every relabelling of the data consistent with the null. The resulting distribution is the null distribution — not an approximation to it — so the p-value is exact up to the discreteness of the permutation set.
It costs computation and it buys the property this essay is about: the p-value means what it says, without relying on a distributional assumption that a small sample cannot check.
That makes it the natural default for small samples, where the assumptions are least checkable and the approximations least reliable. It is not a niche technique; it is the honest answer to the question of how the null distribution is known, and the answer is that it is constructed.