Normal Distribution
nothing is called 95% until it has been counted
Intervals, counted

The shortest interval is the one that misses

Four intervals for the same data, with their widths and their coverage measured together. The narrowest is the one that fails its stated level, which is exactly why it looks the most appealing.
11 min read 6 figures Count it, do not claim itTwo routes to a number

Given two intervals for the same quantity, the narrower one looks better. It is more informative, it makes a sharper claim, and it produces a more publishable sentence. That instinct is exactly backwards unless the coverage has been checked first.

Expected width against coverage, n = 30, p = 0.15The Wald interval is the shortest and covers 94.2%. Clopper–Pearson covers 98.3% and is 13% wider. Shortness is not a virtue on its own — an interval of zero width is the shortest of all.Wald0.241covers 94.2%Wilson0.247covers 96.5%Agresti–Coull0.259covers 96.5%Clopper–Pearson0.273covers 98.3%expected width, and what it buysorange: fails its nominal levelthe shortest interval is the one that misses
Fig. 1 Expected width of four interval procedures on the same data, with the coverage each one achieves. The shortest is shaded, because it does not reach its nominal level.

The degenerate case

The instinct is easy to break. Consider the interval procedure that reports the single point 0.5 every time, regardless of the data.

It is the narrowest interval possible — zero width. It is completely uninformative about the sample, entirely reproducible, and its coverage is zero unless the truth happens to be 0.5.

Nobody would use it, and the reason nobody would use it is that width without coverage means nothing. The moment that is stated the general principle follows: an interval is a pair of properties, and reporting one of them alone is not a summary of anything.

Coverage of four nominal 95% intervals, n = 30Computed exactly by summing over all 31 possible counts, not simulated. The Wald interval drops to 26.0% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted
Fig. 2 The coverage each of the four achieves across the range, computed exactly.

The four, measured

At n = 30 with a true proportion of 0.15:

  • Wald is the narrowest and covers 94.2%
  • Wilson and Agresti–Coull are slightly wider and cover 96.5%
  • Clopper–Pearson is widest and covers 98.3%

The ordering is not a coincidence. Every one of these is buying coverage with width, and the differences between them are differences in the exchange rate.

The Wald interval is short because it uses the observed proportion to estimate its own standard error, which understates the spread whenever the observed proportion is near an edge — the same mechanism that makes its coverage collapse at small samples with small proportions. Its shortness and its failure are the same fact seen twice.

Coverage against sample size, true proportion 0.15Coverage does not improve monotonically. n = 19 covers 93.8% while the larger n = 20 covers 81.9%. The sample space is discrete, so the endpoints jump as n changes.0.8000.8500.9000.950255075100sample sizecoverage at a true proportion of 0.15n = 19n = 20WilsonWaldexact coverage at every n from 10 to 120more data is not automatically better here
Fig. 3 The exact method never drops below its nominal level, and the discreteness that forces the overshoot is visible along this axis.

Conservative is not free either

Clopper–Pearson never falls below 95%, at any sample size measured here. That sounds like the safe choice and it has a cost that is easy to overlook.

An interval that covers 98.3% when it claims 95% is wider than it needs to be. A study using it has less resolution than its sample size should provide — or equivalently, needs a larger sample to reach a given precision. If the sample is people, or animals, or expensive measurements, that cost is real and someone is paying it.

There is also a subtler cost: over-coverage makes the corresponding test conservative, so it rejects less often than its stated level, which means less power. A method that never makes a false-positive error at more than its stated rate is buying that with false negatives, and those are invisible — nothing gets published to report an effect that was there and was missed.

Twenty 95% intervals for a proportion that really is 0.350 of the twenty miss the true value. The 95% is a property of the procedure across repetitions — no single interval has a 95% chance of anything, because it either contains 0.35 or it does not.the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval
Fig. 4 Intervals from twenty samples. Discreteness means the achievable coverage comes in jumps rather than landing on 95%.

Why “exact” is a misleading name

Clopper–Pearson is conventionally called the exact interval, and the name causes trouble.

It is exact in the sense that it is derived from the binomial distribution directly, with no normal approximation anywhere. It is not exact in the sense of achieving its nominal coverage — it cannot, because the sample space is discrete and the available coverage levels come in jumps. It guarantees at least 95%, and lands wherever the discreteness puts it.

That distinction is the same discreteness that makes coverage oscillate with sample size. No procedure for a discrete parameter can hit 95% exactly at every n and p; the choices are to overshoot systematically, to undershoot sometimes, or to aim for the right average and accept both.

Coverage of a 95% interval for a mean, n = 8Measured over 20,000 samples. The t interval covers 95.0% and the z interval 91.3%. The difference is the price of pretending the standard deviation was known.t interval, 7 df95.0%± 0.3 at 2 s.e.z interval91.3%± 0.4 at 2 s.e.20,000 samples, one seed eachnominal 95%counted, not assumed
Fig. 5 The same trade for a mean: the shorter interval is the one that fails to cover.
Where the bootstrap works and where it does notUniform data on [0, 1]. For the mean the percentile bootstrap covers 93.6%. For the maximum it covers 0.0%, because a resample can never contain a value larger than the largest one observed, so the interval cannot reach above it.the sample mean93.6%nominal 95%the sample maximum0.0%nominal 95%3,000 samples of 40the same procedure, two statisticsresampling cannot see past the data
Fig. 6 And a procedure whose width looks reasonable while its coverage has collapsed entirely.

How to choose

The measurement gives a straightforward decision rule, which is worth stating because the literature on this is larger than it needs to be.

Do not use the Wald interval. It is the shortest and it fails, and the alternatives are no harder to compute. There is no regime where it is the right choice that is not better served by Wilson.

Use Wilson or Agresti–Coull by default. They are close to nominal across most of the range, they are short, and their failures are mild.

Use Clopper–Pearson when under-coverage is unacceptable — a regulatory setting, a safety claim, anywhere the cost of a wrong interval is asymmetric. Pay the width knowingly.

And in all three cases, compute the coverage for the planned n and the plausible range of p before collecting the data. It is a loop over n + 1 outcomes. The whole argument above is reproducible in an afternoon, which is the strongest thing that can be said for it.

What an interval is actually for

Behind the width-against-coverage trade is a question about purpose, and different purposes want different points on the curve.

An interval used to rule things out — to say a parameter is not above some value — is doing a one-sided job, and under-coverage on the side being used is the failure that matters. An interval used to summarise precision wants to be narrow and honest, and systematic over-coverage misrepresents the study as less informative than it was.

An interval used as an inverted test, which is what a regulator reads, needs its guarantee to hold in the worst case rather than on average, because the guarantee is the thing being relied on.

Those are genuinely different requirements, and no single procedure is best for all three. That is the honest reason several intervals for the same quantity survive in the literature, and it is a better reason than the one usually given, which is that statisticians cannot agree.

Average coverage against worst-case coverage

The distinction that organises the whole comparison, and which the coverage curve makes visible.

Wilson and Agresti–Coull aim for the right coverage on average across the parameter range. They dip below 95% in places and rise above it in others, and the dips are shallow.

Clopper–Pearson guarantees coverage at every value of the parameter. It can only do that by overshooting nearly everywhere, since discreteness means the achievable levels come in jumps.

Neither is right in general. Which one to want depends on whether the cost of an occasional shortfall is linear in its size — in which case average behaviour is the right target — or whether any shortfall at all is unacceptable, in which case the guarantee is worth its width.

Stating which of those applies is a decision about consequences rather than about statistics, and it should be made by whoever bears them.

Width is not the only cost

The comparison so far treats width as the price of coverage, which understates what is being paid.

A wider interval means less power in the corresponding test, and less power means real effects go undetected and the ones that are detected are inflated. Those costs are invisible: nothing is published to report the effect that was there and was missed.

It also means a larger sample for a given precision, and where the sample is patients or animals or expensive measurements, somebody pays for that in a currency that is not statistical.

So an over-conservative procedure is not the safe default it appears to be. It moves the cost from a visible place — an interval that occasionally misses — to an invisible one, and invisible costs are the ones that go unexamined.

The recommendation, and its limits

Restating the practical conclusion with its scope attached.

Do not use the Wald interval for a proportion. It is the shortest, it fails, and the alternatives cost nothing extra. This is about as close to a universal recommendation as this subject offers.

Prefer Wilson or Agresti–Coull unless there is a specific reason to want a worst-case guarantee.

Compute the coverage for the planned design before collecting anything. A loop over n + 1 outcomes, which turns every argument on this page into a number for the study actually being run.

The limit on all of this: it is about a proportion. For a mean the corresponding story is the t correction and its measured coverage; for a statistic with no closed form it is the bootstrap and its failure modes. The method transfers; the specific recommendations do not.

One-sided and two-sided failure

Coverage as a single number hides something that matters for how an interval is used.

A procedure covering 90% when it claims 95% might be missing 5% above and 5% below, or 9% above and 1% below. Those are different failures. For a two-sided question the first is tolerable and the second is skewed; for a one-sided question — is the rate below the limit? — only the miss on the relevant side counts, and a procedure can be safe in one direction and badly wrong in the other.

The Wald interval’s failure is asymmetric near the edges, because the interval is symmetric about p̂ while the sampling distribution is not. Near zero it extends below the range and gets truncated, so it under-covers from above.

Reporting coverage by side rather than in total is a small refinement and it is the right one when the interval is going to be read one-sidedly, which in safety and regulatory work it usually is.

Expected width against worst-case width

The comparison in this essay uses expected width, averaged over the sample space. That is the right summary for planning and the wrong one for guaranteeing.

A procedure with good expected width can produce very wide intervals for particular outcomes. Clopper–Pearson at k = 0 or k = n is an example: the interval runs to the edge of the range, which is correct and is much wider than its average.

If the study will be reported once, the width that matters is the width for the outcome that occurs, and the distribution of widths is more informative than its mean. That distribution is available from the same loop that produces the coverage, at no extra cost — it is the same n + 1 intervals.

The comparison at other sample sizes

The figures fix n = 30, and the ordering is not fixed with it.

At very small n the differences are large: Wald’s under-coverage is severe and Clopper–Pearson’s conservatism is extreme, so the middle options dominate clearly.

At large n everything converges. By n = 500 all four intervals are close to nominal and close to each other, and the choice stops mattering.

The awkward region is the middle — roughly n between 20 and 200 — which is where a great many real studies live, and where the oscillation means the ordering can change from one n to the next.

That is the argument for computing rather than choosing by reputation: the ranking is not stable across the region where the choice matters.

What generalises

The specific recommendations here are about a proportion. The reasoning is not, and it is worth extracting.

Coverage and width are a pair, and reporting either alone is not a summary of a procedure. This holds for any interval, for any parameter.

A procedure derived by substituting an estimate into a formula that assumed the true value will be too confident, and the error is worst where the estimate is least reliable. The t correction is the same problem solved properly for a different parameter.

And where the parameter is discrete, exact nominal coverage is unattainable, so the choice is which direction to absorb the discreteness in. That is a decision about consequences, and it belongs to whoever bears them.

Where this goes next

0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted What the 95% refers to An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time. 0.8000.8500.9000.950255075100sample sizecoverage at a true proportion of 0.15n = 19n = 20WilsonWaldexact coverage at every n from 10 to 120more data is not automatically better here More data is not monotonically better Coverage of an interval for a proportion does not improve smoothly as the sample grows. It oscillates, and there are larger samples that cover materially worse than smaller ones — a sample of twenty covers twelve points worse than a sample of nineteen. the sample mean93.6%nominal 95%the sample maximum0.0%nominal 95%3,000 samples of 40the same procedure, two statisticsresampling cannot see past the data Where the bootstrap lies Resampling is the most generally useful trick in the subject and it has a failure mode that is easy to state: it cannot see past the data. For a statistic that lives at the edge of the sample, coverage collapses from 95% to almost nothing.