The shortest interval is the one that misses

Given two intervals for the same quantity, the narrower one looks better. It is more informative, it makes a sharper claim, and it produces a more publishable sentence. That instinct is exactly backwards unless the coverage has been checked first.

The degenerate case #

The instinct is easy to break. Consider the interval procedure that reports the single point 0.5 every time, regardless of the data.

It is the narrowest interval possible — zero width. It is completely uninformative about the sample, entirely reproducible, and its coverage is zero unless the truth happens to be 0.5.

Nobody would use it, and the reason nobody would use it is that width without coverage means nothing. The moment that is stated the general principle follows: an interval is a pair of properties, and reporting one of them alone is not a summary of anything.

The four, measured #

At n = 30 with a true proportion of 0.15:

• Wald is the narrowest and covers 94.2%
• Wilson and Agresti–Coull are slightly wider and cover 96.5%
• Clopper–Pearson is widest and covers 98.3%

The ordering is not a coincidence. Every one of these is buying coverage with width, and the differences between them are differences in the exchange rate.

The Wald interval is short because it uses the observed proportion to estimate its own standard error, which understates the spread whenever the observed proportion is near an edge — the same mechanism that makes its coverage collapse at small samples with small proportions. Its shortness and its failure are the same fact seen twice.

The two degenerate rules, and what they bracket #

The point interval has a mirror image, and having both makes the reason width alone is not a criterion unarguable.

Report the single point 0.5. Width zero, coverage zero.

Report the whole line. Coverage one, width infinite.

Every real procedure sits between them, and each degenerate rule maximises exactly one of the two quantities. So neither width nor coverage is a criterion on its own, and any comparison that ranks procedures on one of them is ranking them on an axis whose extreme is useless.

What a point of coverage costs in width #

Within a family the exchange rate is a ratio of quantiles and is worth knowing.

Moving a rule from 95% to 99% multiplies its width by $2.576/1.960$, which is 1.31 — four points of coverage for thirty-one per cent of width. Moving from 95% to 90% saves $1 - 1.645/1.960$, or 16%.

Which prices the shaded procedure directly. An interval that under-covers by about five points is buying roughly a sixth off its width, and a reader comparing widths without coverage attached is being shown that sixth as though it were an improvement in the method.

Conservative is not free either #

Clopper–Pearson never falls below 95%, at any sample size measured here. That sounds like the safe choice and it has a cost that is easy to overlook.

An interval that covers 98.3% when it claims 95% is wider than it needs to be. A study using it has less resolution than its sample size should provide — or equivalently, needs a larger sample to reach a given precision. If the sample is people, or animals, or expensive measurements, that cost is real and someone is paying it.

There is also a subtler cost: over-coverage makes the corresponding test conservative, so it rejects less often than its stated level, which means less power. A method that never makes a false-positive error at more than its stated rate is buying that with false negatives, and those are invisible — nothing gets published to report an effect that was there and was missed.

Why “exact” is a misleading name #

Clopper–Pearson is conventionally called the exact interval, and the name causes trouble.

It is exact in the sense that it is derived from the binomial distribution directly, with no normal approximation anywhere. It is not exact in the sense of achieving its nominal coverage — it cannot, because the sample space is discrete and the available coverage levels come in jumps. It guarantees at least 95%, and lands wherever the discreteness puts it.

That distinction is the same discreteness that makes coverage oscillate with sample size. No procedure for a discrete parameter can hit 95% exactly at every n and p; the choices are to overshoot systematically, to undershoot sometimes, or to aim for the right average and accept both.

How to choose #

The measurement gives a straightforward decision rule, which is worth stating because the literature on this is larger than it needs to be.

Do not use the Wald interval. It is the shortest and it fails, and the alternatives are no harder to compute. There is no regime where it is the right choice that is not better served by Wilson.

Use Wilson or Agresti–Coull by default. They are close to nominal across most of the range, they are short, and their failures are mild.

Use Clopper–Pearson when under-coverage is unacceptable — a regulatory setting, a safety claim, anywhere the cost of a wrong interval is asymmetric. Pay the width knowingly.

And in all three cases, compute the coverage for the planned n and the plausible range of p before collecting the data. It is a loop over n + 1 outcomes. The whole argument above is reproducible in an afternoon, which is the strongest thing that can be said for it.

What an interval is actually for #

Behind the width-against-coverage trade is a question about purpose, and different purposes want different points on the curve.

An interval used to rule things out — to say a parameter is not above some value — is doing a one-sided job, and under-coverage on the side being used is the failure that matters. An interval used to summarise precision wants to be narrow and honest, and systematic over-coverage misrepresents the study as less informative than it was.

An interval used as an inverted test, which is what a regulator reads, needs its guarantee to hold in the worst case rather than on average, because the guarantee is the thing being relied on.

Those are genuinely different requirements, and no single procedure is best for all three. That is the honest reason several intervals for the same quantity survive in the literature, and it is a better reason than the one usually given, which is that statisticians cannot agree.

Average coverage against worst-case coverage #

The distinction that organises the whole comparison, and which the coverage curve makes visible.

Wilson and Agresti–Coull aim for the right coverage on average across the parameter range. They dip below 95% in places and rise above it in others, and the dips are shallow.

Clopper–Pearson guarantees coverage at every value of the parameter. It can only do that by overshooting nearly everywhere, since discreteness means the achievable levels come in jumps.

Neither is right in general. Which one to want depends on whether the cost of an occasional shortfall is linear in its size — in which case average behaviour is the right target — or whether any shortfall at all is unacceptable, in which case the guarantee is worth its width.

Stating which of those applies is a decision about consequences rather than about statistics, and it should be made by whoever bears them.

Width is not the only cost #

The comparison so far treats width as the price of coverage, which understates what is being paid.

A wider interval means less power in the corresponding test, and less power means real effects go undetected and the ones that are detected are inflated. Those costs are invisible: nothing is published to report the effect that was there and was missed.

It also means a larger sample for a given precision, and where the sample is patients or animals or expensive measurements, somebody pays for that in a currency that is not statistical.

So an over-conservative procedure is not the safe default it appears to be. It moves the cost from a visible place — an interval that occasionally misses — to an invisible one, and invisible costs are the ones that go unexamined.

The recommendation, and its limits #

Restating the practical conclusion with its scope attached.

Do not use the Wald interval for a proportion. It is the shortest, it fails, and the alternatives cost nothing extra. This is about as close to a universal recommendation as this subject offers.

Prefer Wilson or Agresti–Coull unless there is a specific reason to want a worst-case guarantee.

Compute the coverage for the planned design before collecting anything. A loop over n + 1 outcomes, which turns every argument on this page into a number for the study actually being run.

The limit on all of this: it is about a proportion. For a mean the corresponding story is the t correction and its measured coverage; for a statistic with no closed form it is the bootstrap and its failure modes. The method transfers; the specific recommendations do not.

One-sided and two-sided failure #

Coverage as a single number hides something that matters for how an interval is used.

A procedure covering 90% when it claims 95% might be missing 5% above and 5% below, or 9% above and 1% below. Those are different failures. For a two-sided question the first is tolerable and the second is skewed; for a one-sided question — is the rate below the limit? — only the miss on the relevant side counts, and a procedure can be safe in one direction and badly wrong in the other.

The Wald interval’s failure is asymmetric near the edges, because the interval is symmetric about p̂ while the sampling distribution is not. Near zero it extends below the range and gets truncated, so it under-covers from above.

Reporting coverage by side rather than in total is a small refinement and it is the right one when the interval is going to be read one-sidedly, which in safety and regulatory work it usually is.

Expected width against worst-case width #

The comparison in this essay uses expected width, averaged over the sample space. That is the right summary for planning and the wrong one for guaranteeing.

A procedure with good expected width can produce very wide intervals for particular outcomes. Clopper–Pearson at k = 0 or k = n is an example: the interval runs to the edge of the range, which is correct and is much wider than its average.

If the study will be reported once, the width that matters is the width for the outcome that occurs, and the distribution of widths is more informative than its mean. That distribution is available from the same loop that produces the coverage, at no extra cost — it is the same n + 1 intervals.

The comparison at other sample sizes #

The figures fix n = 30, and the ordering is not fixed with it.

At very small n the differences are large: Wald’s under-coverage is severe and Clopper–Pearson’s conservatism is extreme, so the middle options dominate clearly.

At large n everything converges. By n = 500 all four intervals are close to nominal and close to each other, and the choice stops mattering.

The awkward region is the middle — roughly n between 20 and 200 — which is where a great many real studies live, and where the oscillation means the ordering can change from one n to the next.

That is the argument for computing rather than choosing by reputation: the ranking is not stable across the region where the choice matters.

What generalises #

The specific recommendations here are about a proportion. The reasoning is not, and it is worth extracting.

Coverage and width are a pair, and reporting either alone is not a summary of a procedure. This holds for any interval, for any parameter.

A procedure derived by substituting an estimate into a formula that assumed the true value will be too confident, and the error is worst where the estimate is least reliable. The t correction is the same problem solved properly for a different parameter.

And where the parameter is discrete, exact nominal coverage is unattainable, so the choice is which direction to absorb the discreteness in. That is a decision about consequences, and it belongs to whoever bears them.

The trade, stated as a single question #

The four methods differ in coverage and in width, and comparing them on either axis alone produces the wrong ranking. The comparison that makes sense is the one a study actually faces: at the sample size available, which method delivers the stated level, and how much resolution does it cost to get there?

Put that way the ordering is stable and the reasoning is short.

The Wald interval is shortest and does not deliver the level, so its shortness is not a saving — it is the level being spent rather than paid. An interval that covers 87.6% while claiming 95% is a 87.6% interval that has been mislabelled, and the correct comparison is against other 87.6% intervals, which would all be narrower still.

Clopper–Pearson delivers the level and more, and pays in width for the excess. Its coverage never falls below nominal, which is what “exact” means here, and the cost is that a study using it has less resolution than its sample size should provide.

Wilson delivers the level to within a point or two and is barely wider than Wald. On the question as posed, it wins at essentially every sample size, which is why the recommendation in the essay is as flat as it is.

Where the recommendation stops applying #

Three situations reverse it, and they are worth naming because a recommendation without a boundary is an instruction rather than an argument.

Where the shortfall is not symmetric in its consequences. If undercoverage on the low side is a safety question and undercoverage on the high side is not — a contamination rate, a failure probability, a dose — then a one-sided guarantee is what is wanted, and Clopper–Pearson’s conservatism is buying exactly the thing that matters. Paying in width for a guaranteed bound is rational when the bound is what the decision turns on.

Where the interval will be combined with others. An interval that is a couple of points below nominal is fine in isolation and compounds when twenty of them are reported together, which is the multiplicity problem arriving through a different door. A conservative method’s excess coverage is a buffer in that setting rather than waste.

Where the count is at or near zero. Every method’s behaviour is dominated by the boundary there, and the general comparison stops being informative. The rule of three is the honest summary — no events in n trials is consistent with a rate up to about 3/n — and the choice among interval formulae matters far less than not reporting the point estimate of zero.

Outside those three, the essay’s recommendation stands, and it is unusually cheap to act on: the better method is the same arithmetic with the hypothesised proportion left in the denominator instead of the observed one.

Why width is so rarely reported beside coverage #

A structural observation, since the omission is near-universal and not an accident.

Coverage is a property of a procedure and is the same for everyone using it, so it can be tabulated once in a methods paper and cited thereafter. Width depends on the data — on the sample size, on the observed proportion, on the spread — so it cannot be tabulated in the same way and has to be computed for the case at hand.

That asymmetry means coverage travels and width does not. A method acquires a reputation for being correct, the correctness is carried in a citation, and the cost that correctness was bought with stays behind in the original paper.

The consequence is a literature in which conservative methods look strictly better than they are, because the axis on which they lose is the one nobody carries. An interval that never falls below its nominal level sounds like a free improvement, and the phrase that hides the cost is “exact” — a word that means “derived from the discrete distribution rather than an approximation to it” and is heard as “correct”.

Reporting the two together is not hard. Every figure on this page does it, and the resulting comparison is the one that supports a decision rather than a preference.