What the 95% refers to
A 95% confidence interval is the most-used object in applied statistics and the most misdescribed. The usual gloss — that the parameter lies inside it with probability 0.95 — is wrong in a way that matters, and the correct statement is both simpler and more useful: 95% is a property of the procedure, and it can be counted.
What the number attaches to
Fix a true proportion — say a coin that really does come up heads 30% of the time. Take a sample of twenty tosses, count the heads, and build an interval from that count by some stated rule. Do it again with a fresh sample, and again.
Some of those intervals contain 0.3 and some do not. The coverage of the rule is the long-run fraction that do. If the rule is any good, that fraction is 0.95.
Notice what the 95% is attached to. It is not attached to any particular interval: the interval either contains 0.3 or it does not, and there is no randomness left in it once the sample is drawn. It is attached to the rule for building intervals, evaluated across the samples that could have been drawn.
That distinction sounds pedantic and it is the reason coverage can be checked. A statement about one interval is not testable, because there is nothing to repeat. A statement about a procedure is testable by running the procedure.
Counting rather than simulating
For a proportion there is a shortcut worth taking, because it removes the last excuse for not checking.
With a sample of n tosses there are only n + 1 possible outcomes: zero heads, one head, up to n. Each has a probability given exactly by the binomial formula, and each produces exactly one interval. So the coverage is a finite sum — go through every possible count, ask whether its interval contains the truth, and add up the probabilities of the ones that do.
No simulation, no seed, no Monte Carlo error. The number is exact.
That matters more than it might seem. The effects on this page are a few percentage points wide, and a simulation of any affordable size would smear them into something a reader could dismiss as noise. Computing the sum exactly means every wobble on the curve above is real.
The interval that fails
The standard interval for a proportion — the one in every introductory course, and the one every spreadsheet computes — takes the observed proportion and adds and subtracts 1.96 standard errors:
It is called the Wald interval. At a sample of twenty with a true proportion of 0.1, its coverage is 87.6%. Not 95%, and not close to it: one interval in eight misses, where the rule promises one in twenty.
The Wilson interval, computed from the same data with arithmetic no harder, covers 95.7%.
Neither of those numbers is an estimate. Both are sums over the twenty-one possible outcomes, and anyone can reproduce them with a spreadsheet and an afternoon.
Why it fails where it does
The failure is not random and it is not uniform across the range. It is worst where the proportion is near zero or one, and that is precisely where proportions are most often reported — a side-effect rate, a defect rate, a conversion rate, a rare event in a small sample.
Two things go wrong at once.
The standard error is computed from the observed proportion rather than the true one. When the observed proportion is small, the estimated standard error is small, so the interval is narrow — narrow because the estimate happened to be low, which is exactly the case where the estimate is furthest from the truth.
And when the count is zero, the interval collapses to the single point zero. It has no width at all. It cannot contain any true proportion other than zero, so every sample that yields no successes produces an interval that is certainly wrong. At n = 20 and p = 0.1 that happens 12% of the time on its own.
The Wilson interval avoids both by inverting the test rather than plugging in an estimate: it asks which true proportions would not have been rejected by the data, which is what a confidence interval was defined to be in the first place.
What “exact” costs
There is an interval that never falls below 95%: Clopper–Pearson, built directly from the binomial distribution rather than from any approximation to it. It is usually called exact, and its coverage across every sample size measured here never dips below the nominal level.
It achieves that by being conservative. At n = 20 and p = 0.1 it covers 98.9%, which sounds better than 95% until what it means becomes clear: the interval is wider than it needs to be, so a study using it has less resolution than its sample size should provide. Coverage above the nominal level is not free; it is paid for in width, and width is paid for in sample size.
What each method pays is a measurement in its own right, and it produces the observation that makes the whole comparison intelligible: the Wald interval is the shortest of the four. Shortness is why it looks good, and shortness is why it misses.
The rule this site runs on
Everything above follows from one decision, which is the site’s rule and is applied without exception:
Nothing is called 95% until its coverage has been counted.
It is a low bar. The computation is a loop over n + 1 outcomes; it takes a few lines and runs instantly. And it is almost never done in teaching material, which is why a method that fails its own stated level by seven percentage points has been taught first for a century.
The same discipline applies to tests. A test at α = 0.05 claims to reject a true null 5% of the time, and that claim can be counted too — and more searchingly, since the whole distribution of p-values under the null must be flat, not merely the fraction below 0.05.
What this does not say
Three things this essay is not claiming, since the subject punishes loose statements.
It is not saying the Wald interval is always wrong. At large n away from the edges it is fine, and the curve shows it. The claim is that it fails badly in a region that is common in practice, and that nothing in its presentation gives warning.
It is not saying Wilson is optimal. It is one of several better choices, and it has its own dips — visible on the curve at some sample sizes.
And it is not saying that a wider interval is a better one. An interval covering 100% of the time is available at no cost: report [0, 1] every time. Coverage alone is never the whole criterion, which is why width is measured beside it.
Where this goes next
The natural expectation is that these problems disappear with more data, and they largely do — but not the way anyone assumes. Coverage does not improve monotonically as the sample grows. There are sample sizes that cover worse than smaller ones, by twelve percentage points, and the reason is not subtle once seen.
Where the better intervals come from
The essay has said Wilson’s interval inverts the test rather than plugging in an estimate. That deserves to be shown, because it turns a list of named methods into one idea.
A confidence interval is defined as the set of parameter values that would not be rejected by the data at the corresponding level. To build one, take the test, and solve for which values of p pass it.
The test statistic for a proportion is (p̂ − p) / √(p(1 − p)/n). Notice that the denominator contains the hypothesised p, not the observed p̂. Setting the statistic equal to ±1.96 and solving the resulting quadratic for p gives Wilson’s interval directly.
The Wald interval is what comes of replacing p with p̂ in the denominator first, so the equation becomes linear and the solution is symmetric about p̂. That substitution is the entire difference between the two methods, and it is the source of the failure: near an edge, p̂ is a poor stand-in for p, and the interval it produces is centred and scaled wrongly.
So the family of methods is not a collection of tricks. There is one construction, and the worse methods are approximations to it made for arithmetic convenience at a time when solving a quadratic by hand mattered.
What happens at zero
The most instructive single case, and one that comes up constantly in practice.
A sample of twenty with no successes. The Wald interval is [0, 0], a single point — the standard error is √(0 × 1 / 20) = 0, so the interval has no width. It asserts that the true proportion is exactly zero, on the basis of twenty observations.
Wilson gives roughly [0, 0.16]. Clopper–Pearson gives [0, 0.17].
The rule of three is the useful approximation here: with no events in n trials, the upper 95% limit is about 3/n. Twenty observations with no failures is consistent with a true failure rate up to 15%, which is a very different message from “zero”.
That case alone is a sufficient argument against the Wald interval, and it is common enough in safety and reliability work that the rule of three is worth memorising on its own.
What coverage does not measure
Coverage is the property this site insists on counting, so it is worth being clear about what it leaves out.
It says nothing about where the interval sits when it misses. Two procedures with equal coverage can fail differently — one missing symmetrically, the other always missing on the same side. For a one-sided question the second is much worse, and coverage alone cannot distinguish them.
It says nothing about width, which is the other half of the trade and the reason an interval of [0, 1] is not the answer.
And it is a frequentist property, evaluated across repetitions that did not happen. A reader who wants a probability statement about the parameter given this data is asking for a credible interval, which needs a prior and answers a different question.
Reporting coverage is therefore necessary and not sufficient, which is why the figures here report width beside it and why the essays keep returning to what the number attaches to.
The history of the misreading
The wrong gloss is old enough to have a documented origin, and knowing it explains why it is so persistent.
Neyman, who introduced confidence intervals in 1937, was explicit that the probability attaches to the procedure and not to any realised interval. He was equally explicit that this was a deliberate restriction: the machinery was built to avoid needing a prior, and the price of avoiding one is that no probability statement about the parameter is available.
The misreading persists because the restricted statement is not what anyone wants. A reader wants to know what the data says about the parameter, and the frequentist interval answers a question about repetitions instead. Faced with an answer to the wrong question, people read it as an answer to theirs.
That is not carelessness. It is the predictable result of a method whose output is one step removed from the quantity of interest, and it is why the Bayesian alternative keeps being proposed — a credible interval does make the statement people want, at the cost of requiring a prior.
Neither is wrong. What is wrong is reporting one and reading the other, which is the default.
Coverage under a misspecified model
Everything measured here assumes the data really is binomial — independent trials with a constant probability. When that fails, coverage fails with it, and in a way no interval formula can repair.
Clustering. Responses grouped by household, clinic or class are correlated, so the effective number of independent trials is smaller than the count. Every interval above becomes too narrow, and the shortfall is proportional to the intraclass correlation. This is the commonest violation in survey and trial data.
A varying probability. If p differs across the sample, the count is overdispersed relative to binomial, and the same narrowing occurs.
Selection. If the sample is not drawn as assumed, the parameter being covered is not the one intended, and coverage of the wrong quantity is not a consolation.
Each of those is a larger effect than the difference between Wald and Wilson, which is worth keeping in proportion. The essay’s recommendation stands — the better interval costs nothing — but a study with clustered data and a perfect interval formula is in worse shape than one with independent data and the textbook formula.
The order of worry should be: is the model right, then is the procedure right, then which procedure.