Normal Distribution
nothing is called 95% until it has been counted
Intervals, counted

More data is not monotonically better

Coverage of an interval for a proportion does not improve smoothly as the sample grows. It oscillates, and there are larger samples that cover materially worse than smaller ones — a sample of twenty covers twelve points worse than a sample of nineteen.
11 min read 6 figures Count it, do not claim itReversals that are nobody's mistake

Everyone believes that a bigger sample is better. For estimating a proportion with a standard interval, that belief is false in a specific and checkable way: coverage goes up and down as the sample grows, and there are larger samples that do worse.

Coverage against sample size, true proportion 0.15Coverage does not improve monotonically. n = 19 covers 93.8% while the larger n = 20 covers 81.9%. The sample space is discrete, so the endpoints jump as n changes.0.8000.8500.9000.950255075100sample sizecoverage at a true proportion of 0.15n = 19n = 20WilsonWaldexact coverage at every n from 10 to 120more data is not automatically better here
Fig. 1 Exact coverage against sample size at a fixed true proportion. Not a trend with noise on it — the computation carries no noise at all — but a genuinely jagged function.

The measurement

Fix the true proportion at 0.15 and compute the exact coverage of the standard interval at every sample size from ten to a hundred and twenty. Because the sample space is finite, each of those numbers is a sum rather than an estimate.

The result: a sample of nineteen covers 93.8%, and a sample of twenty covers 81.9%. One extra observation costs twelve percentage points of coverage.

That is not a fluke of one pair. The curve is jagged throughout, with dips scattered across the range, and the size of the dips does not shrink as fast as intuition expects.

Coverage of four nominal 95% intervals, n = 30Computed exactly by summing over all 31 possible counts, not simulated. The Wald interval drops to 26.0% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted
Fig. 2 Coverage against the true proportion at one sample size. The jaggedness across p is the same discreteness seen along a different axis.

Why it happens

The cause is discreteness, and it is entirely mechanical once seen.

The data can only take n + 1 values. Each of those values produces one interval, and whether the truth falls inside it is a yes-or-no matter. So the coverage is a sum of a handful of binomial probabilities — the ones whose intervals happen to contain 0.15.

Now increase n by one. Every interval endpoint moves, slightly, and the binomial probabilities all change. Somewhere in the sum, an outcome that used to cover stops covering — and that outcome may carry several percentage points of probability. It drops out of the sum in one step.

Coverage therefore changes in jumps as n increases, not smoothly. Which way it jumps depends on where the endpoints happen to fall relative to the truth, and that is close to arbitrary.

The name in the literature is “lucky n”: some sample sizes happen to place their endpoints well for a given true proportion, and their neighbours do not.

Twenty 95% intervals for a proportion that really is 0.350 of the twenty miss the true value. The 95% is a property of the procedure across repetitions — no single interval has a 95% chance of anything, because it either contains 0.35 or it does not.the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval
Fig. 3 Twenty intervals at one sample size. The count of misses is itself random, which is why a picture like this cannot check a procedure.

What the oscillation is not

It is worth separating this from several things it resembles.

It is not simulation noise. These are exact sums. Running the computation again gives identical numbers to the last digit.

It is not a failure of the underlying theory. The asymptotic argument behind the interval is correct: as n grows without bound the coverage does converge to 95%. The convergence is simply not monotone, and the asymptotic statement says nothing about any particular n.

It is not confined to one true proportion. Fix a different p and the curve is jagged in a different place. There is no sample size that is safe for every p, which is the practical difficulty: p is unknown in advance, which is why anyone is building an interval.

Expected width against coverage, n = 30, p = 0.15The Wald interval is the shortest and covers 94.2%. Clopper–Pearson covers 98.3% and is 13% wider. Shortness is not a virtue on its own — an interval of zero width is the shortest of all.Wald0.241covers 94.2%Wilson0.247covers 96.5%Agresti–Coull0.259covers 96.5%Clopper–Pearson0.273covers 98.3%expected width, and what it buysorange: fails its nominal levelthe shortest interval is the one that misses
Fig. 4 What the alternatives cost in width. The flatter methods are barely wider than the jagged one.

What follows for planning a study

The uncomfortable consequence is that a power or precision calculation based on the nominal coverage can be wrong in a way that more data does not fix.

A study planned at n = 20 whose true proportion is near 0.15 has an interval covering 82% and not 95%, and no amount of care in the rest of the analysis recovers that. Increasing to n = 40 helps on average and is not guaranteed to help in any particular case.

There are two honest responses and both are cheap.

Use an interval whose coverage is not jagged. The Wilson curve on the same axes is far flatter — it still oscillates, because the discreteness has not gone away, but the dips are a fraction of the size.

Or compute the coverage for the planned n and the plausible range of p, before collecting anything. This is a loop over n + 1 outcomes. It takes less time to write than the sentence describing it, and it turns a guess into a number.

Where the normal approximation converges, and where it does notRelative error against the exact binomial. At n = 1280 the error at the median is 0.96% and three sigma out it is 25.7% — a factor of 27. The tail is where the approximation is used.sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last
Fig. 5 An asymptotic guarantee arriving at very different speeds depending on where the distribution is read.
Coverage of a 95% interval for a mean, n = 8Measured over 20,000 samples. The t interval covers 95.0% and the z interval 91.3%. The difference is the price of pretending the standard deviation was known.t interval, 7 df95.0%± 0.3 at 2 s.e.z interval91.3%± 0.4 at 2 s.e.20,000 samples, one seed eachnominal 95%counted, not assumed
Fig. 6 Another finite-sample correction, with its coverage counted rather than argued.

The general shape

This is a specific instance of something that recurs across the subject, and it is worth naming because the same reasoning applies elsewhere.

An asymptotic guarantee is a statement about a limit, and says nothing about the sample in hand. The central limit theorem guarantees that the sampling distribution approaches normality; it does not say how close it is at n = 30, and the answer depends enormously on where in the distribution it is read.

The pattern is always the same: the theory is correct, the limit is real, and the question of what happens at a finite sample size is a separate question that has to be answered by computing rather than by appealing to the theorem.

That is why every claim on this site carries a sample size and a measurement beside it, and why the coverage of every interval here has been counted rather than quoted.

The same effect at a fixed sample size

The oscillation has a companion along the other axis, and seeing both makes the cause unmistakable.

Hold the sample size fixed and sweep the true proportion instead: the coverage curve is jagged there too, with the same sawtooth character. That is the figure at the top of the coverage essay, and the two pictures are the same surface cut in perpendicular directions.

The surface is jagged because coverage is a sum of a discrete set of terms, and moving either n or p changes which terms are included. There is no smoothing available, because there is nothing continuous underneath: the data really can only take n + 1 values.

Why the asymptotic argument is still correct

None of this contradicts the theory, and it is worth being explicit about why.

The theorem says coverage converges to 95% as n grows without bound. Convergence does not require monotonicity — a sequence can approach a limit while oscillating, provided the oscillations shrink. These do: the envelope of the sawtooth narrows as n grows, and by n = 500 the dips are a fraction of a percentage point.

So the theory is right, the measurement is right, and the gap between them is entirely in what “eventually” means. The asymptotic statement is about a limit; a study is conducted at a specific n; and nothing in the theorem says how large n must be for the guarantee to be usable.

That gap is the recurring subject of this site. The central limit theorem has exactly the same structure — correct in the limit, and silent about the finite case that is the only one anyone is ever in.

The size of the effect, in context

A twelve-point shortfall sounds dramatic and it is worth calibrating against what else can go wrong.

Twelve percentage points of coverage is roughly the difference between a 95% interval and an 83% one — a rate of missing that is one in six rather than one in twenty. If a regulatory submission, a safety claim or a quality-control limit is built on the nominal level, that is a substantial mis-statement of the risk being run.

It is also, importantly, invisible. The interval looks the same. Nothing in the output signals that this particular n and this particular p landed in a dip. The only way to find out is to compute the coverage, which is why the computation is worth its few lines.

A note on what “exact” methods do here

Clopper–Pearson is often recommended as the way out, and it does avoid undercoverage entirely — its curve never falls below 95% at any n.

It avoids it by overshooting, and the overshoot oscillates too: sometimes it covers 95.1% and sometimes 99%. The discreteness has not gone away; the procedure has simply chosen to absorb it in one direction.

That is a defensible choice and it is not free. The width it costs is real, and it buys a guarantee whose value depends entirely on whether an occasional shortfall would matter.

The alternative — Wilson or Agresti–Coull — accepts small dips in exchange for being close to nominal on average and much narrower. Which is right is a question about consequences, and it can only be asked once the coverage has been computed for both.

What to do when the proportion is unknown

The practical difficulty stated at the top deserves a resolution rather than just a warning.

The coverage depends on the true p, which is unknown. But it does not have to be known exactly — the useful computation is over a range.

Take the plausible range for the study, compute the coverage across it at the planned n, and report the worst case. That converts an unknown into a bound, and the bound is what a design decision needs.

For a proportion expected somewhere between 5% and 20%, at n = 50, the Wald interval’s worst coverage across that range is a number that can be computed in a few lines before anything is collected. If it is unacceptable, either the sample size or the method changes — and both options are visible at the point where they are still free.

Why this is not usually taught

The oscillation is well documented in the statistical literature — Brown, Cai and DasGupta laid it out thoroughly in 2001 — and it is nearly absent from teaching material. The reasons are worth naming because they explain a general pattern.

It is inconvenient. A course that has just derived the standard interval does not want to follow it with a demonstration that it fails.

It requires computation to see. The result cannot be derived on a blackboard; it comes from evaluating a sum at many values of n, which is a computer exercise and not a proof.

And it contradicts a good heuristic. More data is better is right almost everywhere, and a course has limited room for exceptions.

The result is that a generation of practitioners uses an interval whose failure mode was characterised in a well-cited paper a quarter of a century ago. That is not unusual; it is the normal lag between a technical literature and the practice it describes, and it is a fair argument for sites like this one.

The equivalent for a mean

Fairness requires noting that the mean does not have this problem, and why.

For a continuous parameter with continuous data, the sampling distribution is continuous and the coverage varies smoothly with n. There is no sample space to be discrete about, so the t interval’s coverage approaches its nominal level from one side and stays there.

The oscillation is therefore specific to discrete data — proportions, counts, rates — and it is worth knowing that the family of parameters it affects is exactly the family most often reported in medicine, quality control and survey work.

That is an unfortunate coincidence rather than a deep fact, and it is the reason this essay is about a proportion rather than a mean.

Where this goes next

0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted What the 95% refers to An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time. Wald0.241covers 94.2%Wilson0.247covers 96.5%Agresti–Coull0.259covers 96.5%Clopper–Pearson0.273covers 98.3%expected width, and what it buysorange: fails its nominal levelthe shortest interval is the one that misses The shortest interval is the one that misses Four intervals for the same data, with their widths and their coverage measured together. The narrowest is the one that fails its stated level, which is exactly why it looks the most appealing. sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last The tail converges last The central limit theorem is usually shown as a shape arriving. What the demonstration leaves out is the rate — and the rate is wildly different in the middle and in the tail, which is where every approximation in the subject is actually read.