More data is not monotonically better

Everyone believes that a bigger sample is better. For estimating a proportion with a standard interval, that belief is false in a specific and checkable way: coverage goes up and down as the sample grows, and there are larger samples that do worse.

The measurement #

Fix the true proportion at 0.15 and compute the exact coverage of the standard interval at every sample size from ten to a hundred and twenty. Because the sample space is finite, each of those numbers is a sum rather than an estimate.

The result: a sample of nineteen covers 93.8%, and a sample of twenty covers 81.9%. One extra observation costs twelve percentage points of coverage.

That is not a fluke of one pair. The curve is jagged throughout, with dips scattered across the range, and the size of the dips does not shrink as fast as intuition expects.

Why it happens #

The cause is discreteness, and it is entirely mechanical once seen.

The data can only take n + 1 values. Each of those values produces one interval, and whether the truth falls inside it is a yes-or-no matter. So the coverage is a sum of a handful of binomial probabilities — the ones whose intervals happen to contain 0.15.

Now increase n by one. Every interval endpoint moves, slightly, and the binomial probabilities all change. Somewhere in the sum, an outcome that used to cover stops covering — and that outcome may carry several percentage points of probability. It drops out of the sum in one step.

Coverage therefore changes in jumps as n increases, not smoothly. Which way it jumps depends on where the endpoints happen to fall relative to the truth, and that is close to arbitrary.

The name in the literature is “lucky n”: some sample sizes happen to place their endpoints well for a given true proportion, and their neighbours do not.

The oscillation has a scale #

The jaggedness is not arbitrary, and reading its period says what is producing it.

Coverage at a fixed p changes only when the set of counts whose interval contains p changes — that is, when adding an observation moves an endpoint across 0.15. The expected count is $0.15n$, so it gains a whole unit every $1/0.15$, or about 6.7, observations.

So the pattern repeats on a scale of about seven observations at this proportion, and it would repeat every twenty at p = 0.05 and every two at p = 0.5.

That is worth knowing because it says the effect is a property of the proportion as much as of the sample size. A study at a small p has coverage that oscillates slowly, so two nearby sample sizes are likely to behave alike; a study near a half has coverage that changes with almost every observation added.

What the oscillation is not #

It is worth separating this from several things it resembles.

It is not simulation noise. These are exact sums. Running the computation again gives identical numbers to the last digit.

It is not a failure of the underlying theory. The asymptotic argument behind the interval is correct: as n grows without bound the coverage does converge to 95%. The convergence is simply not monotone, and the asymptotic statement says nothing about any particular n.

It is not confined to one true proportion. Fix a different p and the curve is jagged in a different place. There is no sample size that is safe for every p, which is the practical difficulty: p is unknown in advance, which is why anyone is building an interval.

What follows for planning a study #

The uncomfortable consequence is that a power or precision calculation based on the nominal coverage can be wrong in a way that more data does not fix.

A study planned at n = 20 whose true proportion is near 0.15 has an interval covering 82% and not 95%, and no amount of care in the rest of the analysis recovers that. Increasing to n = 40 helps on average and is not guaranteed to help in any particular case.

There are two honest responses and both are cheap.

Use an interval whose coverage is not jagged. The Wilson curve on the same axes is far flatter — it still oscillates, because the discreteness has not gone away, but the dips are a fraction of the size.

Or compute the coverage for the planned n and the plausible range of p, before collecting anything. This is a loop over n + 1 outcomes. It takes less time to write than the sentence describing it, and it turns a guess into a number.

The general shape #

This is a specific instance of something that recurs across the subject, and it is worth naming because the same reasoning applies elsewhere.

An asymptotic guarantee is a statement about a limit, and says nothing about the sample in hand. The central limit theorem guarantees that the sampling distribution approaches normality; it does not say how close it is at n = 30, and the answer depends enormously on where in the distribution it is read.

The pattern is always the same: the theory is correct, the limit is real, and the question of what happens at a finite sample size is a separate question that has to be answered by computing rather than by appealing to the theorem.

That is why every claim on this site carries a sample size and a measurement beside it, and why the coverage of every interval here has been counted rather than quoted.

The same effect at a fixed sample size #

The oscillation has a companion along the other axis, and seeing both makes the cause unmistakable.

Hold the sample size fixed and sweep the true proportion instead: the coverage curve is jagged there too, with the same sawtooth character. That is the figure at the top of the coverage essay, and the two pictures are the same surface cut in perpendicular directions.

The surface is jagged because coverage is a sum of a discrete set of terms, and moving either n or p changes which terms are included. There is no smoothing available, because there is nothing continuous underneath: the data really can only take n + 1 values.

Why the asymptotic argument is still correct #

None of this contradicts the theory, and it is worth being explicit about why.

The theorem says coverage converges to 95% as n grows without bound. Convergence does not require monotonicity — a sequence can approach a limit while oscillating, provided the oscillations shrink. These do: the envelope of the sawtooth narrows as n grows, and by n = 500 the dips are a fraction of a percentage point.

So the theory is right, the measurement is right, and the gap between them is entirely in what “eventually” means. The asymptotic statement is about a limit; a study is conducted at a specific n; and nothing in the theorem says how large n must be for the guarantee to be usable.

That gap is the recurring subject of this site. The central limit theorem has exactly the same structure — correct in the limit, and silent about the finite case that is the only one anyone is ever in.

The size of the effect, in context #

A twelve-point shortfall sounds dramatic and it is worth calibrating against what else can go wrong.

Twelve percentage points of coverage is roughly the difference between a 95% interval and an 83% one — a rate of missing that is one in six rather than one in twenty. If a regulatory submission, a safety claim or a quality-control limit is built on the nominal level, that is a substantial mis-statement of the risk being run.

It is also, importantly, invisible. The interval looks the same. Nothing in the output signals that this particular n and this particular p landed in a dip. The only way to find out is to compute the coverage, which is why the computation is worth its few lines.

A note on what “exact” methods do here #

Clopper–Pearson is often recommended as the way out, and it does avoid undercoverage entirely — its curve never falls below 95% at any n.

It avoids it by overshooting, and the overshoot oscillates too: sometimes it covers 95.1% and sometimes 99%. The discreteness has not gone away; the procedure has simply chosen to absorb it in one direction.

That is a defensible choice and it is not free. The width it costs is real, and it buys a guarantee whose value depends entirely on whether an occasional shortfall would matter.

The alternative — Wilson or Agresti–Coull — accepts small dips in exchange for being close to nominal on average and much narrower. Which is right is a question about consequences, and it can only be asked once the coverage has been computed for both.

What to do when the proportion is unknown #

The practical difficulty stated at the top deserves a resolution rather than just a warning.

The coverage depends on the true p, which is unknown. But it does not have to be known exactly — the useful computation is over a range.

Take the plausible range for the study, compute the coverage across it at the planned n, and report the worst case. That converts an unknown into a bound, and the bound is what a design decision needs.

For a proportion expected somewhere between 5% and 20%, at n = 50, the Wald interval’s worst coverage across that range is a number that can be computed in a few lines before anything is collected. If it is unacceptable, either the sample size or the method changes — and both options are visible at the point where they are still free.

Why this is not usually taught #

The oscillation is well documented in the statistical literature — Brown, Cai and DasGupta laid it out thoroughly in 2001 — and it is nearly absent from teaching material. The reasons are worth naming because they explain a general pattern.

It is inconvenient. A course that has just derived the standard interval does not want to follow it with a demonstration that it fails.

It requires computation to see. The result cannot be derived on a blackboard; it comes from evaluating a sum at many values of n, which is a computer exercise and not a proof.

And it contradicts a good heuristic. More data is better is right almost everywhere, and a course has limited room for exceptions.

The result is that a generation of practitioners uses an interval whose failure mode was characterised in a well-cited paper a quarter of a century ago. That is not unusual; it is the normal lag between a technical literature and the practice it describes, and it is a fair argument for sites like this one.

The equivalent for a mean #

Fairness requires noting that the mean does not have this problem, and why.

For a continuous parameter with continuous data, the sampling distribution is continuous and the coverage varies smoothly with n. There is no sample space to be discrete about, so the t interval’s coverage approaches its nominal level from one side and stays there.

The oscillation is therefore specific to discrete data — proportions, counts, rates — and it is worth knowing that the family of parameters it affects is exactly the family most often reported in medicine, quality control and survey work.

That is an unfortunate coincidence rather than a deep fact, and it is the reason this essay is about a proportion rather than a mean.

What a planned sample size actually buys #

The oscillation has a consequence for study design that is easy to state and rarely acted on: the coverage at the sample size finally achieved is not the coverage that was planned for, because the sample size finally achieved is never exactly the one planned.

Recruitment overshoots. Consent is withdrawn. A batch is lost. A protocol says ninety and the analysis runs on eighty-four. On a smooth curve none of that matters, because a small change in n produces a small change in everything. Here it does not.

Sweeping the Wald interval at a true proportion of 0.15 and asking what the coverage would be anywhere within ten per cent of the planned size:

• planned at 30, coverage 94.2%, worst nearby 86.3%
• planned at 50, coverage 94.1%, worst nearby 89.9%
• planned at 100, coverage 93.3%, worst nearby 91.8%

The gap narrows as the study grows, which is the asymptotic argument doing its work, but at the sizes most studies actually run it is several percentage points wide. A design that computed its coverage at exactly n = 30 has computed a number the study has no particular reason to achieve.

The honest planning quantity is therefore the worst coverage over the range the sample size might land in, not the coverage at its midpoint. That is one extra line in a spreadsheet, and it is the difference between a stated level and a hoped-for one.

How much of this the better interval removes #

It is worth being precise about how much of the problem is the oscillation itself and how much is the Wald interval specifically, because the two are usually conflated.

Both. Over sample sizes from 10 to 100 at the same proportion, the Wald interval’s worst coverage is 79.3% and its worst single-step fall is 12.0 percentage points — from 93.8% at n = 19 to 81.9% at n = 20. The Wilson interval over exactly the same range has a worst coverage of 93.0% and a worst single step of 3.6 points.

So the oscillation does not disappear. It cannot: the underlying cause is that the sample space is discrete, and discreteness does not go away because a better formula is used. What changes is the amplitude, from something that destroys the stated level to something that wobbles a point or two below it.

That distinction matters for how the result should be read. The oscillation is a permanent feature of interval estimation for a discrete quantity, and any claim that a particular sample size delivers exactly 95% is false for every method. The Wald interval’s failure is a separate and larger thing sitting on top of it, and only the second one is optional.

The practical reading is unchanged and now has a magnitude attached. Use an interval whose oscillation is a couple of points rather than twelve, and quote the level as approximate, because for a proportion it always is.

Why the exact sum is what makes this visible #

One methodological point, since the effect would not be reportable by the usual route.

The dips here are a few percentage points wide and they sit at particular sample sizes. To see them by simulation, each point on the curve would need enough trials that its Monte Carlo error is small compared with the dip — and with coverage near 0.95 the standard error of a proportion from t trials is about 0.22 divided by the square root of t. Resolving a one-point feature to a tenth of its size needs roughly fifty thousand trials per sample size, and the curve has ninety of them.

That is affordable, and it is not what makes simulation the wrong tool here. The problem is that a simulated curve is itself noisy, so a reader looking at it cannot tell a real dip from a fluctuation without being told the error bars — and once the error bars are drawn at a width comparable to the effect, the honest conclusion from the picture is that something might be happening.

The exact sum removes the question. For a proportion the sample space is the n + 1 possible counts, each with a known probability, so the coverage is a finite sum and the curve is a computation rather than an estimate. Every wobble on it is real, every dip is exactly as deep as it is drawn, and no error bar is needed because there is no error.

This is the site’s rule paying for itself rather than merely being observed. Counting instead of sampling is not a purity preference; it is what makes a small, structured, genuinely surprising effect reportable at all.