Normal Distribution
nothing is called 95% until it has been counted
Intervals, counted

Where the bootstrap lies

Resampling is the most generally useful trick in the subject and it has a failure mode that is easy to state: it cannot see past the data. For a statistic that lives at the edge of the sample, coverage collapses from 95% to almost nothing.
11 min read 6 figures Count it, do not claim itReversals that are nobody's mistake

The bootstrap is the most generally useful idea in applied statistics. Resample the data with replacement, recompute the statistic, repeat, and use the spread of the results as the sampling distribution. It needs no formula for the statistic, no distributional assumption, and no derivation — which is why it works for things nobody has worked out the theory for.

It also has a failure mode, and the failure is total rather than gradual.

Where the bootstrap works and where it does notUniform data on [0, 1]. For the mean the percentile bootstrap covers 93.6%. For the maximum it covers 0.0%, because a resample can never contain a value larger than the largest one observed, so the interval cannot reach above it.the sample mean93.6%nominal 95%the sample maximum0.0%nominal 95%3,000 samples of 40the same procedure, two statisticsresampling cannot see past the data
Fig. 1 The same procedure applied to two statistics of the same data. For one it covers properly. For the other it covers almost never.

The measurement

Draw forty observations uniformly from the interval [0, 1]. Build a 95% percentile bootstrap interval, and check whether it contains the truth. Do it three thousand times.

For the mean, the interval covers about 93% of the time — close to nominal, and the small shortfall is the well-known mild under-coverage of the percentile method at modest sample sizes.

For the maximum, it covers under 40%.

Not a degraded version of the right answer. A procedure that is wrong most of the time, produced by the same machinery, on the same data, with nothing to signal the difference.

Why

The reason is one sentence and it is worth stating precisely.

A resample can never contain a value larger than the largest value in the original sample. Resampling draws from the observed data, so the maximum of any resample is at most the maximum of the sample. The bootstrap distribution of the maximum therefore sits entirely at or below the observed maximum — and the true maximum of the population is almost surely above it.

The interval is built in the wrong place. Its upper end cannot exceed the sample maximum, and the truth is above the sample maximum, so it misses whenever the sample maximum is not extremely close to the truth.

The mean has no such problem: resample means scatter on both sides of the sample mean, which scatters on both sides of the truth.

Coverage of four nominal 95% intervals, n = 30Computed exactly by summing over all 31 possible counts, not simulated. The Wald interval drops to 26.0% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted
Fig. 2 What a coverage measurement looks like when a procedure is working across its range.

The general condition

The mechanism generalises, and the general statement is more useful than the example.

The bootstrap works when the statistic is a smooth functional of the distribution — roughly, when a small change in the data produces a small change in the statistic, and when the statistic’s sampling distribution is determined by the bulk of the data rather than by a few observations.

It fails for statistics that depend on the extremes: the maximum, the minimum, the range, the number of distinct values, a threshold exceedance near the edge of the data. It struggles at boundaries of a parameter space, and for statistics whose limiting distribution is not normal.

That is a real list rather than a warning to be careful. Each item can be checked by exactly the procedure used above — simulate from a known truth, build the interval, count.

Twenty 95% intervals for a proportion that really is 0.350 of the twenty miss the true value. The 95% is a property of the procedure across repetitions — no single interval has a 95% chance of anything, because it either contains 0.35 or it does not.the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval
Fig. 3 A procedure that always returns an interval gives no signal when the interval is in the wrong place.
Expected width against coverage, n = 30, p = 0.15The Wald interval is the shortest and covers 94.2%. Clopper–Pearson covers 98.3% and is 13% wider. Shortness is not a virtue on its own — an interval of zero width is the shortest of all.Wald0.241covers 94.2%Wilson0.247covers 96.5%Agresti–Coull0.259covers 96.5%Clopper–Pearson0.273covers 98.3%expected width, and what it buysorange: fails its nominal levelthe shortest interval is the one that misses
Fig. 4 Width and coverage measured together, which is the only pairing that can detect the failure.

What this says about method generally

The bootstrap’s appeal is that it removes the need to derive anything. That is genuinely valuable and it carries a specific risk: a procedure that always produces an answer gives no signal when it is producing the wrong one.

A closed-form interval carries its assumptions on its face — the normality assumption is visible in the formula. A bootstrap interval looks identical whether it is working perfectly or failing completely, because the output is a pair of numbers either way.

The response is not to avoid the bootstrap. It is to keep the habit this whole site runs on: before trusting a procedure on a new problem, simulate from a known truth and count. It took a few thousand runs to establish everything in this essay, and the difference between the two cases is not subtle once measured.

Coverage of a 95% interval for a mean, n = 8Measured over 20,000 samples. The t interval covers 95.0% and the z interval 91.3%. The difference is the price of pretending the standard deviation was known.t interval, 7 df95.0%± 0.3 at 2 s.e.z interval91.3%± 0.4 at 2 s.e.20,000 samples, one seed eachnominal 95%counted, not assumed
Fig. 5 Where the theory exists, it can be checked directly — and here it is, with the coverage counted.
Twenty samples of 40, every one of them genuinely normalEach panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 1.52 standard deviations off the line. Anything a reader would reject here would be a false alarm.20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like
Fig. 6 Simulating from a known truth and looking at the spread of outcomes is the same move in a different setting.

What would fix the maximum

For completeness, since the essay is otherwise purely negative.

The maximum of a uniform has known theory: the sample maximum is a sufficient statistic, its distribution is available in closed form, and an exact interval follows directly. Where the theory exists, use it.

Where it does not, variants of the bootstrap exist for this case — the parametric bootstrap, which resamples from a fitted model rather than from the data and so is not trapped below the observed maximum, or the m-out-of-n bootstrap, which resamples fewer points and recovers consistency for some of these statistics.

Both require knowing that there is a problem in the first place. That is the part this essay is for.

Why it works when it works

Having spent the essay on the failure, the mechanism behind the success is worth stating, because it explains the boundary.

The bootstrap substitutes the empirical distribution — the observed data, each point with weight 1/n — for the unknown true distribution, and then computes the sampling distribution of the statistic under that substitute.

That is legitimate when two things hold. The empirical distribution has to be a good approximation to the true one, which it is for moderate n by the Glivenko–Cantelli theorem. And the statistic has to depend on the distribution smoothly, so that a small error in the distribution produces a small error in the statistic’s sampling distribution.

The mean satisfies both. The maximum satisfies the first and fails the second catastrophically: the maximum of the empirical distribution is the largest observation, full stop, and no amount of data makes the empirical maximum behave like the true one in the way the argument needs.

The other failure modes, briefly

Each of these has the same root and a different appearance.

Parameters on a boundary. A variance component estimated as zero, a correlation at one. The bootstrap distribution piles up against the boundary and the interval is wrong on the side that matters.

Statistics with non-normal limiting distributions. The bootstrap’s guarantee is asymptotic, and where the limit is not normal the ordinary percentile method’s coverage does not converge to nominal.

Dependent data. The plain bootstrap resamples observations as though they were independent, which destroys the dependence structure and produces intervals far too narrow. Block bootstraps exist for exactly this and require choosing a block length.

Very small samples. The empirical distribution is a poor stand-in for the true one at n = 8, and the bootstrap inherits that.

The unifying diagnosis: the bootstrap is a statement about what the observed data implies, and it fails whenever the answer depends on something the observed data cannot contain.

The variants, and what each buys

Three worth knowing, since the percentile method used here is the crudest.

The basic (or reverse-percentile) bootstrap reflects the distribution about the observed estimate rather than reading its percentiles directly. It corrects a bias the percentile method has and is no harder to compute.

BCa — bias-corrected and accelerated — adjusts for both bias and skewness in the bootstrap distribution, and has better coverage in most realistic cases. It is the default worth using when the tooling offers it.

The parametric bootstrap resamples from a fitted model rather than from the data, which escapes the trap in this essay: a fitted uniform can generate values above the observed maximum. It buys that by assuming the model, which is the thing the plain bootstrap avoided assuming.

None of them rescues the maximum without additional structure. The failure is not the method’s crudeness; it is the statistic.

What the essay is really arguing

Not that the bootstrap is dangerous — it is one of the most useful ideas in the subject and the alternative for most awkward statistics is nothing at all.

The argument is about how to find out. The bootstrap looks identical whether it is working or failing, because its output is a pair of numbers either way. So the only protection is the one this site applies everywhere: simulate from a known truth, run the procedure, and count.

That took a few thousand runs and produced 93% against 40%. It requires no theory, it works for any procedure and any statistic, and it is the difference between using a method and knowing whether it applies.

How many resamples

A practical parameter that is usually chosen by habit and has a defensible answer.

The number of bootstrap replications controls only the Monte Carlo error — the extra noise from not having enumerated every possible resample. It does not affect the statistical properties of the interval at all.

For a percentile interval at 95%, the relevant quantity is the precision of an extreme quantile of the bootstrap distribution, and a few thousand replications puts that well below the sampling uncertainty already present. Two thousand is the number used here; more is cheap and buys very little.

What more replications cannot buy is coverage. A procedure covering 40% with two hundred resamples covers 40% with two hundred thousand, because the failure is in what resampling can represent rather than in how many resamples were taken.

Why the failure is invisible in the output

Worth dwelling on, because it is the transferable part.

A bootstrap interval for the maximum looks entirely reasonable. It is a pair of numbers in the right range, ordered correctly, with a plausible width. The bootstrap distribution it came from is unimodal and smooth. Nothing about it signals a problem.

Contrast that with a closed-form interval, which carries its assumptions in its formula: a normal quantile is visible, a variance formula is visible, and a reader who knows the assumptions can see them being made.

The bootstrap’s great virtue — that it requires no assumptions to be stated — is exactly why its failures are silent. There is no formula to inspect.

That generalises to any procedure whose output has the same shape whether or not it applies: a method that always produces an answer needs an external check, because it will never signal that it should not have been used. The external check available here is simulation from a known truth, and it is the only one.

What this argues about tools generally

A closing generalisation, since the bootstrap is one instance of a much larger category.

Modern statistical practice is full of methods that apply broadly and fail quietly: cross-validation with dependent data, permutation tests with the wrong exchangeability, maximum likelihood at a boundary, any automatic model selection.

In each case the software returns a number, the number is in the right range, and nothing indicates whether the method’s conditions hold.

The habit that protects against all of them is the same one, and it does not require knowing the theory for each: construct a case where the answer is known, run the procedure, and count. It takes minutes, it needs no derivation, and it is the difference between using a tool and knowing whether it applies to the problem at hand.

That is the argument the whole of this site is making, and the bootstrap is where it is easiest to demonstrate, because the same procedure succeeds and fails on the same data depending only on which statistic it is pointed at.

Where this goes next

Wald0.241covers 94.2%Wilson0.247covers 96.5%Agresti–Coull0.259covers 96.5%Clopper–Pearson0.273covers 98.3%expected width, and what it buysorange: fails its nominal levelthe shortest interval is the one that misses The shortest interval is the one that misses Four intervals for the same data, with their widths and their coverage measured together. The narrowest is the one that fails its stated level, which is exactly why it looks the most appealing. 0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted What the 95% refers to An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time. 02505007501e+3-0.50000.5001effect the study reportsstudiesthe truth, 0.3what gets published, 0.61shaded: reached p below 0.05power 20%, inflation 2.04× The winner's curse Filter honest studies down to the ones that reached significance and the effects they report are systematically too large. At low power the inflation is a factor of two, nobody has done anything wrong, and the selection did all of it.