Where the bootstrap lies

The bootstrap is the most generally useful idea in applied statistics. Resample the data with replacement, recompute the statistic, repeat, and use the spread of the results as the sampling distribution. It needs no formula for the statistic, no distributional assumption, and no derivation — which is why it works for things nobody has worked out the theory for.

It also has a failure mode, and the failure is total rather than gradual.

The measurement #

Draw forty observations uniformly from the interval [0, 1]. Build a 95% percentile bootstrap interval, and check whether it contains the truth. Do it three thousand times.

For the mean, the interval covers about 93% of the time — close to nominal, and the small shortfall is the well-known mild under-coverage of the percentile method at modest sample sizes.

For the maximum, it covers under 40%.

Not a degraded version of the right answer. A procedure that is wrong most of the time, produced by the same machinery, on the same data, with nothing to signal the difference.

Why #

The reason is one sentence and it is worth stating precisely.

A resample can never contain a value larger than the largest value in the original sample. Resampling draws from the observed data, so the maximum of any resample is at most the maximum of the sample. The bootstrap distribution of the maximum therefore sits entirely at or below the observed maximum — and the true maximum of the population is almost surely above it.

The interval is built in the wrong place. Its upper end cannot exceed the sample maximum, and the truth is above the sample maximum, so it misses whenever the sample maximum is not extremely close to the truth.

The mean has no such problem: resample means scatter on both sides of the sample mean, which scatters on both sides of the truth.

The atom, computed #

The failure has an exact cause and it is one line of arithmetic.

A bootstrap resample of size n omits any particular observation with probability $(1 - 1/n)^n$, which at n = 40 is $(39/40)^{40} = 0.363$. So 63.7% of resamples contain the sample maximum, and every one of those has a resampled maximum exactly equal to it.

The bootstrap distribution of the maximum is therefore not a smooth approximation to anything. It is a distribution with an atom of mass 0.637 sitting at its own upper end, and with support entirely at or below the sample maximum.

Two consequences follow and neither is a matter of degree.

The resample distribution cannot reach above the sample maximum, while the truth is above it with probability one. Whatever the interval’s upper end is, it is at most the largest number in the data.

And no number of resamples repairs it. The atom is a property of resampling with replacement, not of how many times it is done, so the usual remedy for a badly behaved bootstrap — take more resamples — moves nothing at all.

That is the shape this collection meets whenever a rule is handed an estimate where it needed a quantity: a plug-in that forgets it estimated inherits an error it cannot see, and a quantile that returns its own bracket reports a boundary as an answer. Here the empirical distribution has no information whatever about what lies beyond its largest point, and the maximum is a statistic that depends on nothing else.

The general condition #

The mechanism generalises, and the general statement is more useful than the example.

The bootstrap works when the statistic is a smooth functional of the distribution — roughly, when a small change in the data produces a small change in the statistic, and when the statistic’s sampling distribution is determined by the bulk of the data rather than by a few observations.

It fails for statistics that depend on the extremes: the maximum, the minimum, the range, the number of distinct values, a threshold exceedance near the edge of the data. It struggles at boundaries of a parameter space, and for statistics whose limiting distribution is not normal.

That is a real list rather than a warning to be careful. Each item can be checked by exactly the procedure used above — simulate from a known truth, build the interval, count.

What this says about method generally #

The bootstrap’s appeal is that it removes the need to derive anything. That is genuinely valuable and it carries a specific risk: a procedure that always produces an answer gives no signal when it is producing the wrong one.

A closed-form interval carries its assumptions on its face — the normality assumption is visible in the formula. A bootstrap interval looks identical whether it is working perfectly or failing completely, because the output is a pair of numbers either way.

The response is not to avoid the bootstrap. It is to keep the habit this whole site runs on: before trusting a procedure on a new problem, simulate from a known truth and count. It took a few thousand runs to establish everything in this essay, and the difference between the two cases is not subtle once measured.

What would fix the maximum #

For completeness, since the essay is otherwise purely negative.

The maximum of a uniform has known theory: the sample maximum is a sufficient statistic, its distribution is available in closed form, and an exact interval follows directly. Where the theory exists, use it.

Where it does not, variants of the bootstrap exist for this case — the parametric bootstrap, which resamples from a fitted model rather than from the data and so is not trapped below the observed maximum, or the m-out-of-n bootstrap, which resamples fewer points and recovers consistency for some of these statistics.

Both require knowing that there is a problem in the first place. That is the part this essay is for.

Why it works when it works #

Having spent the essay on the failure, the mechanism behind the success is worth stating, because it explains the boundary.

The bootstrap substitutes the empirical distribution — the observed data, each point with weight 1/n — for the unknown true distribution, and then computes the sampling distribution of the statistic under that substitute.

That is legitimate when two things hold. The empirical distribution has to be a good approximation to the true one, which it is for moderate n by the Glivenko–Cantelli theorem. And the statistic has to depend on the distribution smoothly, so that a small error in the distribution produces a small error in the statistic’s sampling distribution.

The mean satisfies both. The maximum satisfies the first and fails the second catastrophically: the maximum of the empirical distribution is the largest observation, full stop, and no amount of data makes the empirical maximum behave like the true one in the way the argument needs.

The other failure modes, briefly #

Each of these has the same root and a different appearance.

Parameters on a boundary. A variance component estimated as zero, a correlation at one. The bootstrap distribution piles up against the boundary and the interval is wrong on the side that matters.

Statistics with non-normal limiting distributions. The bootstrap’s guarantee is asymptotic, and where the limit is not normal the ordinary percentile method’s coverage does not converge to nominal.

Dependent data. The plain bootstrap resamples observations as though they were independent, which destroys the dependence structure and produces intervals far too narrow. Block bootstraps exist for exactly this and require choosing a block length.

Very small samples. The empirical distribution is a poor stand-in for the true one at n = 8, and the bootstrap inherits that.

The unifying diagnosis: the bootstrap is a statement about what the observed data implies, and it fails whenever the answer depends on something the observed data cannot contain.

The variants, and what each buys #

Three worth knowing, since the percentile method used here is the crudest.

The basic (or reverse-percentile) bootstrap reflects the distribution about the observed estimate rather than reading its percentiles directly. It corrects a bias the percentile method has and is no harder to compute.

BCa — bias-corrected and accelerated — adjusts for both bias and skewness in the bootstrap distribution, and has better coverage in most realistic cases. It is the default worth using when the tooling offers it.

The parametric bootstrap resamples from a fitted model rather than from the data, which escapes the trap in this essay: a fitted uniform can generate values above the observed maximum. It buys that by assuming the model, which is the thing the plain bootstrap avoided assuming.

None of them rescues the maximum without additional structure. The failure is not the method’s crudeness; it is the statistic.

What the essay is really arguing #

Not that the bootstrap is dangerous — it is one of the most useful ideas in the subject and the alternative for most awkward statistics is nothing at all.

The argument is about how to find out. The bootstrap looks identical whether it is working or failing, because its output is a pair of numbers either way. So the only protection is the one this site applies everywhere: simulate from a known truth, run the procedure, and count.

That took a few thousand runs and produced 93% against 40%. It requires no theory, it works for any procedure and any statistic, and it is the difference between using a method and knowing whether it applies.

How many resamples #

A practical parameter that is usually chosen by habit and has a defensible answer.

The number of bootstrap replications controls only the Monte Carlo error — the extra noise from not having enumerated every possible resample. It does not affect the statistical properties of the interval at all.

For a percentile interval at 95%, the relevant quantity is the precision of an extreme quantile of the bootstrap distribution, and a few thousand replications puts that well below the sampling uncertainty already present. Two thousand is the number used here; more is cheap and buys very little.

What more replications cannot buy is coverage. A procedure covering 40% with two hundred resamples covers 40% with two hundred thousand, because the failure is in what resampling can represent rather than in how many resamples were taken.

Why nothing in the output says so #

Worth dwelling on, because it is the transferable part.

A bootstrap interval for the maximum looks entirely reasonable. It is a pair of numbers in the right range, ordered correctly, with a plausible width. The bootstrap distribution it came from is unimodal and smooth. Nothing about it signals a problem.

Contrast that with a closed-form interval, which carries its assumptions in its formula: a normal quantile is visible, a variance formula is visible, and a reader who knows the assumptions can see them being made.

The bootstrap’s great virtue — that it requires no assumptions to be stated — is exactly why its failures are silent. There is no formula to inspect.

That generalises to any procedure whose output has the same shape whether or not it applies: a method that always produces an answer needs an external check, because it will never signal that it should not have been used. The external check available here is simulation from a known truth, and it is the only one.

What this argues about tools generally #

A closing generalisation, since the bootstrap is one instance of a much larger category.

Modern statistical practice is full of methods that apply broadly and fail quietly: cross-validation with dependent data, permutation tests with the wrong exchangeability, maximum likelihood at a boundary, any automatic model selection.

In each case the software returns a number, the number is in the right range, and nothing indicates whether the method’s conditions hold.

The habit that protects against all of them is the same one, and it does not require knowing the theory for each: construct a case where the answer is known, run the procedure, and count. It takes minutes, it needs no derivation, and it is the difference between using a tool and knowing whether it applies to the problem at hand.

That is the argument the whole of this site is making, and the bootstrap is where it is easiest to demonstrate, because the same procedure succeeds and fails on the same data depending only on which statistic it is pointed at.

What fixes it, measured #

The essay diagnoses the failure. The repairs are worth measuring, because two of the three work and knowing which is the useful part.

Forty observations from a uniform distribution on [0, θ], estimating θ, nominal 95%:

• percentile bootstrap: 0.0%
• parametric bootstrap: 94.1%
• the exact interval: 95.8%

The percentile bootstrap does not merely undercover, it covers never. That is worth sitting with. Across two thousand samples the interval contained the true maximum zero times, because no resample can contain a value larger than the largest one observed, so every interval lies entirely below θ. A procedure with a nominal 95% and an actual 0% is not a degraded version of a good method; it is a method producing an answer with no relationship to the question.

The parametric bootstrap works, and the reason is instructive. Instead of resampling the data, it fits the model — uniform on [0, m] where m is the observed maximum — and simulates fresh datasets from it. Those simulated datasets can and do produce maxima above m, so the resulting interval can reach above the observed maximum, which is exactly the ability the percentile version lacks.

The exact interval works best and requires knowing the distributional form, which is the assumption the bootstrap exists to avoid. It is included as the reference rather than the recommendation.

The condition, stated so it can be checked in advance #

The failure is not a quirk of the maximum. It has a description that covers every case, and applying it takes a few seconds.

The percentile bootstrap works when the statistic’s sampling distribution is smooth in the underlying distribution and centred near the truth — when a resample is, in the relevant respect, like a fresh sample. It fails when the statistic depends on the edge of the data, because resampling can never go past an edge it has already seen.

That gives a test anyone can apply before running anything: ask whether the statistic could take a value outside the range of the observed data if a fresh sample were drawn. If it could, resampling cannot produce that value, and the interval will be wrong on that side.

Applied to the usual suspects:

• The mean. A fresh sample’s mean stays inside the data’s range. Resampling is fine.
• The maximum, the minimum, the range. A fresh sample’s maximum can exceed anything seen. Resampling fails completely.
• Extreme quantiles. The 99th percentile from a sample of 40 is essentially the maximum, and fails for the same reason. The median is fine.
• The number of distinct values. A fresh sample contains values never observed. Fails.
• A variance. Fine in the middle, and degrades when the tail is heavy enough that the variance is dominated by rare large observations.

The pattern is that the bootstrap knows exactly as much about the population as the sample does, and statistics that ask about parts of the population the sample has not visited get an answer built from the parts it has.

Why the failure is invisible in the output #

The last point is the one that makes this dangerous rather than merely wrong, and it applies to a class of tools rather than to this one.

A percentile bootstrap interval for the maximum is not degenerate to look at. It has a lower bound and an upper bound, they are different numbers, the width is plausible, and the routine returns without a warning. Everything about the output resembles a working interval.

The only way to discover that it covers 0% of the time is to run the procedure against a case where the truth is known and count — which is the site’s rule, and which is not what anyone does with a general-purpose method that has a reputation for working.

The general form is worth carrying beyond the bootstrap. A method that reports its own uncertainty is trusted in proportion to how much machinery it appears to contain, and resampling contains a great deal. The check that matters is not whether the output looks reasonable; it is whether the procedure has been run once on a problem whose answer is known.