Normal Distribution
nothing is called 95% until it has been counted
Concept

Bootstrap — where it appears

Resampling the data as though it were the population, in order to estimate a quantity's sampling distribution without a formula for it. What it keeps decides what it repairs, and the choice between resampling observations, blocks of them, or signs attached to them is the whole of the construction.

Named by 18 essays across 14 fields — each of them below, with the objects they name alongside it.

The distribution of the largest statistic in the table. Fit the benchmark to the whole series, resample its residuals, simulate 199 series in which the null is true by construction, re-run the entire eight-variant search on each, and keep the largest statistic. That is the distribution drawn here, and it is the distribution of the thing a specification search actually reports. It is centred at 1.045 — the maximum of eight statistics is not centred at zero however well each of them behaves — and its 5% point is 2.536. A table read against 1.671 is reading the distribution of one statistic; a Bonferroni correction reads it against 2.577 and is nearly right here, because eight variants that each add a different lag are nearly eight separate chances.

A null with a model in it

The distribution to read the winner of a table against cannot be resampled from the data, because the data does not contain the null. It has to be generated from a model — which is the assumption the resampling was chosen to avoid.

search · Bootstrap Least squares estimates persistence low, by an amount with a formula. 3000 series of 50 observations at each persistence. The lower curve is the counted bias of the least-squares estimate of φ, and the open marks on it are −(1 + 3φ)/n, computed rather than fitted. The upper curve is the bias left after adding that quantity back, evaluated at the estimate rather than at the truth nobody has: -0.0020 at φ = 0.3, -0.0023 at φ = 0.5, -0.0039 at φ = 0.7, -0.0059 at φ = 0.8, -0.0108 at φ = 0.9, -0.0165 at φ = 0.95. The formula is a leading-order expression and it understates the bias where the persistence is nearest one — -0.0882 counted against -0.0770 predicted at φ = 0.95, which is the corner of the parameter space every one of these approximations is worst in.

Correcting the persistence

Least squares estimates how much a series remembers of itself as smaller than it is, at every value it can take, by an amount with a closed form. Subtracting that amount back is one line of arithmetic, and what the line costs is variance.

evaluation · Bias Which repair goes with which defect. The share of true nulls rejected at a nominal 5% by a reference distribution generated from the fitted benchmark, over 120 draws with 59 resamples each. Where the errors are well behaved every resampling is fine and all four are conservative. Where the variance is a function of the design, the two that detach a residual from its own row reject 5.8% and 8.3% — and the block bootstrap, which is the resampling three earlier fields on this site reach for, repairs nothing at all, because the dependence it is built for is between origins and the rolling scheme reproduces that on its own. Where the errors are skewed the symmetric multiplier is the one that is wrong, and Mammen's two-point version is the only one of the four that is right in both columns.

Residuals that keep their own variance

A reference distribution for a search has to be generated from a fitted model, and the generator draws residuals. Four ways of drawing them keep four different things — and the one this site has reached for three times repairs nothing at all here.

select · Bootstrap Two constructions on one triangle, and a third that is not. Three resamplings that all keep runs of neighbours, on the same residuals at a block length of 5, with the lags running past ℓ so that the tapers separate. A blocked multiplier never moves a residual; a fixed-length moving block moves every one; and they attenuate identically, worst gap 1.4 standard errors, both sitting on γ_resid(k)(1 − k/ℓ)⁺ and both exactly zero past ℓ — so the attenuation is the block boundary rather than the multiplier. The third is the stationary bootstrap, whose runs are geometric rather than fixed: its taper is γ_resid(k)(1 − 1/ℓ)^k, it agrees with the other two at the first lag and at no other, and at lag 6 it still carries 0.0081 where they carry -0.0005.

The triangle that was not the multiplier's

A resampling that leaves each residual on its own row can keep only what the residuals have, times a triangle. A construction that moves every one of them has the same triangle — and the one in this collection's own table has a different taper entirely.

banded · Bootstrap Each repair is for its own defect, and one is for both. The 95% point of the statistic's own distribution in each world, against the mean 95% point of five reference distributions built from one sample. Where the error variance is a function of the design, the two resamplings that detach a residual from its row fall short and the two multipliers that keep it there do not; where the rows repeat each other it is the other way round. With both defects at once the blocked multiplier — drawn once per run of 5 rows, so the residual never moves and its neighbours share a sign — is the closest of the five, at 2.999 against a truth of 3.803. It is still short by 0.804, and that shortfall is the next figure.

Two defects and one resampling

Four resamplings, each the repair for one defect and wrong about the other. Put both defects in the same world and the statistic's 5% point is 3.8028, where the best of the four reaches 2.8326 — until a multiplier that stays on its own row and shares a sign with its neighbours reaches 2.9988.

proxy · Bootstrap The distribution the table does not have. 599 series simulated from the smaller model fitted to one comparison's own data, the whole rolling comparison re-run on each, and the ordinary statistic recorded. Under this null the two forecasts are the same forecast in population, so what is left in a sample is the larger model's estimation error and the statistic is centred at -1.134 rather than at zero. Its 95% point is 0.264; the standard normal drawn behind it puts that point at 1.645. Reading this statistic against that curve is not a poor approximation, it is a different distribution: the share of this one above 1.645 is 0.2%.

A distribution drawn from the null

Between nested models the ordinary comparison statistic has a null distribution centred at minus one and a 95% point of a quarter. A correction to its mean repairs the centre and leaves the shape; simulating the null repairs both.

ranking · Bootstrap What each construction carries, against what there was. The autocorrelation of a resampled error series at five lags, averaged over 60 samples of 40 resamples each. Three facts are in the picture. The residuals lie below the errors at every lag, which is the ceiling a multiplier cannot exceed. The blocked multiplier and the fixed-length block lie on top of each other below it — they attenuate identically, because the attenuation is the join — while the stationary bootstrap, whose runs are geometric rather than fixed, sits above them both. And the sieve is the exception in kind rather than in degree: at lag six it carries 0.0638 where the residuals have 0.0300 and the multiplier has -0.0011, because a fitted model extrapolates past the lags it was told about and a truncated sample sequence cannot.

Errors generated from a fitted model

The one construction that is not bounded by the residuals, because a model extrapolates past the lags it was told about and a truncated sample sequence cannot. It is nearly exact where the only defect is dependence, and it pays for it where there are two.

banded · Reference What the corner costs when the table is full of hopeless candidates. The benchmark holds two predictors, one of which is worth 1; a third predictor, worth the amount on the horizontal axis, is held only by candidates the benchmark does not contain. At the left the null is true and both procedures hold their level. To the right there is a genuinely better candidate, and the uncorrected reality check finds it 2.7% of the time while the same test with the clearly bad columns recentred finds it 51.0% of the time. The columns doing the damage are the ones nobody would have looked at twice: they are so far behind that they cannot win, and calibrating as though they might is what makes the test blind.

The corner the test is calibrated at

"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.

select · Reference Reading the tail of a sum of thirty from one sample of 30, exponential source. Medians over two thousand samples. Six standard deviations out the normal reads ×6.8e-4 of the exact tail, the empirical saddlepoint ×0.061, a gamma fitted by the sample's skewness ×0.185, and a fixed exponential model ×1.000. The bars run from the tenth to the ninetieth percentile of the two readings that estimate the tail's shape.

A tail the sample never saw

Given the source, the saddlepoint reads the tail of a sum of thirty exponential draws to within a fifth of a per cent six standard deviations out. Given only a sample of thirty from that source, the same tilt reads it at a median of 0.061 of the truth, and nine samples in ten read it low. The error has moved from the approximation to the sample: thirty draws put the skewness at a median of 1.30 against a true 2, and hold nothing past their largest value.

expansion · Rate Where the bootstrap works and where it does not. Uniform data on [0, 1]. For the mean the percentile bootstrap covers 93.5%. For the maximum it covers 0.0%, because a resample can never contain a value larger than the largest one observed, so the interval cannot reach above it.

Where the bootstrap lies

Resampling is the most generally useful trick in the subject and it has a failure mode that is easy to state: it cannot see past the data. For a statistic that lives at the edge of the sample, coverage collapses from 95% to almost nothing.

intervals · Bootstrap The average decay factor each route produces, φ = 0.85, 6 steps ahead. The truth is φ^6 = 0.3771. no correction averages 0.2616 with a spread of 0.1646 and a squared forecast error of 3.2516; the formula, on the persistence averages 0.4213 with a spread of 0.2528 and a squared forecast error of 3.4827; the bootstrap, on the persistence averages 0.4355 with a spread of 0.2655 and a squared forecast error of 3.5120; the bootstrap, on the decay factor averages 0.3375 with a spread of 0.2278 and a squared forecast error of 3.4132. 800 series, 100 bootstrap refits each.

Correcting the forecast instead

The complaint against the usual repair is that a correction aimed at the persistence lands on the wrong quantity. Aiming it at the decay factor the forecast actually uses fixes exactly that — the error stops compounding with the horizon, 69.7% becomes 9.5% at twelve steps — and the forecast still gets worse.

evaluation · Bias Six intervals after a design chosen from the data, 12 runs. Over 800 experiments of 12 runs, a first-order design at a guess of K = 1 where the truth is 3, and 199 parametric resamples per experiment. After the sequential design chosen from the data, the coverage of each interval for K: Wald 91.3%; percentile, design kept 90.6%; percentile, design re-run 91.6%; studentised, design kept 93.0%; studentised, design re-run 92.0%; profile 94.9%.

A bootstrap that repeats the design

After a design that chose its settings from the data, a parametric bootstrap can keep the settings the experiment ran or repeat the whole procedure from the start. On twelve runs repeating it changes coverage by a point at most, because the adaptivity was nearly free. What matters is the interval's shape: the percentile bootstrap covers 90.6%, no better than the Wald interval's 91.3%; the studentised one 93.0%; the profile interval 94.9%. And after a design stuck at a twelvefold wrong guess, the studentised bootstrap — the textbook repair — covers 77.8%, with intervals averaging 54.5 wide for a parameter of 3.

robust · Local design A leak read off a cohort, at two cuts. The threshold rule's leak as a cohort of 1,000 units estimates it from its own two-by-two table: the bar is the middle eighty per cent of 600 cohorts' plug-in estimates, the open circle their median and the stroke the true leak, for five copulas at a Spearman rank correlation of 0.4. At the 90th percentile the estimate is centred on the truth and its middle eighty per cent spans about fourteen points: Gaussian 12.8% to 26.4% around 19.2%, t on four 21.5% to 35.7% around 28.9%, Frank 5.4% to 18.7% around 12.2%, Clayton 0.8% to 9.6% around 4.4%, Clayton turned over 35.9% to 49.5% around 43.1%. At the 99th it is not centred on anything: Gaussian 13.1% to 100.0% around 13.7%, t on four 17.5% to 72.9% around 37.4%, Frank 14.4% to 100.0% around 2.2%, Clayton 16.1% to 100.0% around 0.6%, Clayton turned over 37.0% to 75.7% around 56.3%, because on three of the five copulas between two cohorts in five and four in five have nobody past both cuts.

A leak read off a two-by-two table

The share of an interaction a threshold-balancing rule leaves alone needs one number from the data: how often a unit past the cut on one covariate is past it on the other. A cohort of a thousand estimates it to within a dozen points at the 90th percentile. At the 99th most cohorts have an empty cell in their table, and every one of those reads a leak of exactly one hundred per cent — a reading the bootstrap then confirms with an interval of zero width.

copula · Criterion The two standardised means, and the shape of Fieller's set for their ratio at a = 3, d = 1. Each dot is one pair (zx, zy) drawn around (3, 1). Outside the horizontal band |zy| > 1.96 Fieller's set is a bounded interval, with probability 17.01%; inside the band and outside the disc of radius 1.96 it is everything outside an interval, 75.03%; inside the disc it is the whole line, 7.96%. On 40,000 counted draws Fieller covers ρ = 3.00 95.21% of the time and the delta interval 82.48%.

A ratio whose interval has to be the whole line

The delta interval for a ratio of two means covers 95.61% when the denominator is eight standard errors from zero and 1.10% at a ten-thousandth of one, and ten times as wide it still covers only 3.48%. Linearising is not the fault. Gleser and Hwang proved that every interval that is always finite fails the same way, so an interval that keeps its promise has to be the whole line some of the time.

normal · Clt Which side the misses fall on. Misses of a nominal 95% forecast interval 1 step ahead on 60 observations of a first-order autoregression at 0.7, split by side. normal shocks: the interval as printed 3.1% below and 3.1% above, shocks from the residuals 3.8% below and 4.2% above, the same, coefficient re-estimated 3.3% below and 4.4% above. Student's t on five: the interval as printed 3.3% below and 3.0% above, shocks from the residuals 3.9% below and 3.0% above, the same, coefficient re-estimated 3.6% below and 3.1% above. skewed shocks: the interval as printed 0.0% below and 6.5% above, shocks from the residuals 4.3% below and 4.1% above, the same, coefficient re-estimated 1.8% below and 4.3% above.

A tail sixty residuals have barely seen

A forecast interval that reads a normal quantile has every one of its misses on one side when the shocks are skewed, and the natural repair is to draw future shocks from the model's own residuals. It works on the side: 0.0% below and 6.5% above becomes 4.3% and 4.1%. It pays for that with coverage, 93.5% to 91.6%, and on sixty observations it does this under normal shocks too. And a nominal 99% interval has a ceiling nothing in the resampling can lift, because one new shock beats all of sixty old ones in either direction with probability 3.3%.

forecast · Forecast Seven ways to report the height at the setting a response-surface analysis recommends, at noise 2. Bias and root mean squared error of the reported height about the true height at the chosen setting, over 2,000 studies with a central composite design of thirteen runs, or the same design run twice. 13 runs, the fit's own height: bias 0.884, error 1.537. 13 runs, optimism bootstrapped away: bias 0.183, error 1.396. One confirmation run alone: bias -0.005, error 2.005. Corrected fit and one confirmation run: bias 0.056, error 1.121. 26 runs, the fit's own height: bias 0.475, error 0.998. 26 runs, optimism bootstrapped away: bias 0.050, error 0.924. 26 runs split: choose on 13, estimate on 13: bias -0.014, error 1.213.

The height at the chosen setting

The height a response-surface fit predicts at the setting it recommends reads high, because the setting was chosen where the fit was highest. The obvious repair — choose on some runs, estimate on the rest — is impossible in the usual thirteen-run design, which has nine distinct settings where two separate fits need twelve. A parametric bootstrap of the optimism removes four fifths of it at no cost in runs, 0.884 down to 0.183 at twice the noise. With the design run twice, correcting the full fit beats splitting it: an error of 0.924 against 1.213, at a better setting.

surface · Optimum Five 95% intervals for an AR(1) forecast 6 steps ahead at φ = 0.85, fifty observations. Coverage on 4,000 series, the same series for every interval, each simulated interval built from 499 simulated futures. The plug-in formula: 88.48%. The formula, all three repairs: 92.80%. Simulated: parameters drawn: 92.55%. Simulated: residual bootstrap: 88.25%. Simulated: bootstrap, bias removed: 93.47%.

An interval built from simulated futures

A forecast interval read off simulated futures needs no delta method and no normal multiplier, and was expected to close the residual the three repairs to the formula leave. It does not close it on its own: the plain residual bootstrap covers 88.25% six steps ahead at φ = 0.85, the plug-in formula's 88.48% to within the count, because it simulates around the same biased persistence. With that bias removed inside the bootstrap it covers 93.47%, the best of five intervals, and its shortfall stops growing with the horizon — 93.80% at twelve steps, where the repaired formula covers 91.85%.

evaluation · Bias The thresholds one validation study supports, read from four hundred resamples of it. Prevalence 0.26, just past the jump to flagging everyone. The study's own binormal fit recommends a threshold flagging 65.8% of healthy people; its resamples' recommendations run from 36.6% to 100.0% at the 2.5th and 97.5th percentiles, and 18.5% of them flag everyone. The true best threshold flags everyone.

A threshold reported with its interval

Resample a validation study of a hundred diseased and three hundred healthy people, refit the binormal model each time, and read off the threshold each resample recommends. Far from the prevalence at which the best threshold jumps to flagging everyone, the interval is tight — flagging 12% to 24% of healthy people at a prevalence of 5%. Near the jump, the median study's interval runs from flagging under half of the healthy to flagging everyone, and 91% of intervals cross the jump. The two ends differ by fifteen false alarms' worth of harm per hundred people, and no rule for choosing inside the interval beats the study's own choice at every prevalence.

screening · Baserate

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloReference distributionCoverageAutocorrelationBlock bootstrapCritical valueError rateSpecification searchHeteroskedasticityNull hypothesisOut of sampleResampling

All concepts