Normal Distribution
nothing is called 95% until it has been counted
Concept

Bias-variance — where it appears

The two ways an estimate can be wrong: systematically, in the same direction every time, and by wandering. Almost every dial in this collection trades one against the other, and the setting that minimises either is rarely the setting that minimises the sum.

Named by 23 essays across 12 fields — each of them below, with the objects they name alongside it.

Four blocks, and only the ends matter. The weights a block carries, normalised so that the resample keeps the residuals' variance and only their covariances are attenuated. What separates these shapes, for everything that follows, is the value at the two ends and nothing about the middle: the first-order attenuation is −(w(0)² + w(1)²)/(2∫w²), which is -1.0000 for the rectangle, -0.3896 for the trapezoid cut off at half height, and exactly zero for both windows that reach the axis. The half-height trapezoid is in the table to be the case that separates a shape from a boundary value: it is smooth, it is tapered, and it buys none of the order the other two buy.

A block weighted inside itself

The triangle every block resample attenuates by is not a fact about blocks. It is the self-convolution of a rectangle, and a block weighted down towards its own ends has a different one — whose leading term is the squared value at the two ends and nothing else about the shape.

taper · Bootstrap What a sample shows, and what the algebra does. The difference between a rectangular block's implied long-run variance and a trapezoidal one's, as a share of the truth. Above the axis the rectangle is less biased and below it the trapezoid is. The heavy line is exact — computed from the law's own autocovariances — and it crosses at 19.2. The others are what samples of 120, 240, 480, 960 rows report, and every one of them exaggerates whichever window is ahead: at ℓ = 20, where the exact difference is 0.28 points, a sample of 120 rows shows 4.31 points — 15 times larger. That is the number the earlier reading of this comparison was missing: three tenths of a point is what the algebra says and not what a hundred and twenty rows report.

The gap a sample shows

The exact difference between two block windows at a block length of twenty is three tenths of a point. What a hundred and twenty rows report is four and a third, because the autocovariances the window is applied to are attenuated too.

crossing · Bootstrap Three rules and a target none of them is aimed at. Which block length each rule picks, over 400 samples of 120 rows, for the tapered window. Two of the rules are points: a length written into a protocol is 8.00 on every draw and the rule of thumb is 4.00, because n to the one third does not read the data at all. The plug-in reads the sample's own persistence and lands at 14.36 with a standard deviation of 2.93. The length that would actually have been best on that draw averages 24.57 with a standard deviation of 16.23 and runs from 10 to 48 between its tenth and ninetieth percentiles. The target moves five times as much as the best estimate of it does, which is why no rule can be close to it and why the two that do not try are not merely worse — they are somewhere else.

The length nobody has

Every comparison of block windows in this collection is made at each window's own best block length. That length has a standard deviation of sixteen across draws and averages twenty-five. No rule is aimed at it.

feasible · Bootstrap Two instruments, two block lengths. The block length that would actually have been best on each draw, for each of the two error readings, averaged over 400 samples of 120 rows. For the rectangular window the implied long-run variance wants 18.92 and the 95% point wants 16.05; for the tapered window, 21.82 against 17.74. The quantile wants a shorter block under both windows — a ratio of 0.848 and 0.813. That is the mechanism the whole field turns on: a rule for choosing a block length is a way of guessing a target, and the two instruments do not have the same target. A rule tuned to one is systematically long for the other, and the two windows do not pay the same price for being long.

A length for each instrument

The block length that is best for an implied variance is 18.92; the one best for the 95% point of the same resamples is 16.05. A rule is a way of guessing a target, and there are two targets.

readout · Bootstrap Which window is better depends on who chose the block length. The margin between a rectangular block and a tapered one, on 400 samples of 120 rows, under four rules for choosing the block length. At the length that would actually have been best on each draw the taper is ahead by 2.12 points of a 35.8% error, at 22.1 paired standard errors; at a length estimated from the sample's own persistence it is ahead by 1.89. At the length this field's own figures use — eight — the rectangle is ahead by 2.04, and at the rule of thumb by 4.54. Every rule sees the same draws. What separates them is the length: the two rules that lose to the rectangle pick 4.00 and 8.00 where the best available is 24.57, and a tapered window at a quarter of the right length has thrown away most of what it was weighting.

An ordering that depends on the rule

The tapered block beats the rectangular one at the best available block length and at one estimated from the data. At a length written into a protocol, and at the rule of thumb, the rectangle wins — at every sample size measured.

feasible · Bootstrap What a longer block buys and what it costs. A trapezoidal block at 120 rows, with the error split into the two things it is made of. The bias falls with the block length, because a longer block attenuates less, and it flattens at 23.2% because the sample's own autocovariances are short whatever window is applied to them. The spread rises with it, because a longer block means fewer of them. Their sum in quadrature has a minimum at ℓ = 16, which is not where either of the two has one. The faint line is the rectangle's total error, for scale: it is above the trapezoid's from ℓ = 12 onwards.

Bias is not the whole of it

A window that reaches zero at its ends attenuates less and uses less of each block. The block length that minimises its bias is not the one that minimises its error, and comparing two windows at one length compares one of them mis-tuned.

crossing · Bootstrap The crossing is in the dependence, not in the split. Regret of each rule as the design and the errors are made persistent at the same coefficient, scored on fresh rows because the closed form assumes exactly what is being taken away. An optimism theorem counts rows; when the rows repeat each other there are fewer of them than there are rows, the penalty is too small for the fit it is correcting, and the criterion starts buying coefficients it should not — its average winner grows from 3.31 coefficients to 3.90. The hold-out never used the theorem and overtakes at ρ ≈ 0.81. Schwarz's criterion, worst of the three on independent rows, is best on repeating ones — its heavier penalty is right for the wrong reason.

Where the two searches cross

The obvious dial between a criterion and a hold-out is how much of the sample to hold out, and moving it never changes the answer. The dial that does is one nobody chooses — how much each row repeats the one before it — and the two rules change places at about 0.81.

proxy · Forecast A wider band is always a better fit. The likelihood maximised over the band, at five widths, averaged over 30 samples. A band at L lags is a band at L + 1 with the last entry held at zero, so the families are nested and the maximised likelihood cannot fall — it does not, on any draw. What it does is rise at 0.984 of log-likelihood a lag. A parameter that is doing nothing buys half a unit in expectation and Akaike's criterion charges one, so this is a criterion very nearly indifferent between every width on offer. The dashed line is what a charge of one unit a lag would exactly cancel. Nothing in the fit chooses a width, and what does choose one is a charge somebody has to pick.

Nothing in the fit picks the width

A wider band is always a better fit, and it is better by about one unit of log-likelihood a lag — which is the order of what a criterion charges for a parameter. Three defensible rules choose widths a factor of three apart.

family · Order selection The window a whitening wants is not the memory of the errors. Regret under a five-period moving average as the tapered estimate is given more lags, over 120 draws at n = 120. The best window is L = 30; the automatic bandwidth is 4 and the error model's own likelihood chooses 9.7 on average. Both land in the same place and both are short, and the reason is the taper: a Bartlett weight at lag k is 1 − k/(L + 1), so a window of 8 keeps 0.556 of whatever the fourth lag carries and a window of 30 keeps 0.871. A window has to be several times the memory before it stops removing the memory. The dashed line is the rule told the errors are a first-order autoregression, which needs no window at all.

The window a whitening wants

Every law here is best whitened by a window several times longer than its own memory, including the one whose memory ends at the fourth lag. The three ways of choosing it from the sample all land in the same place, and it is the wrong one.

general · Order selection The argument is a third of the size of the thing it is inside. Three quantities on one scale, in points of the error in a block resample's implied long-run variance, at 120 rows. The gap between the two windows at the best available block length — the whole subject of the comparison this field inherited — is 2.12 points. What the best rule a practitioner could actually run gives up against that same best length is 7.26, a factor of 3.42. What the rule of thumb gives up is 26.01. So the ordering between windows is worth establishing and is not worth arguing about, and the sentence that follows from it is not use the taper but estimate the block length, because that is where the points are.

What choosing the length costs

The gap between two block windows at the best available length is 2.12 points. What the best rule a practitioner could run gives up against that same length is 7.26. The argument is a third of the size of the thing it is inside.

feasible · Bootstrap The threshold buys accuracy and spends exceedances. The mean squared error of the estimated shape against the threshold, split into the square of its bias and its spread, over 600 records of 2000 readings from a a normal parent. At the 0.9 quantile 199 exceedances are left, the bias is -0.1708, the spread is 0.0701 and the total error is 0.0341. The bias falls as the threshold rises because the exceedances get closer to being generalised Pareto; the spread rises because there are fewer of them. The sum is smallest at the 0.925 quantile, at 0.0340, of which 80.6% is still bias — so even the best threshold on this grid is one where accuracy, not spread, is the binding constraint.

The threshold is a dial

A peaks-over-threshold analysis has one knob, and raising it buys accuracy with exceedances. For a normal parent the error is smallest at the 0.925 quantile and 80.6% of it is still bias there — and both diagnostics practitioners use to set the knob lose to a fixed 0.90 rule, one by a factor of 1.590 and one by 11.881.

extreme · Extremes One t interval with its scale read three ways, 120 rows in blocks of 32. What a 95% t interval for the mean of a first-order autoregression at 0.7 covers, and how wide it is, when its block-means scale is read from the 3 disjoint blocks of 32 rows, from a block at every one of the 89 starting rows, and from those overlapping blocks tapered — over 8,000 draws. Disjoint: 94.4% at a width of 1.560, with a multiplier of 4.30. Every starting row: 94.8% at 1.054, multiplier 2.74. Tapered: 95.0% at 1.020, multiplier 2.57.

A block at every starting row

The t interval on the block count reads its scale from the disjoint blocks, three of them when 120 rows are cut into blocks of thirty-two, and throws away the other eighty-six blocks the series contains. Reading a block at every starting row leaves the scale's attenuation where it was — the two expectations agree to within a hundredth at every cell — and nearly doubles the degrees of freedom it realises, so the interval loses a third of its width at the same coverage. Where the interval falls short, at short blocks, it falls short by the same amount however the blocks are read.

taper · Bootstrap A bias against a variance, with the answer in between. How wrong one sample's reference distribution is, split into the two things it is wrong by. Sharing the multiplier over more rows keeps more of the dependence and closes the bias from 1.688 to 0.835; every row it is shared over also removes an independent sign from the 101 the sample started with, and the spread of the resulting quantile rises from 1.307 to 2.172. The distance a practitioner with one sample is actually exposed to is the two together, and it is smallest at ℓ = 5.

How long a block a multiplier shares

Sharing a sign over more rows keeps more of the dependence and leaves fewer independent signs to build a distribution from. The bias falls from 1.6885 to 0.8479 and the spread rises from 1.3073 to 2.1716, and the rejection rate walks straight through its nominal level on the way from 11.3% to 1.3%.

proxy · Reference Where the taper's case begins, and it is not where the algebra says. The block length at which a trapezoidal block's implied variance stops being more biased than a rectangular one's, against the length of the sample. Computed exactly — from the law's own autocovariances, with no sampling in it — the answer is 19.2 and does not depend on the sample at all. What a sample of 120 rows reports is 13.3, and the reported crossing walks out towards the exact one as the sample grows: 13.3, 15.0, 16.4, 18.0. The mechanism is that the autocovariances the window is applied to are themselves attenuated, worst at the longest lags, and the window that discards those lags loses less of them.

The error no window repairs

Every block window's best estimate of a long-run variance is wrong by about forty per cent at a hundred and twenty rows, and the largest part of that is not a bias at all. Choosing the window moves a twentieth of it.

crossing · Bootstrap Coverage against block length, 120 rows. The coverage of the t interval for the mean of a first-order autoregression at 0.7 on 120 rows, at every block length on a fine grid, with its scale read from the disjoint blocks and from a block at every starting row, over 8,000 draws. The squared-error length for disjoint blocks at the true persistence is 13.9 rows, and the same algebra for the overlapping scale gives 16.0, marked. The overlapping interval first reaches 95% — to within a tenth of a point — at blocks of 32 rows, where it covers 94.9% at a width of 1.056; at the overlapping squared-error length it covers about 92.5%.

A length chosen for the interval

The squared-error rule for a block length weighs the scale's bias against its variance. Rebuilt for a block at every starting row, whose variance is two thirds as large, it asks for blocks fourteen per cent longer and buys four tenths of a point of coverage. A t interval has already paid for the scale's variance in its multiplier, so its own coverage is predicted by the bias alone — within seven tenths of a point up to blocks of twenty-eight rows — and a rule that chooses the shortest length that prediction says will cover asks for blocks two and a half to three times longer and lands within about a point of 95% at 120, 240 and 480 rows. The price is width, and it is paid where the series is short.

taper · Bootstrap A premium that shrinks and a payout that grows. The relative error of the trapezoid's implied variance with the block length read off the whole autocovariance sequence, minus its error with the length from the lag-one correlation alone, in points, at four sample sizes and 400 draws each. Under the first-order autoregression the plug-in's algebra was written for, reading the sequence costs 3.87 at 120, 2.37 at 240, 2.48 at 480, 1.15 at 960. Under the five-period moving average, which the plug-in reads as an autoregression at 0.8, it is 0.43 at 120, -1.72 at 240, -3.01 at 480, -3.77 at 960 — a tie at 120 rows and a gain that grows with the sample.

A length read off the whole sequence

The automatic block length reads one number off the sample — the lag-one correlation — and assumes the rest of the dependence is geometric. Reading the whole autocovariance sequence instead removes the assumption and puts the length where the law wants it. At a hundred and twenty rows it buys nothing, even where the assumption is wrong; it costs 3.87 points where the assumption is right, and by 960 rows that premium is 1.15 and the payout under a law the plug-in misreads is 3.77. Under long memory neither rule, nor the best length on each draw, gets within ninety per cent.

feasible · Bootstrap Post-stratifying a simulated 5% rate on a companion correlated 0.9: the share of its variance removed, by the number of strata. The best any function of the companion can do is 48.1%; its own rate at the same cut, 38.2%. Strata spent across the band where the conditional probability changes reach 95% of the best at 8 strata; equal-probability strata need 32. At eight strata the two remove 46.8% and 29.9%.

A companion cut into many

A simulated rate can be estimated within bands of a companion whose distribution is exact, and the bands weighted back by their exact probabilities. One band boundary at the target's threshold is the companion's own rate; infinitely many are the best any function of the companion can do, which for a 5% rate at a correlation of 0.9 removes 48.1% of the variance. Eight bands placed where the target's conditional probability is changing remove 46.8%; eight at equal probabilities remove 29.9%, and need thirty-two to catch up. On the lognormal t test's size, thirty-two bands of the logarithms' t statistic were worth 2.99 times the draws and two hundred and fifty-six, 2.14.

splits · Routes How far each estimate of the small groups sits from their truth, when the size relation is flattening below the median size. The small groups' average true effect is −0.259. Their estimates are off on average by −0.012 (own means), +0.276 (one fitted centre), −0.173 (fitted line in log size), −0.025 (fitted quadratic in log size), −0.003 (the true relation).

The relation the table has to estimate

Pooling a league table's groups towards a centre that rises with size repaired the small groups' bias — when the size relation was known. Fitted from the table itself, a straight line in log size loses almost nothing: the small groups' error is 0.750 against 0.719 with the true relation. When the relation bends below the median size, the fitted line extrapolates the large groups' slope into the region where it is barely observed and misplaces the small groups by −0.173 if the relation flattens and +0.264 if it steepens — smaller than one centre's error, of either sign. A quadratic in log size removes nearly all of it, −0.025 and +0.014, and costs 0.025 in error when the relation was straight.

borrowed · Shrinkage The error of three estimates against the length of the series, AR(1) at 0.7. Under AR(1) at 0.7, the root-mean-square error of three estimates of the long-run variance, as a share of the truth, at 120, 240, 480, 960 rows. a first-order autoregression: 45.5%, 33.3%, 23.1%, 15.8%; an autoregression, order by AIC: 50.0%, 39.3%, 26.3%, 18.7%; the trapezoid block, best length: 42.4%, 32.9%, 25.4%, 19.5%.

A model against a window

A block resample's long-run variance is attenuated by its window, and the construction that escapes the window is a model: fit an autoregression and sum its dependence to every lag in closed form. Measured on both halves of the answer, the escape is narrower than it looks. Correctly specified, on 120 rows of a first-order autoregression, the model's error is 45.5% of the truth against the trapezoid block's 42.4% — it removes the bias and buys a spread larger than the bias was — and it pulls ahead only from about 480 rows. Wrongly specified, on a five-period moving average, it is off by 84% to 105% at every length, and even an order chosen by the criterion trails the window throughout.

crossing · Bootstrap What two exact companions are worth to a simulated rate: the lognormal t test's size post-stratified three ways, by the number of cells. To first order, bands of the logarithms' t statistic are worth at most 2.77 times the draws; a 32 × 32 grid in the pair 4.68; bands of a score fitted to the pair on a thousand samples 4.64 at sixteen bands and 4.69 at sixty-four. Counted over runs of 4,000 samples, the grid of 1,024 cells was worth 1.97, with 94 cells a run left empty.

A companion with two coordinates

The logarithms' t statistic, cut into bands, is worth at most 2.77 times the draws to a simulation of the lognormal t test's size. It is one function of two exactly known statistics, the logarithms' mean and spread, and the pair is worth 4.64 when a score fitted to it on a pilot of a thousand samples is cut into sixteen bands whose probabilities are integrated exactly. A grid of cells in the pair needs a thousand cells to reach the same and, counted on runs of four thousand samples, is worth 1.97. Band weights estimated from a pilot instead of computed are worth less than the pilot's draws would have been had they simply been counted — exactly nothing at a pilot the size of the run.

splits · Routes Twenty studies of a thousand readings estimate the regression of the extremes: the t, four degrees parent. Each thin line is one study of 1000 readings from the t, four degrees parent: Tweedie's formula with the log-density's slope estimated by a degree-5 log-spline, drawn up to that study's largest reading. The thick line is the share kept by integration over the true score, and the dashed line the correlation, 0.6. The top ten readings of the median study begin at 2.45. Over 400 studies the corrected share kept by the top one per cent averages 0.8190, with a spread of 0.0806, against 0.7842 by integration; a Gaussian kernel averages 0.7914 with a spread of 0.0853.

The slope of a density nobody can see

Tweedie's formula corrects a reading by the slope of the readings' own log-density, and a study has its readings. Estimated from a thousand of them, the correction for the top one per cent beats the correlation's linear rule on 84.0% to 98.0% of studies from heavy-tailed populations and on 75.0% to 81.5% from a bounded one — and costs an error of 0.09 to 0.12 where the population is normal and the rule was already exact. At 250 readings the log-spline loses to the rule it replaces, and at 16,000 the same log-spline gets worse on a power tail.

paradox · Rtm Approximate weights against the size of the run. What sixteen bands of the logarithms' t statistic are worth to a simulation of the lognormal t test's lower-tail size, in multiples of the draws, against the size of the run, with the band weights from Student's t exactly and from three approximations to it. Exact weights are worth 2.709 at every size. Fisher's one-term expansion is worth 2.708 at four thousand draws and 2.523 at a million, and breaks even with counting at 23.7 million draws. The variance-matched normal is worth 1.849 at four thousand and breaks even at 14,698. The standard normal is worth 0.762 at a thousand and breaks even at 669: a bias that does not shrink is multiplied by every draw added.

Weights from an approximate law

A companion whose law is only approximately known can still be cut into bands, with each band's weight computed from the approximation. The weight's error is a bias that does not shrink, and the run multiplies it: the stratified rate is worth p(1 − p)/(v + Nb²) times the draws. Reading Student's t on nineteen degrees of freedom as normal breaks even at 669 draws; the normal with the t's variance at about fifteen thousand; Fisher's one-term expansion at twenty-four million. What decides the bias is how the weight errors line up with the band rates, not how large they are — Fisher's is wrong by sixty per cent in one band and biases the rate by six hundredths of one.

splits · Routes The weight each of a hundred outcomes carries in the fitted value at the 97.5th percentile of a lognormal covariate. One design of a hundred lognormal covariate values. The fitted value at a covariate value of 7.10 is a weighted sum of the outcomes, with weights fixed by the design. Under a line and its square it rests on 10.7 effective points, the largest carrying 19.1% of its variance. Under a natural spline with knots at 0.17, 0.57, 1.18, 3.95 it rests on 3.8, the largest carrying 41.1%.

A value read off a curve

No coefficient of a spline is read; what is read is the fitted curve's value at a covariate value of interest, and that value is a weighted sum of the outcomes with weights the design fixes before any outcome is seen. In the tail of a lognormal covariate a natural spline's value rests on 4.8 of a hundred points, fewer than the squared term's coefficient's 6.9, and the squared term's own value there rests on 10.2 — yet it is three times further from the truth, because its extra points are borrowed from where the curve has a different shape and buy no precision at all. At four hundred points, moving the last knot past the value raises its count threefold and halves its error; doubling the data does neither.

spread · Summary

Named alongside it

The objects these essays reach for when they reach for this one.

Closed formMonte CarloBlock bootstrapLong-run varianceBlock lengthMean squared errorTaperingDependenceResamplingSample autocovarianceTuning parameterAutocorrelation

All concepts