Normal Distribution
nothing is called 95% until it has been counted
Concept

Plug in estimate — where it appears

Substituting an estimate for an unknown quantity in a formula derived for the known case. The formula is then being evaluated at something with its own uncertainty, which the formula does not account for and which is usually where the shortfall in coverage comes from.

Named by 35 essays across 18 fields — each of them below, with the objects they name alongside it.

What the interim sees, at an effect of 1. The same 60 observations, estimated two ways. Keeping the arms separate gives 0.995, which is σ. Pooling them without separating the arms — the price of staying blind to the comparison — gives 1.114, against the identity √(1 + Δ²/4σ²) = 1.118. The sample size is proportional to the variance, so a blinded design at this effect asks for 25% more units than it needs, and it does so systematically rather than by chance.

Choosing n after looking

Re-estimating the sample size from an interim is the one adaptation with a defence, and the defence is exactly what it costs: an analyst kept blind to the arms measures a spread that contains the effect, so the design overshoots by 1 + Δ²/4σ². Re-estimating the effect instead breaks the error rate.

adaptive · Stopping The area under the window is what the band actually costs. The three windows' weight sequences at a width of 30 lags, drawn against the lag as a share of the window. A truncated window applies a weight of one to every lag inside it and zero outside, which is why its sum is the width and why every conventional charge is right for it — and it is a covariance matrix on almost no sample, so it cannot be used. The Bartlett window falls linearly to zero and its weights sum to exactly 15.000000000000004, which is half the width, at every width: Σ(1 − k/(L+1)) over k = 1 … L is L − L/2. The Parzen window sums to 11.13 here, three eighths of the width, and it gets there by holding a weight near one over the first few lags and then falling faster. A plug-in estimate multiplied by a weight below one is a shrunk estimate, and a shrunk estimate is worth less than a free one — which is the whole of why a charge levied per lag is a charge for parameters the window has already spent.

The charge nobody derived

A band of lags is charged one log-likelihood unit apiece, because that is what a regression coefficient costs. A band's numbers are not regression coefficients, and measuring what they actually cost puts the convention out by a factor of nearly three.

dimension · Criterion The curvature is in the denominator. The optimism a Bartlett band of each width actually costs, divided by that width, on two ways of measuring the width, over 2000 draws at 120 rows. Measured in the weights the band spends — Σ w(k), which is what the earlier field levies its charges on — the reading falls from 0.9528 at two lags to 0.7486 at thirty, so a charge proportional to the summed weights is too dear at one end and too cheap at the other. Measured in the pairs the band uses — Σ w(k)(1 − k/n), because a lag of k is an average over n − k products — the same readings are flat from 4 lags up, at 0.0084 of χ² per width against 0.2359. The correction has no fitted parameter in it: it is a function of the window, the width and the sample size.

The width a band is measured in

A tapered covariance band spends 84% of its own weights at two lags and 74% at thirty. Every charge in the collection is a straight line through the origin in those weights, so it is too dear at one end and too cheap at the other.

curve · Criterion One forecast, and the band the arithmetic puts round it. An AR(1) with φ = 0.75, 60 observations, fitted by least squares and forecast 14 steps ahead. The point forecast decays towards the fitted mean at φ̂^h; the band is ±1.96 standard errors from σ̂²Σψ̂², which grows with the horizon and stops at the unconditional spread 1.72. The dashed pair is the same band computed at the true parameters, which nobody has. The marks past zero are what actually arrived: 12 of 14 inside the band this once, which is one draw and settles nothing.

What the model says next

The usual account of a time series stops at estimation. A forecast asks the other question — not what the parameter is but what the next observation will be — and the band round it is a closed form that grows with the horizon and then stops growing, at a value the series was going to reach anyway.

forecast · Forecast What eight groups say about τ, when the truth is 1. The posterior density for the population spread after eight groups whose standard errors run from 0.5 to 2.1. The shaded band is the central 95% interval, from 1.02 to 4.44; the posterior median is 1.99 and the mean 2.18. The vertical mark at 1.07 is the moment estimate that empirical Bayes substitutes and then treats as known.

What the plug-in forgets

The shrinkage weight needs a population spread, and the population spread has to be estimated from eight numbers. Empirical Bayes estimates it, substitutes it, and proceeds as though it were known — and the interval that comes out covers 79% rather than the 95% it claims.

fullbayes · Shrinkage What x-bar plus or minus 2 sample standard deviations holds, at n = 10. The content of the band is a random variable. Across 20,000 normal samples of 10 it averages 91.1%, its fifth percentile is 74.7%, and it falls short of 95% on 59.9% of samples. The band that is drawn to show where 95% of the data lies.

Two standard deviations of what

The 95.45% inside two standard deviations is a fact about a curve whose centre and width are given. Drawn from ten observations, the same band holds 91.1% on average and less than 95% on 59.9% of samples — and the average is the reading that hides it.

estimated · Bands The curvature is in the denominator. The optimism a Bartlett band of each width actually costs, divided by that width, on two ways of measuring the width, over 2000 draws at 120 rows. Measured in the weights the band spends — Σ w(k), which is what the earlier field levies its charges on — the reading falls from 0.9528 at two lags to 0.7486 at thirty, so a charge proportional to the summed weights is too dear at one end and too cheap at the other. Measured in the pairs the band uses — Σ w(k)(1 − k/n), because a lag of k is an average over n − k products — the same readings are flat from 4 lags up, at 0.0084 of χ² per width against 0.2359. The correction has no fitted parameter in it: it is a function of the window, the width and the sample size.

A lag the sample has less of

A sample autocovariance at lag k is an average over n − k products, not n. Count a band's width in the pairs it actually has and the curvature in its charge goes away, on a correction with nothing fitted in it.

curve · Criterion How wrong the ratio is allowed to be. λ enters only through the weights, so misstating it leaves the estimate unbiased and moves two things — the interval's calibration and its efficiency — both of which are closed forms of the design. Coverage stays at its level over a factor of two in either direction (94.93% at half the truth, 94.27% at twice it) and starts to go at a factor of five. An estimate on hundreds of within-arm degrees of freedom is never wrong by anything like that, which is what makes the feasible rule usable rather than merely definable.

Blinded, and still exact

The one number the exact interval needs is a ratio of within-arm spreads, which is a contrast and contains no mean — so a rule forbidden to look at the effect may compute it, on more degrees of freedom than the interval itself has.

corner · Width What a 95% forecast interval covers, counted. 1200 series of 25 observations from an AR(1) with φ = 0.7, at each horizon, on one set of seeds. The upper line is the interval computed at the true parameters — it covers 95.3% on average, which is the check that σ²Σψ² is the right formula rather than a claim about anything a forecaster can do. The lower line is the same formula fed σ̂² and φ̂: 92.8% at one step and 87.3% at 6. The interval that would cover what it claims is 6.9% wider at one step.

The interval that forgets it estimated

The forecast band is derived for a model whose parameters are known, and then computed by putting estimates into it. Counted, the 95% interval covers 87.3% six steps ahead on twenty-five observations, and the point forecast inside it returns to the mean a third faster than the series does.

forecast · Forecast Four windows, one line, and one that is off it. The optimism measured for each window at a band of 30 lags, against what that window's weights sum to, on 2000 pairs of independent samples of 120 rows. The diagonal is where a window that spent exactly its summed weights would sit. Three of the points are one shape at three levels — the Bartlett window, its square and its cube, whose sums stand in the ratio 6 : 4 : 3 — and they lie on a line through the origin at 0.767 of the diagonal, with 0.033 between the highest and the lowest. Scaling the weights scales the charge by the factor the weights predict, which is what makes the weights the mechanism. The Parzen window has a comparable sum and a different shape, and it sits at 0.871: its weights stay near one over the first few lags, and the first few lags are where the information is. A weight sum treats every lag as equally informative and no sample does.

What a window leaves free

A Bartlett window's weights sum to exactly half its width, which is a candidate for what the band costs. Varying the weights without varying anything else says the weights are the mechanism; varying the shape at the same weight says they are not the arithmetic.

dimension · Criterion Least squares estimates persistence low, by an amount with a formula. 3000 series of 50 observations at each persistence. The lower curve is the counted bias of the least-squares estimate of φ, and the open marks on it are −(1 + 3φ)/n, computed rather than fitted. The upper curve is the bias left after adding that quantity back, evaluated at the estimate rather than at the truth nobody has: -0.0020 at φ = 0.3, -0.0023 at φ = 0.5, -0.0039 at φ = 0.7, -0.0059 at φ = 0.8, -0.0108 at φ = 0.9, -0.0165 at φ = 0.95. The formula is a leading-order expression and it understates the bias where the persistence is nearest one — -0.0882 counted against -0.0770 predicted at φ = 0.95, which is the corner of the parameter space every one of these approximations is worst in.

Correcting the persistence

Least squares estimates how much a series remembers of itself as smaller than it is, at every value it can take, by an amount with a closed form. Subtracting that amount back is one line of arithmetic, and what the line costs is variance.

evaluation · Bias Where to look depends on the answer. The information a single run at time t carries about the rate of an exponential decay, (∂η/∂θ)² = t²·exp(−2θt), at three values of θ. Each curve has one maximum and it is at t = 1/θ exactly — marked, and found by a search over 8,001 settings that was never told the formula. Nothing in a linear model behaves this way: there the information matrix is X′X and the parameters are not in it, so a design can be chosen once and used whatever the answer turns out to be. Here the design is optimal at a guess, and the three curves are three different experiments for one model.

The design that needs the answer

Every design this site has computed is optimal whatever the experiment turns out to say, because X′X does not contain the parameters. For a non-linear model it does, so the best place to take a measurement is a function of the number the measurement exists to find — and guessing it three times too low costs two and a half times more than guessing it three times too high.

criteria · Local design A filled value is not an observation. What a 95% interval for the slope actually covers after each way of handling 35.0% missing outcomes, counted over 4000 studies of 200 rows. Dropping the incomplete rows covers 95.93%. Filling with the observed mean covers 13.85%, because the estimate itself has moved. Filling with a fitted value covers 80.85% against a closed prediction of 79.73%: the estimate is right and the reported standard error is short by a factor of 0.6567 against a predicted 0.6500, because the residual sum of squares is divided by the whole sample's degrees of freedom. Adding residual noise recovers the spread and covers 85.78% against a predicted 84.62%, since the interval still ignores the variance of having imputed at all.

One imputation is not an observation

Three ways of filling a missing outcome, under a mechanism that makes dropping the rows beyond reproach. Filling with the observed mean covers 13.85%, filling with a fitted value covers 80.85%, adding noise covers 85.78%, and the thing all three were meant to improve on covers 95.93%.

missing · Missingness What a pilot buys, σ = 1 against 3. Each point is 6,000 two-stage experiments of 100 units: a pilot of m per arm, then the rest split by the pilot's own estimate of the two spreads. Above the line the pilot has made the experiment worse than not bothering. The best pilot here is 8 per arm at 0.809, against 0.800 for a designer who knew the spreads — so the rule recovers 96% of what knowing them is worth. A larger pilot estimates the ratio better and has less left to apply it to, which is why the curve turns.

Allocating on a guess

Every allocation rule in this field is a function of quantities the experiment is being run to find out. Fed a pilot's estimate of them, the rule that minimises the variance makes the experiment worse than not bothering — until the arms differ by about a factor of two, which is further than anyone would guess.

allocation · Allocation The correction does not arrive at the truth, it passes it. The average decay factor a forecast applies to the last observation, at φ = 0.85 and 50 observations, 3000 series per horizon. The middle curve is φʰ, what the model actually does. Below it is the uncorrected forecast, which uses φ̂ʰ and reverts too fast — 24.8% short at h = 4, 30.0% short at h = 6, 32.7% short at h = 8. Above it is the forecast built on the corrected estimate, which overshoots, and the reason is arithmetic rather than a bad correction: raising an unbiased estimate to a power does not give an unbiased estimate of the power, and the higher the power the more the spread of φ̂ is converted into overshoot.

The repair that moves the wrong number

Correcting the bias in a persistence parameter is one line of arithmetic that works. Feeding the corrected estimate into a forecast repairs the number everybody looks at, makes the forecast worse by squared error at moderate persistence, and improves the interval for a reason that has nothing to do with bias.

evaluation · Bias The scale moves the width; the curve does not. The band width each charge picks, averaged over 150 draws of 120 rows. The two conventions — a unit a lag and half a log n a lag — pick 6.08 and 3.65 lags. The four charges derived from the measured optimism pick 15.05, 15.60, 15.47 and 15.44, against a best width on the draw of 14.13. So the scale a charge is levied on moves the width by a factor of 4.27 and the shape of the charge moves it by 3.6%. None of the six is an estimate of the draw's own best width: the correlations are -0.006, -0.003, 0.017, -0.001, 0.016, -0.004.

What a better charge buys

Four charges derived from the same measurements pick band widths within six per cent of each other and deliver errors within two per cent of the gap any of them leaves. The scale a charge is levied on decides the width; the shape of the charge decides nothing.

curve · Criterion The extra 1/m, and the correction nobody quotes. What a pooled 95% interval covers against the number of imputations, counted over 2000 studies of 200 rows at 35.0% of outcomes missing. Rubin's rules — total variance W̄ + (1 + 1/m)B, read against a t distribution on (m − 1)(1 + W̄/((1 + 1/m)B))² degrees of freedom — cover 94.10% at two imputations and reach their promise by 5, at 95.25%. Dropping the (1 + 1/m) factor takes two imputations to 93.10%; using a normal quantile instead of the degrees-of-freedom correction takes it to 92.55%; dropping both takes it to 91.45%. The median degrees of freedom at two imputations is 12.95, which is why the second correction is the larger.

The variance between imputations

Pooling several filled datasets covers 94.10% at two imputations and reaches its promise at five, where a single fill covered 85.78%. The correction everybody quotes is the smaller of the two doing the work — 1.00 ± 0.22 points against 1.55 ± 0.28.

missing · Missingness Coverage against block length, 120 rows. The coverage of the t interval for the mean of a first-order autoregression at 0.7 on 120 rows, at every block length on a fine grid, with its scale read from the disjoint blocks and from a block at every starting row, over 8,000 draws. The squared-error length for disjoint blocks at the true persistence is 13.9 rows, and the same algebra for the overlapping scale gives 16.0, marked. The overlapping interval first reaches 95% — to within a tenth of a point — at blocks of 32 rows, where it covers 94.9% at a width of 1.056; at the overlapping squared-error length it covers about 92.5%.

A length chosen for the interval

The squared-error rule for a block length weighs the scale's bias against its variance. Rebuilt for a block at every starting row, whose variance is two thirds as large, it asks for blocks fourteen per cent longer and buys four tenths of a point of coverage. A t interval has already paid for the scale's variance in its multiplier, so its own coverage is predicted by the bias alone — within seven tenths of a point up to blocks of twenty-eight rows — and a rule that chooses the shortest length that prediction says will cover asks for blocks two and a half to three times longer and lands within about a point of 95% at 120, 240 and 480 rows. The price is width, and it is paid where the series is short.

taper · Bootstrap Largest where least is needed. What the pairs correction supplies against what each window's measured profile needs, across this field's plateau, over 2000 draws at 120 rows. Both are stated as the multiplicative rise the charge per unit of width has to take between four lags and thirty. What the correction supplies is arithmetic — (1 − μ(4)/n)/(1 − μ(30)/n), where μ is the mean lag of the weight the band adds — and it runs 1.0795, 1.0580, 1.0456, 1.0539 for the four windows. What the measurement needs runs 1.1076, 1.2928, 1.2296, 1.6550. The two orderings are opposite: the plain Bartlett window has the longest mean lag, so it gets the biggest correction, and the flattest profile, so it needs the smallest. They coincide to 0.9746 of each other, and nowhere else does the correction account for more than 85.0% of the fall.

What the correction assumes

A correction with nothing fitted in it repairs one window of four. The reason is that its size is set by where a window puts its weight and the curvature it must repair is set by something else — and for one window at one sample size the two happen to agree.

curve · Criterion Estimating a weight you already know is worth doing. The variance of an inverse-probability estimate weighted by a propensity fitted from the sample, over the variance of the same estimate weighted by the true propensity, paired on the same 500 samples of 600 units at each of five settings. Every reading is below one: the stabilised estimator keeps 27.8% of its true-weight variance where the assignment is nearly a coin toss and 72.0% where it is nearly decidable, and the unstabilised one 30.0% and 49.3%. Neither estimator is materially biased, so this is a variance rather than a trade. The true weights are right about the population and know nothing about the draw; the fitted weights are the value that sets this draw's own imbalance to zero, and that imbalance was what the variance was made of.

The estimated weight is the better one

The propensity is known exactly here, so it can be weighted by — and estimating it from the same data and weighting by that gives a variance ratio of 0.4769 on paired draws. The reason is a projection: the draw's own imbalance explains 56.33% of the true-weight variance and 0.05% of the estimated-weight one.

weights · Weighting Storey's estimate of the share of true nulls over twenty thousand families, independent and correlated at 0.6. The true share is 0.5. Independent tests: mean 0.610, spread 0.160, below half the truth in 0.92% of families. Correlated at 0.6: mean 0.609, spread 0.240, below half the truth in 9.33%.

Estimating how many nulls are true

Benjamini–Hochberg at 5% delivers 2.55% when half of twenty nulls are false, because it cannot tell how many are. Storey's estimate of that share, read off the p-values above one half, spends the rest and finds 81.93% of the real effects instead of 74.70% on independent tests. Correlated at 0.9, the same procedure reports a finding in 19.29% of families in which every null is true.

multiplicity · Multiplicity The average decay factor each route produces, φ = 0.85, 6 steps ahead. The truth is φ^6 = 0.3771. no correction averages 0.2616 with a spread of 0.1646 and a squared forecast error of 3.2516; the formula, on the persistence averages 0.4213 with a spread of 0.2528 and a squared forecast error of 3.4827; the bootstrap, on the persistence averages 0.4355 with a spread of 0.2655 and a squared forecast error of 3.5120; the bootstrap, on the decay factor averages 0.3375 with a spread of 0.2278 and a squared forecast error of 3.4132. 800 series, 100 bootstrap refits each.

Correcting the forecast instead

The complaint against the usual repair is that a correction aimed at the persistence lands on the wrong quantity. Aiming it at the decay factor the forecast actually uses fixes exactly that — the error stops compounding with the horizon, 69.7% becomes 9.5% at twelve steps — and the forecast still gets worse.

evaluation · Bias What the guess is worth, when it is worth anything. The variance cost of an even split relative to the variance-minimising one for a risk difference, against the first arm's proportion, with the second at 0.3. The cost is a pure number: it does not depend on the trial's size. It is exactly zero at 0.3 and at 0.70, where the two arms have the same p(1 − p); it is 0.19% at a half and 4.36% at a tenth. Across the whole range from a tenth to nine tenths it never exceeds 4.36%, which is what the variance-minimising rule is worth here — and what it is worth is the reason it is safe to use with a guess.

The arm whose variance is its answer

With a binary outcome the allocation rule is a function of the proportions the trial exists to estimate. It costs at most 4.36% of variance to ignore it anywhere between a tenth and nine tenths, because √(p(1−p)) stays within a factor of two of its peak across 98% of the unit interval.

allocation · Allocation What four length rules cover, 120 rows. The coverage of the t interval on the overlapping scale at 120 rows, when the block length is the shortest one predicted to reach 94.5% under each of four readings of the share, over 4,000 draws of each law. AR(1) at 0.7: geometric 96.2%, read to the chosen lag 93.2%, read to lag 6 93.7%, known 96.5%. a five-period moving average: geometric 96.5%, read to the chosen lag 93.5%, read to lag 6 94.5%, known 95.3%. a break, 0.95 then 0.65: geometric 95.1%, read to the chosen lag 93.0%, read to lag 6 92.5%, known 95.2%. the break turned over: geometric 94.8%, read to the chosen lag 92.5%, read to lag 6 92.2%, known 95.0%.

A share read off the sample

A block length chosen for an interval's coverage needs the share of the mean's variance the scale will find, and the published rule reads it from a first-order autoregression at the sample's lag-one value. Reading it from the sample's own autocovariances instead was supposed to survive series nobody had modelled. At 120 rows it covers 92.5% to 93.5% on all four laws — worse than the wrong shape on every one — because a centred series' autocovariances sum to exactly zero and the share they imply is too generous; read to lag forty it covers 84%. The assumed shape costs width, not coverage: 17% too wide where dependence stops at lag four, and nothing visible under a break, where the interval covers for a reason the share knows nothing about.

taper · Bootstrap The error of three estimates against the length of the series, AR(1) at 0.7. Under AR(1) at 0.7, the root-mean-square error of three estimates of the long-run variance, as a share of the truth, at 120, 240, 480, 960 rows. a first-order autoregression: 45.5%, 33.3%, 23.1%, 15.8%; an autoregression, order by AIC: 50.0%, 39.3%, 26.3%, 18.7%; the trapezoid block, best length: 42.4%, 32.9%, 25.4%, 19.5%.

A model against a window

A block resample's long-run variance is attenuated by its window, and the construction that escapes the window is a model: fit an autoregression and sum its dependence to every lag in closed form. Measured on both halves of the answer, the escape is narrower than it looks. Correctly specified, on 120 rows of a first-order autoregression, the model's error is 45.5% of the truth against the trapezoid block's 42.4% — it removes the bias and buys a spread larger than the bias was — and it pulls ahead only from about 480 rows. Wrongly specified, on a five-period moving average, it is off by 84% to 105% at every length, and even an order chosen by the criterion trails the window throughout.

crossing · Bootstrap Which side the misses fall on, skewed shocks, 95.0% intervals. With skewed shocks and the model's parameters known, the share of 10,000 forecasts that fall below and above a nominal 95.0% normal interval, at each horizon: 1 ahead, 0.0% below and 5.2% above; 2 ahead, 0.0% below and 4.9% above; 4 ahead, 0.0% below and 4.6% above; 8 ahead, 0.0% below and 4.2% above; 12 ahead, 0.0% below and 4.9% above.

Shocks that are not normal

Every forecast interval in this field reads a normal quantile, and every series it was scored on had normal shocks. Give a first-order autoregression shocks with heavy tails and the 95% interval still covers about 95%; its 99% interval covers 98.2%. Give it skewed shocks and the 95% interval covers 94.8% one step ahead with every miss on one side — 0.0% below, 5.2% above — and the horizon does not repair it: twelve steps ahead the misses are still 0.0% below and 4.9% above, because a persistence of 0.7 keeps the error's skewness at 1.11 however far out the forecast goes. A two-sided coverage count cannot see any of this.

forecast · Forecast A penalty that counts independent rows. Regret of five selection rules over fifteen candidates as the design and the errors are made persistent at the same coefficient, 300 draws at each of 7 settings, scored on fresh rows. Akaike's criterion gives up 0.0193, 0.0196, 0.0294, 0.0428, 0.0666, 0.1075, 0.1281; charging each coefficient its own variance inflation, with the persistences read off the sample, gives up 0.0188, 0.0231, 0.0339, 0.0375, 0.0496, 0.0656, 0.0652, and with them given 0.0193, 0.0236, 0.0348, 0.0374, 0.0426, 0.0469, 0.0318. The hold-out gives up 0.0358, 0.0365, 0.0393, 0.0453, 0.0594, 0.0913, 0.1087 and Schwarz's criterion 0.0355, 0.0295, 0.0331, 0.0397, 0.0575, 0.0849, 0.1047.

A penalty that counts independent rows

Akaike's penalty charges a fit two error variances a coefficient because each coefficient is fitted from n independent rows. When the design and the errors both persist, a row is worth less than a row, and the honest charge is twice the coefficient's variance inflation. On a hundred and twenty rows at persistence 0.7 that charge is 34.7 against an optimism counted at 34.5, where Akaike's charges 10. Charged it, the criterion's regret at 0.95 falls from 0.128 to 0.032 with the persistences given and to 0.065 when they are read off the sample — and at 0.5, where the charge is right and Akaike's is half of it, Akaike's still selects better.

proxy · Forecast Reading the draw changes what is charged, not what is tracked. The correlation between the band width each rule picks and the best band width on the same draw, over 400 draws. The three fixed charges read -0.069, -0.066, 0.012. The three that read the sample read -0.012, -0.019, -0.041. None of the six is distinguishable from nothing. The statistic the first plug-in reads does vary — the draw's own summed squared autocorrelation runs from 2.06 to 10.19 with a mean of 4.36 — so the failure is not that the charge stopped moving. It is that what it moves with carries no information about which width this draw wanted.

A charge that reads the draw

Three charges built to read the sample track the best band width on their own draw at −0.012, −0.019 and −0.041, deliver more error than the fixed rule they are calibrated to, and pick a width half again as variable. The statistic moves; the answer does not.

curve · Criterion Five treatments of an estimate above one, φ = 0.95, n = 25. The correction exceeds one on 31.1% of series at this setting. left where it lands: squared forecast error 12.828, average decay factor 0.7974 against a true 0.7351; capped at 0.995: squared forecast error 5.680, average decay factor 0.5950 against a true 0.7351; capped at 1 − 1/n: squared forecast error 5.535, average decay factor 0.5256 against a true 0.7351; correction scaled to fit: squared forecast error 5.535, average decay factor 0.5256 against a true 0.7351; correction refused where it leaves: squared forecast error 5.868, average decay factor 0.4423 against a true 0.7351.

The correction that leaves the region

The bias correction adds (1 + 3φ̂)/n whatever φ̂ is, so it pushes the estimate above one whenever φ̂ exceeds (n − 1)/(n + 3) — on 31.1% of series at φ = 0.95 and twenty-five observations. Five obvious things to do about it differ by a factor of 2.3 in squared forecast error, and none of them is documented as a choice.

evaluation · Bias Too small breaks it and too large does not. Coverage of the weighted interval against the factor the true likelihood ratio is multiplied by, at a test population 80.0% drawn from the noisier group and 200 calibration points. The exact weight is the factor of 1 and covers 95.70%. Overstating it costs nothing: 96.13% at sixteen times too large. Understating it costs, and costs steeply below about a half — 94.93% at half, 88.37% at an eighth and 67.90% at a thirtieth. The question this answers was whether a wrong weight degrades smoothly or falls off a cliff, and the answer is that it does neither symmetrically: the curve is smooth and one-sided.

The weight that has to be estimated

A likelihood ratio sixteen times too large costs 5.5% of interval width and no coverage at all; one a thirtieth of the right size covers 67.90%. The estimate from a batch of five unlabelled covariates covers 95.10% against an exact repair's 95.30%, and the binomial says why.

conformal · Exchangeability How often seven 95% intervals cover the small groups of a league table whose size relation is steepening below the median size. Groups of twenty patients or fewer, 600 tables of a hundred groups. Coverage: one centre, plugged in 58.0% (average width 2.81); one centre, its uncertainty added 58.5% (average width 2.83); a fitted line, plugged in 83.1% (average width 1.58); a fitted line, its uncertainty added 88.4% (average width 1.77); a fitted quadratic, plugged in 85.5% (average width 1.53); a fitted quadratic, its uncertainty added 93.7% (average width 1.92); the true relation and spread 94.8% (average width 1.55).

The interval a small group is given

A league table's 95% interval for a small group, pooled towards a size relation fitted from the table, covers 92.8% of the time when the relation is straight and 83.1% when it steepens below the median — and 77.4% for the groups of eight patients or fewer. Adding the fitted centre's own uncertainty, weighted by how much the group borrows from it, brings the line to 88.4% and a fitted quadratic to 93.7%, and the quadratic's interval covers 94.6% of the smallest groups at every shape tried. One centre, plugged in, covers the smallest groups 35.2% of the time when the relation steepens, and no added variance repairs a centre in the wrong place.

borrowed · Shrinkage What the forecast interval is short by, φ = 0.85, 6 steps ahead. The plug-in interval covers 88.42% against a claimed 95%. Correcting the variance recovers 0.56 points, propagating the persistence's own standard error recovers 0.40, correcting the persistence recovers 2.66, and all three together recover 4.20 — leaving 2.38 points unaccounted for.

What the interval is short by

The forecast interval covers 88.42% where it claims 95%. Correcting the persistence recovers 2.66 points, correcting the innovation variance 0.56, propagating the persistence's own standard error 0.40 — and all three together recover 4.20 of the 6.58, leaving a residual none of the standard repairs reaches.

evaluation · Bias Every group's 90% rank interval in one league table whose size relation is steepening below the median size, with its true rank. A hundred groups from four to four hundred patients, ranked from their posterior under a fitted quadratic size relation whose uncertainty is drawn once per table. The groups of twenty patients or fewer have rank intervals 30.1 places wide on average, and 34 of their 36 true ranks fall inside; the groups of over a hundred, 32.1 places, with 27 of 30 inside.

A ranking with its uncertainty

A league table's reader wants a rank, and a rank interval inherits a fitted size relation's errors differently from an effect interval. When the relation steepens below the median size, a fitted line's 95% interval for a small group's effect covers 83.1% of the time; its 90% interval for the same group's rank covers 95.6%, because the line's error lifts the small groups into the crowded middle of the table, where rank intervals are wide. The error surfaces instead in the large groups' ranks, 87.6%, and in width: 36.9 places against the 27.3 the true relation needs. A quadratic's uncertainty has to be drawn once per table, since it moves every small group together — added group by group it over-covers at 92.8%.

borrowed · Shrinkage Four laws, one flat floor. The error a band of each fixed width delivers, averaged over 400 draws of 120 rows under each law, drawn above that law's own best single width. From one lag to the best width the error falls by 0.02763 under AR(1) at 0.8, 0.02517 under a five-period moving average, 0.01122 under long memory at d = 4/9, 0.03507 under a break in the persistence; from six lags to thirty the whole surface spans only 0.00269, 0.00445, 0.00255, 0.00521. The best single width is 12, 24, 24, 16 lags. The flattest surface end to end is the one under long memory at d = 4/9, which is the law a steeper surface was expected from.

The surface under every law

A draw-reading charge for a covariance band failed on one law, and the failure was deferred to a law with a steeper error surface. Run under all four laws in the collection, the surface is flat from six lags to thirty under every one, no draw-reading charge tracks anything, and long memory — the law expected to be steepest — is the flattest end to end. What the law moves is the level of the charge, not whether a draw can be read.

curve · Criterion Five 95% intervals for an AR(1) forecast 6 steps ahead at φ = 0.85, fifty observations. Coverage on 4,000 series, the same series for every interval, each simulated interval built from 499 simulated futures. The plug-in formula: 88.48%. The formula, all three repairs: 92.80%. Simulated: parameters drawn: 92.55%. Simulated: residual bootstrap: 88.25%. Simulated: bootstrap, bias removed: 93.47%.

An interval built from simulated futures

A forecast interval read off simulated futures needs no delta method and no normal multiplier, and was expected to close the residual the three repairs to the formula leave. It does not close it on its own: the plain residual bootstrap covers 88.25% six steps ahead at φ = 0.85, the plug-in formula's 88.48% to within the count, because it simulates around the same biased persistence. With that bias removed inside the bootstrap it covers 93.47%, the best of five intervals, and its shortfall stops growing with the horizon — 93.80% at twelve steps, where the repaired formula covers 91.85%.

evaluation · Bias

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloDependenceMean squared errorCoverageDegrees of freedomClosed formInformation criterionParameter uncertaintyForecast horizonLong-run varianceOptimismSample autocovariance

All concepts