Normal Distribution
nothing is called 95% until it has been counted
Concept

Forecast horizon — where it appears

How many steps ahead a forecast reaches, and the argument its point estimate and error variance are both functions of. An interval built without reference to it is wrong at every horizon but one, which is a defect no coverage check at a single horizon can see.

Named by 12 essays across 4 fields — each of them below, with the objects they name alongside it.

One forecast, and the band the arithmetic puts round it. An AR(1) with φ = 0.75, 60 observations, fitted by least squares and forecast 14 steps ahead. The point forecast decays towards the fitted mean at φ̂^h; the band is ±1.96 standard errors from σ̂²Σψ̂², which grows with the horizon and stops at the unconditional spread 1.72. The dashed pair is the same band computed at the true parameters, which nobody has. The marks past zero are what actually arrived: 12 of 14 inside the band this once, which is one draw and settles nothing.

What the model says next

The usual account of a time series stops at estimation. A forecast asks the other question — not what the parameter is but what the next observation will be — and the band round it is a closed form that grows with the horizon and then stops growing, at a value the series was going to reach anyway.

forecast · Forecast Three quantities, and only one of them crosses zero. Two forecasts of an AR(1) — the last value carried forward and the mean of the last 60 observations — at 1 step ahead. The curve through zero is σ₁² − σ₂², the difference in expected squared error that a comparison of accuracy tests; it changes sign at φ = 0.4922. The two curves above it are σ₁² − σ₁₂ and σ₂² − σ₁₂, the quantities the two encompassing tests are about, and neither of them comes near zero anywhere: the smallest value either takes across the range is 0.008 times the variance of the series. All three are closed forms in φ, R and h with no simulation in them. Equal accuracy is one hypothesis about this picture and encompassing is another, and a set of numbers can satisfy either without the other.

What the other forecast adds

Two forecasters, one series, and two different questions about them. Which is more accurate has an answer that changes with the persistence of the series; whether either is redundant has an answer that never changes at all.

ranking · Forecast One comparison, and the two error bars it can be given. 60 rolling origins, a window of 60 observations, forecasts 4 steps ahead, at the persistence φ = 0.8256 where the two benchmarks have exactly equal population mean squared error. Each mark is one origin's difference in squared error; the horizontal line is their mean, 0.6522. The two vertical bars at the right are ±1.96 standard errors round that mean computed two ways — 0.5337 treating the differences as independent, 0.6880 allowing for the overlap between neighbouring forecasts. The null is true here by construction, so an interval that excludes zero is a mistake, and the narrow one does it far more often than the wide one.

Which forecast is better

Two forecasters, one series, and a difference in mean squared error. Whether that difference is real is a hypothesis test, its terms are not independent, and the standard error it needs is not the one a t-test computes.

evaluation · Forecast What a 95% forecast interval covers, counted. 1200 series of 25 observations from an AR(1) with φ = 0.7, at each horizon, on one set of seeds. The upper line is the interval computed at the true parameters — it covers 95.3% on average, which is the check that σ²Σψ² is the right formula rather than a claim about anything a forecaster can do. The lower line is the same formula fed σ̂² and φ̂: 92.8% at one step and 87.3% at 6. The interval that would cover what it claims is 6.9% wider at one step.

The interval that forgets it estimated

The forecast band is derived for a model whose parameters are known, and then computed by putting estimates into it. Counted, the 95% interval covers 87.3% six steps ahead on twenty-five observations, and the point forecast inside it returns to the mean a third faster than the series does.

forecast · Forecast The correction does not arrive at the truth, it passes it. The average decay factor a forecast applies to the last observation, at φ = 0.85 and 50 observations, 3000 series per horizon. The middle curve is φʰ, what the model actually does. Below it is the uncorrected forecast, which uses φ̂ʰ and reverts too fast — 24.8% short at h = 4, 30.0% short at h = 6, 32.7% short at h = 8. Above it is the forecast built on the corrected estimate, which overshoots, and the reason is arithmetic rather than a bad correction: raising an unbiased estimate to a power does not give an unbiased estimate of the power, and the higher the power the more the spread of φ̂ is converted into overshoot.

The repair that moves the wrong number

Correcting the bias in a persistence parameter is one line of arithmetic that works. Feeding the corrected estimate into a forecast repairs the number everybody looks at, makes the forecast worse by squared error at moderate persistence, and improves the interval for a reason that has nothing to do with bias.

evaluation · Bias The average decay factor each route produces, φ = 0.85, 6 steps ahead. The truth is φ^6 = 0.3771. no correction averages 0.2616 with a spread of 0.1646 and a squared forecast error of 3.2516; the formula, on the persistence averages 0.4213 with a spread of 0.2528 and a squared forecast error of 3.4827; the bootstrap, on the persistence averages 0.4355 with a spread of 0.2655 and a squared forecast error of 3.5120; the bootstrap, on the decay factor averages 0.3375 with a spread of 0.2278 and a squared forecast error of 3.4132. 800 series, 100 bootstrap refits each.

Correcting the forecast instead

The complaint against the usual repair is that a correction aimed at the persistence lands on the wrong quantity. Aiming it at the decay factor the forecast actually uses fixes exactly that — the error stops compounding with the horizon, 69.7% becomes 9.5% at twelve steps — and the forecast still gets worse.

evaluation · Bias Which side the misses fall on, skewed shocks, 95.0% intervals. With skewed shocks and the model's parameters known, the share of 10,000 forecasts that fall below and above a nominal 95.0% normal interval, at each horizon: 1 ahead, 0.0% below and 5.2% above; 2 ahead, 0.0% below and 4.9% above; 4 ahead, 0.0% below and 4.6% above; 8 ahead, 0.0% below and 4.2% above; 12 ahead, 0.0% below and 4.9% above.

Shocks that are not normal

Every forecast interval in this field reads a normal quantile, and every series it was scored on had normal shocks. Give a first-order autoregression shocks with heavy tails and the 95% interval still covers about 95%; its 99% interval covers 98.2%. Give it skewed shocks and the 95% interval covers 94.8% one step ahead with every miss on one side — 0.0% below, 5.2% above — and the horizon does not repair it: twelve steps ahead the misses are still 0.0% below and 4.9% above, because a persistence of 0.7 keeps the error's skewness at 1.11 however far out the forecast goes. A two-sided coverage count cannot see any of this.

forecast · Forecast Five treatments of an estimate above one, φ = 0.95, n = 25. The correction exceeds one on 31.1% of series at this setting. left where it lands: squared forecast error 12.828, average decay factor 0.7974 against a true 0.7351; capped at 0.995: squared forecast error 5.680, average decay factor 0.5950 against a true 0.7351; capped at 1 − 1/n: squared forecast error 5.535, average decay factor 0.5256 against a true 0.7351; correction scaled to fit: squared forecast error 5.535, average decay factor 0.5256 against a true 0.7351; correction refused where it leaves: squared forecast error 5.868, average decay factor 0.4423 against a true 0.7351.

The correction that leaves the region

The bias correction adds (1 + 3φ̂)/n whatever φ̂ is, so it pushes the estimate above one whenever φ̂ exceeds (n − 1)/(n + 3) — on 31.1% of series at φ = 0.95 and twenty-five observations. Five obvious things to do about it differ by a factor of 2.3 in squared forecast error, and none of them is documented as a choice.

evaluation · Bias What each wrong count costs, 4 steps ahead. Squared forecast error 4 steps ahead at each imposed rank, relative to the correctly specified fit, at 200 observations. With 1 genuine relations, imposing 0 costs 13.3% and imposing 2 costs 4.8%. With 2 genuine relations, imposing 1 costs 15.6% and imposing 3 costs 2.5%. Under-counting is the more expensive mistake in both systems, and it is the one the procedure's level does not bound.

Which mistake about the rank costs

On a system with two relations, imposing none costs 29.2% of squared forecast error and imposing three costs 2.5%. The expensive mistake is under-counting, which is the error the procedure's 5% does not bound — so the guarantee protects the cheap side.

systems · Rank Which side the misses fall on. Misses of a nominal 95% forecast interval 1 step ahead on 60 observations of a first-order autoregression at 0.7, split by side. normal shocks: the interval as printed 3.1% below and 3.1% above, shocks from the residuals 3.8% below and 4.2% above, the same, coefficient re-estimated 3.3% below and 4.4% above. Student's t on five: the interval as printed 3.3% below and 3.0% above, shocks from the residuals 3.9% below and 3.0% above, the same, coefficient re-estimated 3.6% below and 3.1% above. skewed shocks: the interval as printed 0.0% below and 6.5% above, shocks from the residuals 4.3% below and 4.1% above, the same, coefficient re-estimated 1.8% below and 4.3% above.

A tail sixty residuals have barely seen

A forecast interval that reads a normal quantile has every one of its misses on one side when the shocks are skewed, and the natural repair is to draw future shocks from the model's own residuals. It works on the side: 0.0% below and 6.5% above becomes 4.3% and 4.1%. It pays for that with coverage, 93.5% to 91.6%, and on sixty observations it does this under normal shocks too. And a nominal 99% interval has a ceiling nothing in the resampling can lift, because one new shock beats all of sixty old ones in either direction with probability 3.3%.

forecast · Forecast What the forecast interval is short by, φ = 0.85, 6 steps ahead. The plug-in interval covers 88.42% against a claimed 95%. Correcting the variance recovers 0.56 points, propagating the persistence's own standard error recovers 0.40, correcting the persistence recovers 2.66, and all three together recover 4.20 — leaving 2.38 points unaccounted for.

What the interval is short by

The forecast interval covers 88.42% where it claims 95%. Correcting the persistence recovers 2.66 points, correcting the innovation variance 0.56, propagating the persistence's own standard error 0.40 — and all three together recover 4.20 of the 6.58, leaving a residual none of the standard repairs reaches.

evaluation · Bias Five 95% intervals for an AR(1) forecast 6 steps ahead at φ = 0.85, fifty observations. Coverage on 4,000 series, the same series for every interval, each simulated interval built from 499 simulated futures. The plug-in formula: 88.48%. The formula, all three repairs: 92.80%. Simulated: parameters drawn: 92.55%. Simulated: residual bootstrap: 88.25%. Simulated: bootstrap, bias removed: 93.47%.

An interval built from simulated futures

A forecast interval read off simulated futures needs no delta method and no normal multiplier, and was expected to close the residual the three repairs to the formula leave. It does not close it on its own: the plain residual bootstrap covers 88.25% six steps ahead at φ = 0.85, the plug-in formula's 88.48% to within the count, because it simulates around the same biased persistence. With that bias removed inside the bootstrap it covers 93.47%, the best of five intervals, and its shortfall stops growing with the horizon — 93.80% at twelve steps, where the repaired formula covers 91.85%.

evaluation · Bias

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloForecast errorPlug in estimateForecast intervalMean squared errorStationarityCoverageAutocorrelationBias correctionDelta methodParameter uncertaintyAutoregression

All concepts