Normal Distribution
nothing is called 95% until it has been counted
Concept

Forecast error — where it appears

The difference between an observation and what a model said it would be, made of shocks that had not happened when the forecast was issued. Its variance grows with the horizon, and an interval built for one step is not an interval for two whatever the point forecast does.

Named by 15 essays across 6 fields — each of them below, with the objects they name alongside it.

One forecast, and the band the arithmetic puts round it. An AR(1) with φ = 0.75, 60 observations, fitted by least squares and forecast 14 steps ahead. The point forecast decays towards the fitted mean at φ̂^h; the band is ±1.96 standard errors from σ̂²Σψ̂², which grows with the horizon and stops at the unconditional spread 1.72. The dashed pair is the same band computed at the true parameters, which nobody has. The marks past zero are what actually arrived: 12 of 14 inside the band this once, which is one draw and settles nothing.

What the model says next

The usual account of a time series stops at estimation. A forecast asks the other question — not what the parameter is but what the next observation will be — and the band round it is a closed form that grows with the horizon and then stops growing, at a value the series was going to reach anyway.

forecast · Forecast One comparison, and the two error bars it can be given. 60 rolling origins, a window of 60 observations, forecasts 4 steps ahead, at the persistence φ = 0.8256 where the two benchmarks have exactly equal population mean squared error. Each mark is one origin's difference in squared error; the horizontal line is their mean, 0.6522. The two vertical bars at the right are ±1.96 standard errors round that mean computed two ways — 0.5337 treating the differences as independent, 0.6880 allowing for the overlap between neighbouring forecasts. The null is true here by construction, so an interval that excludes zero is a mistake, and the narrow one does it far more often than the wide one.

Which forecast is better

Two forecasters, one series, and a difference in mean squared error. Whether that difference is real is a hypothesis test, its terms are not independent, and the standard error it needs is not the one a t-test computes.

evaluation · Forecast What a 95% forecast interval covers, counted. 1200 series of 25 observations from an AR(1) with φ = 0.7, at each horizon, on one set of seeds. The upper line is the interval computed at the true parameters — it covers 95.3% on average, which is the check that σ²Σψ² is the right formula rather than a claim about anything a forecaster can do. The lower line is the same formula fed σ̂² and φ̂: 92.8% at one step and 87.3% at 6. The interval that would cover what it claims is 6.9% wider at one step.

The interval that forgets it estimated

The forecast band is derived for a model whose parameters are known, and then computed by putting estimates into it. Counted, the 95% interval covers 87.3% six steps ahead on twenty-five observations, and the point forecast inside it returns to the mean a third faster than the series does.

forecast · Forecast A test between nested models, under a null that is true. 1000 comparisons: an AR(1) truth, forecast by a fitted AR(1) and by a fitted AR4 whose extra coefficients are zero. In population the two forecasts are identical, so every rejection is false. The larger model's mean squared error is 1.1663 against 1.0583 — worse, by exactly the noise in estimating coefficients that are not there — and the ordinary test therefore declares the smaller model significantly better 67.2% of the time. Read one-sided in the direction anybody asks about, it finds the larger model better 0.0% of the time. Adding the squared difference between the two forecasts back into the loss differential puts the level at 4.9%.

When one model contains the other

The comparison a forecaster most often wants is between a model and the same model with one more term. That is exactly the comparison the standard test cannot make — and it fails by declaring the smaller model significantly better, more confidently the more data it is given.

evaluation · Forecast What each criterion selects, at 50 observations. 700 series from an AR(2) with coefficients 0.6 and -0.3, every order from 0 to 8 fitted to the same 42 responses so the log-likelihoods are comparable. AIC finds the true order 55.1% of the time and lands above it 25.7%; BIC finds it 54.1% and lands above it 4.0%. The closed form for one extra lag is P(χ²₁ > 2) = 15.73% for AIC, which does not depend on n at all, and P(χ²₁ > ln n) = 4.79% for BIC at this size, which falls to zero. Under the true order is the other failure and it is BIC's: 41.9% against 19.1%.

Choosing the order

One criterion is consistent and one is not, which is the whole of what gets said about them. At two hundred observations the consistent one is right 95% of the time and the other 70%; at fifty they are both right 54% of the time and wrong in opposite directions, and consistency has not started to mean anything yet.

forecast · Order selection What the long-run relation is worth, at α = -0.2. Root mean squared one-step forecast error of the error-correction model divided by that of the model fitted on differences alone; below one means the levels helped. With the equilibrium known the ratio is 0.929 at 100 observations and settles on 0.905 by 3,200, against a closed form of 0.905 that mentions no sample size at all; the excess at short series is the cost of fitting three coefficients on fifty observations. With the equilibrium estimated as well it is 1.127 at 100 — worse than differencing — and 0.914 at 3,200. The gap between the two curves is the cost of not knowing β.

The cost of differencing a pair

Differencing two cointegrated series makes every standard error honest and throws away the one thing known about where they are going. The error-correction model forecasts better by exactly what a closed form says — and at four hundred observations it is better on four series in five and worse on average.

cointegration · Dependence The correction does not arrive at the truth, it passes it. The average decay factor a forecast applies to the last observation, at φ = 0.85 and 50 observations, 3000 series per horizon. The middle curve is φʰ, what the model actually does. Below it is the uncorrected forecast, which uses φ̂ʰ and reverts too fast — 24.8% short at h = 4, 30.0% short at h = 6, 32.7% short at h = 8. Above it is the forecast built on the corrected estimate, which overshoots, and the reason is arithmetic rather than a bad correction: raising an unbiased estimate to a power does not give an unbiased estimate of the power, and the higher the power the more the spread of φ̂ is converted into overshoot.

The repair that moves the wrong number

Correcting the bias in a persistence parameter is one line of arithmetic that works. Feeding the corrected estimate into a forecast repairs the number everybody looks at, makes the forecast worse by squared error at moderate persistence, and improves the interval for a reason that has nothing to do with bias.

evaluation · Bias The ceiling a multiplier cannot reach past. A wild-type resampling forms e*_t = e_t·w_t with the multiplier independent of the residual, so what comes out has autocovariance γ_resid(k)·γ_w(k) — the residuals' own, multiplied by the multiplier's. Since |γ_w| ≤ 1 the reference distribution's dependence is bounded above by the residuals', and the residuals' is already below the errors'. The two shortfalls compose. For a block of ℓ the multiplier's autocorrelation is exactly the triangle (1 − k/ℓ)⁺, drawn here as the dashed prediction against the realised resamples at ℓ = 5; the bound is attained only at ℓ = n, where the reference distribution is built from one sign.

What a multiplier cannot keep

Two reasons were named for the quarter a blocked resampling falls short, and taking either away makes the gap larger. What is left is a bound — a multiplier can only take dependence out, and the residuals' own is already below the errors'.

effective · Reference The average decay factor each route produces, φ = 0.85, 6 steps ahead. The truth is φ^6 = 0.3771. no correction averages 0.2616 with a spread of 0.1646 and a squared forecast error of 3.2516; the formula, on the persistence averages 0.4213 with a spread of 0.2528 and a squared forecast error of 3.4827; the bootstrap, on the persistence averages 0.4355 with a spread of 0.2655 and a squared forecast error of 3.5120; the bootstrap, on the decay factor averages 0.3375 with a spread of 0.2278 and a squared forecast error of 3.4132. 800 series, 100 bootstrap refits each.

Correcting the forecast instead

The complaint against the usual repair is that a correction aimed at the persistence lands on the wrong quantity. Aiming it at the decay factor the forecast actually uses fixes exactly that — the error stops compounding with the horizon, 69.7% becomes 9.5% at twelve steps — and the forecast still gets worse.

evaluation · Bias The displacement per parameter, under five predictor laws. The out-of-sample differential one extra parameter costs a correct candidate, at a window of 80 rows, by two routes that share no arithmetic: counted from rolling forecasts over 1,500 draws of the fifteen-candidate table, and implied by the expected leverage of a fresh row over 80,000 draws of the predictors alone. The Gaussian closed form is 0.01386. normal: 0.01382 counted, 0.01386 from the leverage; Student's t, 8: 0.01379 counted, 0.01407 from the leverage; Student's t, 5: 0.01419 counted, 0.01448 from the leverage; Student's t, 3: 0.01751 counted, 0.01780 from the leverage; exponential: 0.01465 counted, 0.01487 from the leverage.

The displacement on a heavy tail

A correct candidate with one more parameter is behind out of sample by a closed-form amount, and the closed form rests on one Gaussian fact: the expected leverage of a fresh row is q/(R − q − 1). On predictors that are not Gaussian it becomes whatever the leverage is, and two routes that share no arithmetic agree on what that is. With a finite fourth moment it barely moves — 1.5% more per parameter at eight degrees of freedom, 4.5% at five, 7.3% on an exponential. At three degrees of freedom it is 28% more, and the estimate of it will not settle: four batches of twenty thousand draws disagree by more than a quarter.

select · Multiplicity Which side the misses fall on, skewed shocks, 95.0% intervals. With skewed shocks and the model's parameters known, the share of 10,000 forecasts that fall below and above a nominal 95.0% normal interval, at each horizon: 1 ahead, 0.0% below and 5.2% above; 2 ahead, 0.0% below and 4.9% above; 4 ahead, 0.0% below and 4.6% above; 8 ahead, 0.0% below and 4.2% above; 12 ahead, 0.0% below and 4.9% above.

Shocks that are not normal

Every forecast interval in this field reads a normal quantile, and every series it was scored on had normal shocks. Give a first-order autoregression shocks with heavy tails and the 95% interval still covers about 95%; its 99% interval covers 98.2%. Give it skewed shocks and the 95% interval covers 94.8% one step ahead with every miss on one side — 0.0% below, 5.2% above — and the horizon does not repair it: twelve steps ahead the misses are still 0.0% below and 4.9% above, because a persistence of 0.7 keeps the error's skewness at 1.11 however far out the forecast goes. A two-sided coverage count cannot see any of this.

forecast · Forecast Five treatments of an estimate above one, φ = 0.95, n = 25. The correction exceeds one on 31.1% of series at this setting. left where it lands: squared forecast error 12.828, average decay factor 0.7974 against a true 0.7351; capped at 0.995: squared forecast error 5.680, average decay factor 0.5950 against a true 0.7351; capped at 1 − 1/n: squared forecast error 5.535, average decay factor 0.5256 against a true 0.7351; correction scaled to fit: squared forecast error 5.535, average decay factor 0.5256 against a true 0.7351; correction refused where it leaves: squared forecast error 5.868, average decay factor 0.4423 against a true 0.7351.

The correction that leaves the region

The bias correction adds (1 + 3φ̂)/n whatever φ̂ is, so it pushes the estimate above one whenever φ̂ exceeds (n − 1)/(n + 3) — on 31.1% of series at φ = 0.95 and twenty-five observations. Five obvious things to do about it differ by a factor of 2.3 in squared forecast error, and none of them is documented as a choice.

evaluation · Bias What each wrong count costs, 4 steps ahead. Squared forecast error 4 steps ahead at each imposed rank, relative to the correctly specified fit, at 200 observations. With 1 genuine relations, imposing 0 costs 13.3% and imposing 2 costs 4.8%. With 2 genuine relations, imposing 1 costs 15.6% and imposing 3 costs 2.5%. Under-counting is the more expensive mistake in both systems, and it is the one the procedure's level does not bound.

Which mistake about the rank costs

On a system with two relations, imposing none costs 29.2% of squared forecast error and imposing three costs 2.5%. The expensive mistake is under-counting, which is the error the procedure's 5% does not bound — so the guarantee protects the cheap side.

systems · Rank Which side the misses fall on. Misses of a nominal 95% forecast interval 1 step ahead on 60 observations of a first-order autoregression at 0.7, split by side. normal shocks: the interval as printed 3.1% below and 3.1% above, shocks from the residuals 3.8% below and 4.2% above, the same, coefficient re-estimated 3.3% below and 4.4% above. Student's t on five: the interval as printed 3.3% below and 3.0% above, shocks from the residuals 3.9% below and 3.0% above, the same, coefficient re-estimated 3.6% below and 3.1% above. skewed shocks: the interval as printed 0.0% below and 6.5% above, shocks from the residuals 4.3% below and 4.1% above, the same, coefficient re-estimated 1.8% below and 4.3% above.

A tail sixty residuals have barely seen

A forecast interval that reads a normal quantile has every one of its misses on one side when the shocks are skewed, and the natural repair is to draw future shocks from the model's own residuals. It works on the side: 0.0% below and 6.5% above becomes 4.3% and 4.1%. It pays for that with coverage, 93.5% to 91.6%, and on sixty observations it does this under normal shocks too. And a nominal 99% interval has a ceiling nothing in the resampling can lift, because one new shock beats all of sixty old ones in either direction with probability 3.3%.

forecast · Forecast How often a test of a supplied cointegrating relation rejects it, by how wrong it is and how long the sample is. 800 systems a point. At 100, 200, 400, 800 observations the right relation is rejected 9.4%, 5.4%, 5.4%, 6.4% of the time; one with c = 0.9, 19.9%, 37.8%, 78.5%, 98.5%; c = 1.1, 16.8%, 37.9%, 76.8%, 97.5%.

A restriction the data can check

A rank-two system of three series is identified by exactly two restrictions on its relations, and any two that describe the estimated plane fit it equally well — their forecasts agree with the unrestricted fit's to the last digit, so the data can neither confirm nor refuse them. Supply one relation in full instead, one restriction more than identification needs, and the surplus becomes a test: a relation a tenth wrong is rejected 37.8% of the time at two hundred observations and 98.5% at eight hundred, the error found half the time halving as the sample doubles. A right relation buys the forecast almost nothing; a fifth wrong costs 13% of it at two hundred observations and 25% at eight hundred.

systems · Rank

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloForecast horizonAutocorrelationMean squared errorPlug in estimateStationarityForecast intervalLeast squaresCoverageModel selectionBias correctionCointegration

All concepts