Shocks that are not normal

The interval that forgets it estimated found the textbook forecast interval for an autoregression too narrow: it treats the fitted coefficients as the truth, the forecast error it describes is the one a known model would make, and the shortfall grows with the horizon as the estimation error compounds. It ended by saying what its numbers were a lower bound on. The model was correctly specified, its order was known, the series was stationary, and the shocks were independent and identically distributed — four assumptions true by construction, and a forecaster has none of them.

One of the four is worth taking away on its own, because the interval’s construction leans on it in a way the others do not. The interval is the point forecast plus or minus 1.96 forecast standard deviations, and 1.96 is a normal quantile. It is right when the forecast error is normal, which it is when the shocks are, and it is a claim about tails when they are not. This essay scores the same interval on the same model with shocks that are not normal: heavier-tailed in both directions, and skewed.

The model and the shocks #

The model is the field’s: a first-order autoregression at 0.7, sixty observations, forecast one to twelve steps ahead. Each interval is scored twice — as printed, with the coefficient and the shock variance estimated from the sixty observations, and with the parameters known, which isolates what the shock law does from what estimation does.

Three shock laws, each with mean zero and variance one so that the forecast variance is the same under all of them:

• normal, the field’s own;
• Student’s t on five degrees of freedom, rescaled — symmetric, with tails far heavier than a normal’s;
• a centred exponential — bounded below at −1 and long above, with a skewness of two.

Ten thousand forecasts per cell, so a coverage near 95% has a standard error of about two tenths of a point.

Heavier tails #

At the 95% level the t shocks make almost no difference. With the parameters known the interval covers 95.0% one step ahead and between 94.7% and 95.2% out to twelve steps; with normal shocks, 94.7% to 95.3%. As printed it covers 93.7% one step ahead and 91.6% twelve steps ahead, against 94.0% and 91.2% with normal shocks — the earlier essay’s shortfall, and the shock law adds nothing to it.

That is because a t on five degrees of freedom, scaled to unit variance, happens to put very nearly five per cent of its mass beyond 1.96 — its tails are heavier, but its variance is spent partly in a sharper centre, and at this one quantile the two balance.

At the 99% level they do not. With the parameters known the interval covers 98.2% one step ahead where it promises 99%, and between 98.2% and 98.6% at every horizon; as printed, 97.5% one step ahead. The misses are split evenly, 0.9% below and 0.9% above one step ahead, so the interval is symmetric and simply too narrow in both tails at once. A forecaster quoting a 99% interval on a series whose shocks are this heavy is quoting one that misses nearly twice as often as it says.

The level at which a heavy tail starts to cost is a property of the tail, not of the interval. A t on five degrees of freedom crosses the normal’s tail probability near the 95% point and stays above it beyond, so every interval wider than 95% is short and every one narrower is long, and the size of the error grows the further out the level is set. That is the same shape a tail the sample never saw found for a sum of exponential draws: the tail of the sum is readable from the source and not from a sample of thirty, because a sample that size holds almost nothing in the region that decides the statement. A series of sixty holds a handful of shocks in the region that decides a 99% interval, and a model that fixes the tail’s shape in advance — here, by reading a normal quantile — is fixing exactly the part the data could not have checked. The forecaster’s 95% interval is protected by a coincidence about where two tails cross; the 99% interval is not, and the 99.9% interval a risk report would quote is worse again.

Skewed shocks, and which side the misses fall on #

With the skewed shocks the 95% interval with the parameters known covers 94.8% one step ahead. By the usual reading that is an interval doing its job. It misses 0.0% of forecasts below and 5.2% above.

One step ahead that has to be so. The forecast error is the next shock, and a centred exponential cannot fall below −1, which is inside the interval’s lower end at −1.96. Every miss is above, and the count of misses happens to land near five per cent because the exponential’s upper tail beyond 1.96 holds about that much.

The horizon was supposed to fix it. The error $h$ steps ahead is a weighted sum of $h$ shocks, and sums of skewed variables become symmetric. They do not here. Two steps ahead the misses are 0.0% below and 4.9% above; four steps, 0.0% and 4.6%; eight steps, 0.0% and 4.2%; twelve steps, 0.0% and 4.9%. The interval’s lower half is empty at every horizon.

So a two-sided coverage of about 95%, the number every essay in this field has scored intervals by, is a sum of two errors, a lower tail with nothing in it and an upper tail with twice its share, and a forecaster who counts misses without asking which side they fall on is told everything is fine.

Why the horizon does not wash it out #

The forecast error $h$ steps ahead is $\sum_{j<h} \psi_j \varepsilon_{n+h-j}$, with $\psi_j = 0.7^j$, and its skewness has a closed form in those weights: the shock’s skewness times $\sum\psi_j^3 / (\sum\psi_j^2)^{3/2}$. One step ahead that is 2.000, the shock’s own. Two steps ahead, 1.477; four, 1.195; eight, 1.114; and it settles at 1.109, not at zero.

A central limit theorem makes a sum symmetric when it is a sum of many comparable terms. The forecast error is a sum whose weights decay geometrically, so however far out the forecast goes it is dominated by the last few shocks, and the effective number of terms is about $(\sum\psi^2)^2 / \sum\psi^4$, which at 0.7 is about three. A sum of three exponentials is still clearly skewed. The interval’s normal quantile is the right one only in the limit of a persistence near one, where the weights decay slowly enough for the sum to have many terms — the limit in which the forecast is least useful.

What the 99% interval does #

At 99% the skewed shocks cost coverage outright. With the parameters known the interval covers 97.3% one step ahead, 97.5% at two steps, 98.0% at four, 98.3% at eight and 97.6% at twelve, every miss above. As printed it covers between 96.1% and 96.7% — nearly three times the misses it promises, all in one direction.

The 95% interval got away with it because the exponential’s upper tail beyond 1.96 holds about five per cent of its mass; its upper tail beyond 2.58 holds more than one, and nothing compensates in the lower tail. A normal quantile applied to a skewed error is right at one level by coincidence and wrong at the others.

The interval as printed #

The comparison so far held the parameters known, so that the shock law’s effect could be seen on its own. A forecaster never has them, and the printed interval carries both defects.

The as-printed columns repeat the earlier essay’s finding under every shock law: the plug-in interval is short by about one point one step ahead and three to four twelve steps ahead, because it forgets that the coefficient was estimated. Under the skewed shocks the estimation adds a small lower tail — 0.0% below one step ahead but 0.9% at eight steps and 0.7% at twelve — because an estimated coefficient below the truth pulls the point forecast down and occasionally lets a forecast fall under the interval.

The two defects are separate and they add. The estimation’s shortfall is symmetric and grows with the horizon; the shock law’s is one-sided and does not. An interval built from simulated futures repaired the first by simulating forecasts from the fitted model with its residuals resampled. A simulation that resamples the residuals rather than drawing normal shocks repairs the second too, in principle — it would draw the skewed shocks the data has — and the earlier essay’s residual bootstrap is exactly that construction. Whether it recovers the lower tail’s emptiness and the upper tail’s excess on sixty observations, which contain only a handful of shocks in the region that decides a 99% interval, is the question it raises.

What a forecaster can check #

The defect is visible before any forecast is scored. The model’s residuals are estimates of its shocks, and their skewness is an estimate of the shocks’ skewness: under the exponential law, a sample of sixty residuals will usually show a skewness well above one. Feeding that into the closed form above gives the forecast error’s skewness at every horizon, and if it is above about a half, a symmetric interval is the wrong shape.

The repair that needs no simulation is an asymmetric interval: the point forecast plus the error’s own quantiles rather than ±1.96 standard deviations. For the exponential at one step ahead the quantiles are known exactly; at longer horizons they are the quantiles of a weighted sum of exponentials, which a few lines of convolution produce. Either way the interval stops being centred on the forecast, and its lower end moves up and its upper end out.

What the model says next set out the forecast as a distribution rather than a number. That distribution has a shape, and the shape is the shocks’ shape blurred by a few steps of averaging — not a normal one, unless the shocks were.

The interval with the right shape #

The asymmetric interval can be computed exactly for the skewed shocks, because the forecast error is a weighted sum of exponentials whose weights are known. Its quantiles, from four hundred thousand draws of that sum, say what the normal interval should have been.

One step ahead, the right 95% interval runs from −0.97 to +2.70 forecast standard deviations around the point forecast, against the normal interval’s ±1.96. It is 6.3% narrower and sits almost entirely above the forecast: its lower end is the exponential’s floor, and it does not waste width below where no forecast can fall. The right 99% interval runs from −0.99 to +4.32, 3.2% wider than the normal one, because its upper end has to reach much further into the exponential’s tail than 2.58 does.

Twelve steps ahead the right 95% interval runs from −2.02 to +3.38 times the one-step shock’s scale, against the normal interval’s ±2.74 at that horizon — 1.7% narrower and still shifted upward by more than a third of a standard deviation. The right 99% interval is 2.7% wider than the normal one. The asymmetry the skewness figure describes is the asymmetry of the interval itself: at every horizon the right interval reaches less far below the forecast and further above it.

So the normal interval is wrong in two directions at once, and which is worse depends on the level. At 95% it is too wide and badly placed; at 99% it is too narrow and badly placed. Its coverage near 95% at the lower level is the coincidence of a width error and a placement error cancelling in the count.

Where it sits among the field’s other defects #

The field has now taken four assumptions out of the textbook interval one at a time. Choosing the order took away the known order, and the interval after the choice found that choosing it from the same data costs coverage symmetrically. The earlier essay took away the known coefficients and found a shortfall that grows with the horizon. Correcting the forecast instead found that removing the coefficient’s bias makes the point forecast worse. And the shocks’ shape is the first defect that is one-sided: every other defect makes the interval too narrow or too wide on both sides at once, and this one moves its centre.

That has a practical consequence for how the defects combine. A forecaster who widens an interval to repair the estimation shortfall — multiplying its width by a factor fitted to the horizon — widens the empty lower half along with the overfull upper one, and at 95% under skewed shocks makes the interval wider than it needs to be without moving the misses off the upper side. The repairs are not interchangeable, and a forecaster has to know which defect is present before choosing one.

And what the 95% refers to is the frame for all of them: a coverage is a probability over repetitions, and a two-sided coverage is a probability that adds two tails. Saying an interval covers 95% when it misses nothing on one side and twice its share on the other is a true statement about the sum and a misleading one about the interval.

Ten thousand series at each law, level and horizon #

Ten thousand series at each shock law, level and horizon, each of sixty observations after a burn-in of two hundred, from a first-order autoregression at 0.7 with unit shock variance. The interval with the parameters known is the true conditional mean plus or minus the normal quantile times the true forecast standard deviation; as printed, the least-squares coefficient and residual variance replace them. Student’s t is drawn as a normal over the root of an independent chi-square on five degrees of freedom over five, rescaled by $\sqrt{3/5}$; the skewed shock is an exponential of mean one less one. The skewness of the forecast error is computed from the moving-average weights with nothing simulated.

Heavy tails cost the 99% interval, skew moves the misses to one side #

The claims are that under shocks from a t on five degrees of freedom the normal forecast interval covers its nominal 95% and falls short at 99%, by about eight tenths of a point symmetrically; that under centred exponential shocks the 95% interval with the parameters known covers 94.8% to 95.8% at every horizon from one to twelve with all its misses above; that the forecast error’s skewness falls from 2 to 1.11 and stays there; and that the 99% interval under skewed shocks covers between 97.3% and 98.3% with known parameters, all misses above.

Not claimed: anything about other persistences, where the effective number of terms in the error changes and the plateau of skewness with it; anything about shocks that are dependent or heteroskedastic, which break a different one of the four assumptions; and nothing about the residual bootstrap’s repair, which is named and not measured.

Still open: the bootstrap’s repair on sixty observations #

A residual bootstrap draws future shocks from the model’s own residuals, so its forecast intervals inherit their skewness and need no normal quantile. It is the natural repair for both defects at once. What limits it is the tail: a 99% interval is decided by the upper one per cent of the shocks, and sixty residuals contain perhaps one of them. Scoring the residual bootstrap’s 95% and 99% intervals under the three shock laws here, with the misses split by side, would say whether the one-sidedness is repaired by resampling or only reproduced by it — whether an interval built from sixty draws of a skewed law can find a tail it has barely seen.