A tail sixty residuals have barely seen

Shocks that are not normal found the printed forecast interval doing two different things wrong. Under heavy tails its 95% is fine and its 99% is short, by about eight tenths of a point. Under skewed shocks its coverage looks fine at 95% and every miss is on one side: 0.0% below and 5.2% above at one step, with the parameters known. It closed by naming the obvious repair. A residual bootstrap draws future shocks from the model’s own residuals, so its intervals inherit their skewness and need no normal quantile. It is a repair for both defects at once. What limits it is the tail, since a 99% interval is decided by the upper one per cent of the shocks and sixty residuals contain perhaps one of them.

So the question has three parts. Does the one-sidedness go away? What happens to the coverage? And can an interval built from sixty draws find a tail it has barely seen?

The first answer is yes. The second is a price nobody asked about. The third is a hard number, and it can be stated without running anything.

The side comes back #

The series is a first-order autoregression at 0.7, as in the earlier essay, fitted by least squares with a constant to sixty observations. The interval as printed is the point forecast plus or minus a normal quantile times the plug-in forecast standard deviation. The bootstrap keeps the fitted coefficient and constant, draws each future shock from the sixty centred residuals, runs the forecast forward and reads the percentile interval of the values it reaches. A second form, usually recommended, also redraws the whole series from the fitted law and re-estimates the coefficient on each path, so that the interval carries the coefficient’s own error.

The figure at the head of this essay splits the misses of a nominal 95% interval, one step ahead, by side.

Under skewed shocks, an exponential less its mean, the printed interval misses 0.0% below and 6.5% above. The bootstrap misses 4.3% below and 4.1% above. As a repair of the thing it was asked to repair, that is complete. The bootstrap has found that the shocks cannot fall very far and can rise a long way, and it has moved the interval over and stretched it upward to match, on sixty observations.

The re-estimated form sits between them, at 1.8% and 4.3%, and that is a curious thing: re-estimating the coefficient, a change that has nothing to do with the shape of the shocks, moves the lower misses. Its interval is wider and its lower end is further from the distribution’s hard edge, and fewer than half as many observations fall below it. It is the closest of the three to what the skew justifies, since a shock has an exact floor of −1 and an interval that sits below that floor has nothing to miss.

But look at the other two rows of the figure, where the shocks are normal and heavy-tailed and nothing needed repair. The printed interval misses 3.1% a side under normal shocks and has no business being beaten. The bootstrap misses 3.8% and 4.2%. And that is the second part of the answer.

It pays for the side with coverage #

At 95%:

At 99%:

So the repair for a problem of shape costs about two points of a problem of width. The cause can be seen in the widths: at 95% and one step the printed interval is 3.92 wide under normal shocks and the bootstrap 3.81, and at 99% it is 5.15 against 4.54. Three things pull the bootstrap’s interval in, and the printed interval has correction for all three. Residuals are smaller than shocks: they are shocks with a fit taken out, and their sum of squares is short by about two degrees of freedom in sixty. The printed variance divides by $n - 2$, and centred residuals drawn with equal probability do not. An interval whose ends are read from a finite sample of residuals is also narrower than the distribution’s tail quantile on average, because the sample’s extreme points are inside the tail. And a normal quantile extrapolates past the sample where a percentile of the sample cannot.

Re-estimating the coefficient recovers a little of this, and the table is surprisingly flat otherwise. At 95% under normal shocks it moves coverage from 92.0% to 92.3%, on 1,000 series with a standard error of 0.8, which is to say it does not measurably move. Under skew it moves 91.6% to 93.9%, a real change that comes with the lower misses shown earlier. A reader working from the interval that forgets it estimated would expect the coefficient’s error to matter most at long horizons, and it does there as well, but at twelve steps the difference between the two bootstrap forms is at most two points in either direction.

The horizon does not rescue anything. At twelve steps under skewed shocks the printed interval covers 92.0% with misses of 0.7% and 7.3%; the held bootstrap covers 88.7% with 6.1% and 5.3%; re-estimation 90.9% with 4.7% and 4.4%. The sidedness is gone at every horizon, and so are about two points of coverage.

A cap made of a count #

Now the third part. The question was whether an interval built from sixty draws can find a tail it has barely seen, and at one step that has an exact answer that does not depend on any of this.

A one-step bootstrap value is the point forecast plus a residual. Its percentile interval therefore has ends that are, at best, the largest and smallest residuals. A fresh shock is exchangeable with the sample’s shocks, so it exceeds all $n$ of them with probability $1/(n + 1)$ and falls below all of them with the same probability. So no one-step interval whose ends are residuals can cover more than

$$1 - \frac{2}{n + 1}.$$

At $n = 59$ residuals that is 96.7%. It does not matter whether the nominal level says 99% or 99.99%, and it does not matter how many bootstrap replications are drawn. It is a statement about ranks, it holds for any continuous shock law, and it is not sampling error: taking B to infinity removes the error in reading the quantile and leaves the cap exactly where it was. The argument is the same one that gives conformal prediction its guarantee in the opposite direction, where a rank in a calibration set says how much coverage is assured rather than how much is possible.

The argument is approximate only because residuals are not quite shocks, and in practice they are smaller, which pushes the coverage further under the cap and not over it.

The measured coverage follows it closely at small samples. At 20 observations the cap is 90.0% and the three laws reach 87.4%, 88.6% and 87.2%. At 30 it is 93.3% against 92.0%, 93.0% and 91.8%. At 60 it is 96.7%, against 96.2%, 96.6% and 95.6%. The $t$ on five does best because a heavy tail makes the largest residuals larger, and they are in the right place more often. Up to about sixty observations a nominal 99% interval from the residuals simply is an interval from the two extreme residuals, with a coverage set by its sample size, and calling it a 99% interval conveys very little.

Past that the cap stops being what binds. At 240 observations it is 99.2% and the three laws sit at 98.2%, 98.5% and 98.1%, and at 480 it is 99.6%, with coverage at 98.4%, 98.1% and 97.8%, which at 4,000 series has a standard error of about 0.2 points. These stay more than half a point below nominal at the largest sample here, and they do not appear to be closing fast. So there are two separate things limiting this interval, and the second one, the difference between residuals and shocks and the noise in a tail quantile, has not gone away.

What the sample size buys #

On 240 observations, things change in the expected direction. Under skewed shocks at 99% the printed interval is 97.6%, with all of its misses above, and the bootstrap has moved to 98.3%, with 0.9% and 0.8%, better than the printed interval in both coverage and balance, and narrower too. Under the $t$ on five at 99%, it is 97.6% against 98.1%. Under normal shocks, where the printed interval has every advantage, it is still 98.8% against 98.0%.

So there is a crossover, and at this one comparison it sits somewhere between 60 and 240 observations for the 99% interval. What a forecaster gets at sixty is a different thing from the textbook’s result: the normal quantile is mistaken about the shape in a way resampling can correct, and resampling is mistaken about the size in a way a sample of sixty cannot correct. It is not obvious that the first error is worse.

What sixty residuals can honestly report #

Between the two, there is a choice about what kind of wrong to prefer. Under skewed shocks and at 95%, a user of the printed interval has an interval 3.87 wide, covers 93.5%, and is wrong in only one direction. A user of the bootstrap has an interval 3.62 wide, covers 91.6%, and is wrong in both. Which one matters depends on what a miss above costs. A forecaster of a quantity with an upper risk, such as demand or a claim count, may well be better served by knowing that misses are all on one side and widening that side alone, which is a third option nobody on this page has drawn.

At 99% the honest report on sixty observations is shorter still. No rule that reads a percentile from sixty residuals covers 99%, and the printed interval only approaches it where the shocks are actually normal. A reader who needs a 99% claim from a sample this size needs a parametric assumption about the tail, and needs to say that is where the number comes from. The essay on a threshold’s dial is the honest route for that: a fitted tail model is an extrapolation, stated as one, where a percentile of residuals has no more room to move than its largest point.

And it is worth saying plainly that the cap says nothing about the printed interval, which is not built from ranks. That interval reaches 98.4% under normal shocks at sixty observations because its normal quantile extrapolates: it is assuming something, and when the assumption is right it is rewarded for that. Under skewed shocks the same extrapolation gives 96.5% and all of the misses on one side. The bootstrap is free of that assumption and is held to the count. What resampling removes is the assumption, and the assumption was also the only source of information about the tail. Where the bootstrap lies makes the general form of the same point for a sample’s extreme statistics.

The edge a shock law has #

The skewed law here has a feature the other two do not. A centred exponential cannot fall below −1, and the 95% lower end the printed interval reads, about −1.96 times the forecast standard deviation, sits nearly a whole spread below that edge. Its 0.0% below is therefore not a coverage statistic at all. An observation cannot get there. The side of the interval that seems to be doing perfectly is doing nothing, and all of the interval’s width on that side is wasted.

That raises the question of why a bootstrap with residuals that know about the edge misses 4.3% below when the promise is 2.5%. The rank arithmetic puts a floor under it: a fresh shock falls below every one of 59 residuals with probability 1.7%, whatever the level. On a one-sided edge that floor is especially exposed, because the exponential’s lower tail is short and its density highest exactly at the edge, so a fresh shock lands near the smallest residual often, and the residuals’ smallest member is pulled above the true edge by the fit’s error in the constant. The point forecast’s own error adds the rest, since it moves the interval while the shock law stays put, and a short tail has no room to absorb the move. Under the longer upper tail the same shift hardly matters.

So 4.3% below under skew and 3.8% below under normal shocks are not the same kind of number, though they look it. The second is a slightly narrow interval, and the first is an interval whose lower end sits close to a hard edge with a good deal of probability right next to it. At 99% the same thing appears as 2.4% below for the held form against 0.4% for the re-estimated one, on the 1,000 series the second was run on, and at that level it is the largest difference between the two forms anywhere in the table. Re-estimating helps exactly where the interval sits at an edge, since it widens the interval by carrying the point forecast’s error, and that error is what the short tail could not absorb.

How the numbers were made #

Three thousand series at each combination of sample size, shock law and horizon, with 299 bootstrap draws for the interval that holds the coefficient, and 1,000 of those series with 199 re-estimated paths each for the second form; a 200-row burn-in removes the start. The rank-cap sweep uses a further 4,000 series and 399 draws at each of six sizes. Standard errors are about 0.45 points at 95% on 3,000 series, 0.2 at 99%, and about 0.8 at 95% on the 1,000 re-estimated series, so a difference of less than about a point between the two bootstrap forms should not be read at all.

The residuals are centred. They are not rescaled by $\sqrt{n/(n-2)}$ or by leverage, as they can be, which is a choice worth stating, since rescaling would widen the interval a little and could put an extreme residual beyond its own rank, at the cost of a further assumption. The cap as written applies to the plain form. Three checks hold the comparison in place. The printed interval’s misses must be one-sided under skew, and the bootstrap’s on both sides. The 99% interval must sit under its cap, within one point of tolerance, at every sample size and law. And the claim that sixty residuals find the 99% tail, set at 98%, is refused.

Still open: a horizon’s sum, and a tail with a shape #

Why the horizon does not lift the cap. At four or twelve steps the bootstrap’s value is a sum of several residuals and its range is far larger than one residual’s, so the cap as stated no longer applies; the sum of $h$ draws can land past anything seen. The measured coverage under a nominal 99% does rise less than that suggests: 95.4% and 94.1% for the held form under skew, at four and twelve steps. The question of what bounds it there is a question about how much of a sum’s tail lies in the largest term, and at a persistence of 0.7 the first few weights carry the sum, so the largest-shock term keeps much of its weight. An exact statement would be a formula for this cap as a function of the weights.

A tail model on the residuals. A generalised Pareto fitted to the largest few residuals, and sampled past them, would give an interval that can pass its largest residual, which the plain resampling cannot. Whether that reaches the 99% on sixty observations, and whether a tail fitted to six points is worth more than a normal quantile for the same purpose, a question the two intervals for one return level raises in its own setting, is the comparison this essay’s conclusion is asking for and has not made.

What the residuals do and do not supply #

This essay takes the residual bootstrap’s forecast interval for a first-order autoregression at 0.7. Its claims are that resampling repairs the one-sidedness under skewed shocks, from 0.0% and 6.5% to 4.3% and 4.1% at one step and a nominal 95%; that it costs about two points of coverage on sixty observations, under normal shocks as well; that a one-step interval read from $n$ residuals cannot cover more than $1 - 2/(n + 1)$, which is 96.7% at sixty observations whatever the nominal level; that measured 99% coverage sits just under that cap up to about sixty observations and remains more than half a point below nominal at 480; and that on 240 observations the bootstrap overtakes the printed interval at 99% under both non-normal laws.

What stays out, as a decision: other persistences; rescaled or leverage-adjusted residuals; forecast intervals for orders chosen from the data, which the order table’s essay took on separately; and conditional heteroskedasticity, a separate assumption. The boundary against the earlier essay on shocks is that it named the defects in a printed interval and this one prices the repair.