The interval that forgets it estimated

The previous essay derived the forecast interval and checked it. The derivation assumed φ and σ were known. The check ran the simulation with φ and σ known. Both were honest and neither was about anything anybody can do, because a forecaster has a series and not a model.

What happens next is the substitution that every implementation of this makes and that no output mentions. Fit the model, get φ̂ and σ̂, put them where φ and σ were, and quote the interval. The formula is correct. The numbers going into it are not the numbers it was derived for, and the question this essay asks is what that costs, in the only currency the site accepts: how often the interval contains what it said it would.

What it costs, counted #

Twelve hundred series at each horizon, one set of seeds, both intervals computed on the same data. The known-parameter interval sits on 95.0% at six steps and its counterpart on 87.3%.

The shortfall has two properties worth separating, because they have different causes and different remedies.

It grows with the horizon. At one step the plug-in interval covers 92.8% and at six it covers 87.3%. That is not a small drift: the miss rate has gone from one in fourteen to one in eight, on the same data, from the same fit, with nothing changing except how far ahead the statement reaches.

It shrinks with the length of the series. At a hundred observations the one-step interval covers 94.3% and at four hundred 94.4%, both within a standard error of the target. This is estimation error and behaves like estimation error: it is a fact about how much data went into φ̂, and it goes away in the direction more data goes.

Is that a gap or a wobble #

Two numbers a few percentage points apart deserve the question this site asks of every counted rate before reading anything into it. Twelve hundred trials at a nominal 5% miss rate gives a standard error of 0.63 percentage points on each line. The gap at six steps is 7.7 points, which is a dozen standard errors, and the gap at one step is 1.9 points, which is three.

The comparison is also within seeds: the same twelve hundred series produce both lines, so the two are not two independent estimates being differenced. Every series contributes a known-parameter verdict and a plug-in verdict, and the pairs disagree on the series where the fit happened to be poor. That is why the upper line can be read as a control rather than as a second measurement — if the machinery had a bug in the forecast recursion, or in the ψ weights, or in the way the future observations are held out, both lines would move and the gap would not be attributable.

The upper line does one more job, and it is the reason it is drawn at all rather than assumed to be 95%: it is a live check on the arithmetic of the whole essay. If it drifted off the target the conclusion below would be unavailable, because the shortfall could be a defect in the simulation rather than in the substitution. It sits within a standard error of 95% at every horizon and every sample size in this essay, which is what makes the lower line’s departure mean something.

How much wider it would have to be #

Coverage is the honest measure and it is not the useful one, because a forecaster cannot act on “87%”. What can be acted on is the width, and the width the interval should have is available without any theory at all: simulate, collect the forecast errors, and read off the interval that contains 95% of them.

That number is not a derivation and does not pretend to be. It is an empirical quantile of the error distribution, and comparing it with the mean plug-in width says how much the interval was short by. At twenty-five observations the honest interval is 25.7% wider at six steps and 6.9% wider at one.

A quarter more width is not a rounding error. It is the difference between a band a reader treats as tight and a band a reader treats as barely informative, and the whole of it is missing because a formula derived for two known numbers was handed two estimates.

There is a general habit here that this site has met before under a different name. The plug-in interval for a hierarchical model estimates the population spread and then uses it as though it were known, and covers 78.8% against a claimed 95%. That is the same mistake in a different field, and the two are worth putting side by side because the shape of the repair is the same: carry the uncertainty in the estimated quantity instead of dropping it. There the repair was an integration over the posterior for τ. Here the comparable repair is a bootstrap over the fitted residuals, which this field measures rather than implements — the point of this essay is the size of the hole, and the honest interval above is the measurement of it.

The other defect, which is in the forecast rather than the interval #

The interval is centred on a point forecast, and the point forecast has its own problem. It is a separate defect in the same arithmetic and it is easier to state than to notice.

Least squares estimates the persistence of a series as smaller than it is. The bias is about (1 + 3φ)/n and it is downward at every φ. On forty observations from a series with φ = 0.8, φ̂ averages 0.7074.

By itself that is a bias of a tenth, which is unremarkable. What makes it worth an essay is that the h-step forecast does not use φ̂ — it uses φ̂ raised to the power h, and a proportional error raised to a power is a larger proportional error. The decay factor the forecast applies is

• 11.6% short of the truth at one step,
• 24.8% short at three,
• 30.9% short at five,
• and 32.8% short at eight.

So the forecast returns to the mean a third faster than the series does, systematically, in every run. This is not visible in any single forecast — a third of a decay factor is well inside the noise of one series — and it is not visible in the coverage either, because the interval is wide enough to absorb it. It is visible only across many series, which is exactly the arrangement the twenty fans show.

The practical reading is uncomfortable and worth stating without softening. A forecaster using a short series will, on average, under-predict how long a departure from the mean persists. Not sometimes — on average, in the direction the estimator is biased, at every horizon, and by more the further ahead the question reaches.

The two defects pull in the same direction, which is why neither shows #

It is worth putting the two together, because they are not independent and their interaction is what makes the whole thing survivable in practice.

The interval is too narrow: it should be a quarter wider at six steps on a short series. The point forecast is too close to the mean: its decay factor is a third short at five steps. A forecaster therefore publishes a band that is centred too near the average and is too tight around that centre — two errors, both of which make the published statement look more informative than it is.

They do not cancel. A narrow band around a wrong centre misses more often than a narrow band around the right one, and the counted coverage above already contains both, because the same fitted φ̂ that produced the shrunken forecast produced the band. Separating them is possible — put the true parameters in the band and keep the fitted forecast, or the reverse — and the arrangement matters enough that the field’s library computes the fully-known interval rather than the half-known one. The first version of that function used the true variance around the fitted forecast, which is neither of the two things anybody has, and reported the known-parameter line at 93.2% instead of 95.0%. Two per cent of the gap belonged to a hybrid nobody would ever use.

The lesson is one this site keeps meeting from different directions: a control has to be a procedure somebody could run. The known-parameter interval is a control because a forecaster who happened to be told φ and σ could use it. A band computed from the true variance around a fitted forecast is not a procedure at all, and using it as a reference makes the defect look smaller than it is.

The shortfall is width and not centre #

Two defects are measured — a band that is too narrow and a centre that is too close to the mean — and the coverage figures say how the seven and a half points are split between them.

An interval that should be 6.9% wider covers 2Φ(1.96/1.069) − 1 = 93.3%, and the counted one-step figure is 92.8%. One that should be 25.7% wider covers 88.1%, and the counted six-step figure is 87.3%.

So the width accounts for the coverage shortfall to within about half a point at both horizons, and the biased centre contributes under a point of the seven and a half. That is worth knowing before the two defects are read as equally urgent: the interval misses because it is narrow, and the forecast inside it is wrong for a reason that barely shows up in whether the interval contains the future.

Which is the direction that makes the bias hardest to find, and it is a stronger statement than the essay’s own. The bias is not merely absent from the accuracy measure a forecaster computes; it is nearly absent from the coverage measure as well.

The convexity that softens it is the one that ruins the repair #

The decay shortfalls — 11.6%, 24.8%, 30.9% and 32.8% at one, three, five and eight steps — are smaller than the bias in φ̂ alone predicts, and the gap is worth reading.

φ̂ averages 0.7074 against a truth of 0.8, a ratio of 0.884. Raising that ratio to the horizon gives shortfalls of 11.6%, 30.9%, 46.0% and 70.9%. The counted figures match at one step exactly and fall progressively below the prediction after it: the shortfall is 20%, 33% and 54% less than raising the mean would give.

That gap is Jensen’s inequality. E[φ̂ʰ] exceeds (E[φ̂])ʰ because a power is convex, so the spread of φ̂ across samples pushes the average decay factor back up — and it pushes harder the higher the power. It is exactly the mechanism that makes a corrected estimate overshoot when it is raised to a horizon, arriving here to partially rescue an uncorrected one.

So the two halves of that arithmetic are the same fact used twice. The convexity that makes a bias-corrected forecast overshoot is what stops the uncorrected forecast’s shortfall from compounding, and it is why the counted sequence flattens at about a third rather than heading for the 71% the bias alone would imply.

Why the direction is the one that flatters #

There is a reason this defect has survived so comfortably, and it is the direction.

A forecast that reverts too quickly is a forecast that is closer to the mean than it should be. A forecast closer to the mean is a less extreme forecast, and less extreme forecasts are, by construction, harder to be badly wrong about. Measured by squared error against the actual future, the shrunken forecast is barely worse than the correct one and is sometimes better — the same arithmetic that makes shrinkage a good idea when it is done on purpose makes it hard to detect when it is done by accident.

So the accuracy measure a forecaster is most likely to compute does not reveal the bias. Nothing reveals it except comparing φ̂ with φ, which requires knowing φ, which is the situation nobody is in. That is the general shape of every finding on this site that has survived a long time: the symptom is absent from the statistic anybody looks at.

Where in the range it is worst #

Everything so far holds the persistence at φ = 0.7. The shortfall is not the same at every φ, and the shape of its dependence is the last thing worth measuring, because it says which series a forecaster should be most careful with.

Two effects run against each other. A more persistent series has a more precisely estimated φ in absolute terms — the regressor has more variance, so the standard error of φ̂ falls — which argues that the interval should be better at high φ. But a more persistent series has a forecast that depends on φ̂ over more horizons before it decays, and the h-step variance is a longer sum of terms each of which involves the estimate, which argues the other way.

The second effect wins, and it wins for a reason that is arithmetic rather than statistical: the error in φ̂ enters the h-step forecast through φ̂ʰ and the h-step variance through a sum of φ̂ to even powers, so the number of places the estimate can be wrong grows with the horizon, and the horizon over which the forecast is doing anything at all grows with φ. A persistent series is the case where a model has the most to say and the case where what it says is least reliable, which is the combination least likely to prompt anybody to check.

The refusal, and it is the mistake a printout invites #

A fitted model reports one number for the spread of its errors: the residual standard deviation. It is the only spread on the output, it is correct, and it is correct at one step.

Using it at every horizon is the natural thing to do and it is what an interval computed by hand from a regression printout will be. At six steps on a persistent series that interval covers 75.9% where it claims 95%, against 92.5% for the same forecast given the horizon’s own variance. The refusal in this field’s library requires the first to fail, because the check that the horizon matters is worth nothing unless it has been shown an interval that ignores the horizon and rejected it.

The two numbers are also the cleanest available statement of what the ψ weights are for. They are not a technicality of the derivation; they are the difference between an interval that misses one time in four and one that misses one time in thirteen.

What is left, and it is named rather than measured #

Three things this essay establishes and does not repair.

The interval is too narrow and the honest width is known. The repair is a bootstrap over the fitted residuals, or an analytic correction with an extra term of order 1/n. Neither is built here; what is built is the measurement that says how much either would have to supply.

The point forecast is biased and the bias is estimable. There are bias-corrected estimators of φ and they are not free — correcting a bias generally costs variance, and whether the corrected forecast is better by squared error is a question this field does not answer.

And both defects are conditional on the order being right. Every number above was computed with the order given: a first-order model fitted to a first-order series. No forecaster is handed the order. Choosing it is the next essay, and choosing it from the same data the interval is then computed on turns out to cost more than the estimation measured here.

That last item is the one worth carrying, because it changes how the numbers above should be read. Everything in this essay is a lower bound on what a real forecaster’s interval is short by. The model was correctly specified, its order was known, the series was genuinely stationary, and the shocks were genuinely independent and identically distributed — four assumptions that were true by construction because the series was simulated from them. A forecaster has none of those guarantees and has, in addition, the one this essay held fixed. The 87.3% is what the interval covers when everything except the parameter values is right.

Finally, a boundary. This essay measures the shortfall and does not price it, because pricing it requires saying what a miss costs, and that is a decision problem rather than a statistical one. The site’s rule holds here as everywhere: nothing is called 95% until it has been counted, and once it has been counted what to do about it is somebody else’s question.