Which forecast is better

The forecast field ends by ranking three forecasters. A fitted first-order model, the last value carried forward, and the sample mean are each run over the same series, their squared errors are averaged, and the smallest average wins. On fifty observations the fitted model wins over a band of persistence running from φ = 0.1186 to φ = 0.9231, and outside it one of the two things that estimate nothing wins instead.

Every number in that comparison is an average over a finite sample. Nothing in it says whether the winner would win again.

That is not a quibble about precision, it is a different question with its own machinery. Two forecasters have a difference in expected squared error, which is the quantity anybody cares about, and a difference in average squared error, which is what a comparison produces. Deciding whether the second is evidence about the first is a hypothesis test, and it turns out to be a test whose terms break the assumption that every test of a mean starts from.

The object is a difference, one per origin #

Write the comparison down as it is actually run. Two forecasters produce a forecast at each of P origins, each origin yields two errors, and the whole comparison is the sequence

dₜ = e₁ₜ² − e₂ₜ²

one number per origin. The comparison says forecaster 1 is worse when the average of the dₜ is positive, and the test asks whether that average is far enough from zero to be worth reporting. The statistic is the one every test of a mean uses — the mean divided by its own standard error — and the whole of the argument below is about the denominator.

The two forecasters drawn there are the two that estimate no dynamics — the last observation carried forward, and the mean of the window. They are chosen for a reason that matters more than it looks: their comparison has an exactly true null available.

A null that is true by construction rather than approximately #

Measuring how often a test rejects when it should not requires a situation where it should not. The usual way to get one is to set two things equal by fiat and hope the setup is otherwise fair. Here the equality can be solved for.

The last value carried forward has mean squared error 2γ₀(1 − φʰ) at horizon h, where γ₀ is the unconditional variance. The window mean has γ₀ multiplied by a longer expression, because that forecast carries its own sampling error and is correlated with the future it is forecasting — both terms are functions of φ and of the window length R, and dropping either puts the answer in the wrong place. Setting the two equal and solving for φʰ gives

φʰ = (1 − a) / (2(1 − b))

with a the variance of the window mean and b its correlation with the future, both of which depend on φ, so it is a fixed point rather than a formula: three iterations from φ = ½^(1/h), which is what the expression collapses to when the window is long enough for a and b to vanish. That limit is worth keeping. Two forecasts that estimate nothing are equally good when the series has forgotten half of itself over the horizon.

At a window of sixty the crossings are φ = 0.4922 at one step ahead, 0.6944 at two, 0.8256 at four and 0.9021 at eight. At those values the two forecasters have identical expected squared error to the last digit the arithmetic carries — the closed forms agree at 3.3634 at h = 4 — and a simulation that was never told the formula finds a mean difference of 0.0020 against a standard error of 0.0047. Every rejection rate below is measured at one of those four numbers, so a rejection is a false one by construction and not by approximation.

The differences are not independent, and the reason is arithmetic #

At one step ahead the loss differentials look like what a t-test assumes: their lag-one correlation is −0.003, which is nothing.

At four steps ahead they are not. Forecasts made at neighbouring origins four steps ahead are about overlapping futures: the forecast made at origin t and the one made at t + 1 both have to survive the shocks that arrive between t + 2 and t + 4, and those are the same shocks. The errors share h − 1 of them, so the differences do too.

The theory says the loss differential of h-step forecasts is a moving average of order h − 1 — that correlations run to lag h − 1 and vanish after it — and the measurement is 0.584, 0.303, 0.115 at lags one, two and three, and −0.011 at lag four. That is not approximately the prediction; it is the prediction, on a quantity nobody tuned.

What ignoring it costs, computed before it is counted #

A mean of P terms with correlation ρ₁ … ρₖ has variance (γ₀/P)(1 + 2Σρₖ) rather than γ₀/P. Nothing in that sentence is about forecasting. It is the same arithmetic the dependence field uses to say that fifty correlated observations are worth about six, applied to a different sequence.

Put the measured correlations into it. At h = 4, 1 + 2(0.584 + 0.303 + 0.115) = 3.004, so a standard error that leaves the correlations out is too small by a factor of 1.733, the statistic is too large by the same factor, and a test that rejects outside ±1.96 is really rejecting outside ±1.13. The share of a standard normal outside that is 25.4%.

Counted, it is 25.7%.

The agreement is the point. The over-rejection is not a mysterious small-sample effect to be guarded against with a rule of thumb; it is a computable consequence of a known correlation structure, and its size follows from three numbers a comparison already contains.

At eight steps ahead the naive test rejects 44.2% of true nulls at a nominal 5%. A reader handed that comparison would be told that one forecaster beat the other with a p-value below 0.05, and would be right to conclude nothing at all.

One construction detail carries more weight than it looks. Both forecasts at every origin are made from the same window, and the difference is taken within the origin rather than between two averages computed separately. Pairing that way removes everything the two forecasters have in common — the shock that arrives after the origin hits both of them, and cancels — so what is left in dₜ is only the difference in how they handled the information they shared.

Lining the two error sequences up differently, which is what happens when one forecaster’s file starts at a different date, puts that common noise back. Measured over four hundred comparisons: the standard error of the mean difference goes from 0.2437 to 0.3295, a factor of 1.35, and the rejection rate from 7.2% to 2.0% — without a single forecast changing.

The repair, and the part of it that does not go away #

The repair is to estimate the variance of the mean rather than assume it: add the autocovariances back, weighted, up to some bandwidth. That is the standard construction and it does most of the job.

The five rows are worth reading in order. The naive test rejects 26.4%. Truncating the long-run variance at h − 1 lags — the textbook prescription, and now visibly the right bandwidth rather than a convention — brings it to 10.7%. The weighted version gives 12.7% and the small-sample correction with a t reference 9.8%. And a test handed the true variance of the mean rejects 4.9%, which is 5% to within the standard error of the count.

So the correlations account for the whole of the null being true, and the estimator accounts for the rest of the failure. The residue is the denominator’s own bias. A long-run variance estimated from the same sixty numbers whose mean is being tested is biased low — it must be, since it is a sum of estimated autocovariances each shrunk towards zero by the estimation of the mean — and a denominator that is 10% too small on average turns a 5% test into a 10% one. That does not improve with more origins in the way the naive test’s failure does not improve at all: it shrinks slowly, and it is still there when the comparison has four hundred origins in it.

The p-values say the same thing one level down, which is the check that a rejection rate cannot make. The naive test’s p-values sit 0.278 from flat by the Kolmogorov–Smirnov distance; the corrected test’s, 0.094; and a test with the right denominator would be at the sampling noise of three thousand draws.

The inflation has a closed form in the horizon #

The over-rejection at h = 4 is predicted above from three measured autocorrelations, and the autocorrelations themselves can be predicted, which turns the whole horizon curve into one expression.

An h-step forecast error is a sum of h shocks and neighbouring origins share h − k of them at lag k. For Gaussian errors the covariance of two squared errors is twice the square of their covariance, so

ρₖ = ((h − k)/h)²

which at h = 4 gives 0.5625, 0.2500 and 0.0625 against the counted 0.584, 0.303 and 0.115 — the same shape, a little low, the shortfall being that a loss differential is not a single squared error. Summing it, Σ((h − k)/h)² = (h − 1)(2h − 1)/(6h), so the naive standard error is too small by

√( 1 + (h − 1)(2h − 1)/(3h) )

which is 1 at one step, 1.225 at two, 1.658 at four and 2.318 at eight, against the 1.733 the measured correlations give at h = 4. Pushing those through a normal predicts naive rejection rates of 5.0%, 11.0%, 24.0% and 39.8%, against counted values of 5.2%, — , 26.4% and 44.2%.

Two things follow that the single measured horizon cannot say. The expression grows like √(2h/3) for large h, so the naive test’s size approaches one as the horizon lengthens rather than settling at some tolerable level — at twenty steps the inflation is 3.7 and the size is about 60%, and at fifty it is 5.8 and 74%. And it says the whole curve without any comparison being run, so an evaluation can know before it starts how much of its own reported significance is overlap.

The expression is also the right bandwidth argument, stated as arithmetic rather than as a convention. The correlations it gives are exactly zero past lag h − 1 and positive before it, which is what a truncation at h − 1 lags is for; and they fall as the square of the shared fraction, which is why the first lag carries most of the correction — at h = 8 the first lag alone is 0.766 of the sum’s leading term and the last three together are under a tenth of it.

And the effective count is not P over h #

The essay reads sixty origins at horizon four as “about fifteen independent comparisons”, which is P/h and is the count of non-overlapping windows. The variance arithmetic gives a different and larger number: P/(1 + 2Σρₖ) = 60/3.004 = 20.

The gap is real and in the useful direction. Non-overlapping windows throw away every partially overlapping origin; the effective sample size keeps the fraction of each that is new. At h = 4 that is worth a third more information than the non-overlap count admits, and by the closed form above the ratio tends to 3/2 as the horizon grows — the effective count settles at 3P/(2h) rather than P/h.

So a forecaster choosing between overlapping and non-overlapping evaluation is not choosing between a biased standard error and an honest one. Overlapping origins with the right denominator carry half again as much as non-overlapping ones, and the only thing wrong with them is the denominator.

The repair is not free where there was nothing to repair #

The advice in circulation is unconditional: use a long-run variance when comparing forecasts. At one step ahead there is no overlap, the differences are uncorrelated, and estimating a long-run variance means estimating several autocovariances that are zero and adding their noise to the denominator.

Two points of size, bought for nothing. It is a small cost and it is in the same direction as the failure it was meant to fix, which is worth stating plainly: the standard repair is a repair at horizons past one and a mild defect at horizon one, and which of those applies is decided by the horizon rather than by preference.

What the test can find when there is something to find #

A rejection rate at the crossing is the test’s size. Everywhere else it is power, and the two are the same measurement taken at different truths.

At φ = 0.9 the last value carried forward is much the better forecast and the test finds it 93.0% of the time. At φ = 0.6, one tenth of the way from the crossing, it finds it 11.0% of the time. The curve is not symmetric about the crossing, and the asymmetry is a fact about the two forecasters rather than about the test: moving right makes the last value better at an accelerating rate, and moving left improves the mean slowly.

Sixty origins at horizon four contain about fifteen independent comparisons, and that is the quantity power is a function of. The effective number of comparisons, not the number of rows in the table, is what a forecast evaluation has to report — which is the dependence field’s effective sample size arriving from a third direction.

That has a practical consequence worth separating from the arithmetic. Lengthening the evaluation window buys power at the usual rate only while the added origins are contributing something new; past the point where the horizon is a large fraction of the window, each new origin brings h − 1 shocks already counted. Two evaluations reporting “two hundred out-of-sample forecasts” can carry two hundred and twenty-five pieces of information respectively, and nothing in the phrase distinguishes them.

What is claimed here, and what is not #

This field takes forecast comparison as a testing problem: the loss differential, the variance of its mean, the reference distribution, and the nested case the next essay is about. The forecast field named it as explicitly not claimed when it was written, and it is claimed now.

What stays out and is named as a decision rather than an oversight: comparisons of more than two forecasters at once, which is a multiplicity problem this site has a field for and which needs a reference distribution over the whole set of comparisons rather than over one; forecast combination as a subject, though the arithmetic that makes an equal-weight average beat both its parts is measured in the forecast field; loss functions other than squared error, where the same construction runs with a different dₜ and every number changes; and forecast encompassing, which asks a different question about the same pair.

The boundary against the exact-inference field is worth stating. That field builds a reference distribution by re-running the mechanism that assigned the data. Nothing here can: the mechanism that generated the series is the thing being estimated, so the reference distribution has to come from an asymptotic argument, and the whole content of this essay is that the argument has an assumption in it that the practice does not carry a warning about.

The checks, and the refusal that makes them mean something #

Four claims are gated in this field’s library. The crossing solved from the autocovariances is the crossing a simulation finds, and the closed forms agree to nine decimals while the simulation agrees to its own noise. The correlations of the loss differential are required to be large inside the horizon and small past it, at two horizons. The counted over-rejection is required to match the rate the correlations predict, to within three points, with no simulation in the prediction. And the oracle test — the one handed the true variance — is required to be exact, because if it were not, every other number in the essay would be measuring the setup rather than the test.

The refusal is a comparison whose two error sequences are not paired by origin. It is the mistake a rolling comparison makes easy — two files that start at different dates line up without complaint — and the standard it fails is that a paired comparison’s standard error is a property of the two forecasters rather than of how their files were aligned. It inflates it by 35%, and the check requires that failure, because a check that has never rejected anything proves nothing.