A table of nested models

A set of forecasters is compared by putting their mean squared errors in a column and looking at the smallest. Eight of them on one series turns out to cost the multiplicity of about two and a half independent comparisons rather than of eight, and the reference distribution for the winner has to be over the whole set, because the set is what was searched.

Every candidate in that comparison is a rule: the last value carried forward, the mean of the last thirteen observations, the mean of the last thirty-four. Nothing in any of them is estimated. That is a real comparison and it is not the one anybody runs. What a working forecaster has on the desk is a table of fitted models — a benchmark, and a row for each variant of it that seemed worth trying — and the rows are related to the benchmark in a way the moving averages were not: every one of them contains it.

That single change alters both halves of the problem at once, and it alters them in opposite directions.

The table #

The benchmark is a first-order autoregression, fitted to a rolling window and used to forecast one step ahead. Variant j is that same model with one further lag added — lag 2, lag 3, and so on to lag 9 — so there are eight rows, each of which is the benchmark plus one coefficient.

The series is an AR(1). Every extra coefficient is therefore zero in population, every variant is a correct model of the same data, and in population every variant’s forecast is identical to the benchmark’s. Whatever a comparison of the table reports, it is reporting on estimation noise.

The benchmark itself is not free either. Its two coefficients cost it σ²(1 + 2/n) out of sample, which at 71 rows is 1.0282 against an innovation variance of 1; counted over the same two thousand tables it comes to 1.0256. Everything below is a difference between two quantities that are both above σ², and the difference is the whole of what a comparison sees.

Two details of the arrangement are part of the claim rather than under it. Every model is fitted to the same rows: the largest lag in the table decides where an estimation sample can start and all nine models start there, because an information criterion or a loss comparison across fits given different numbers of observations is a comparison of sample sizes wearing a comparison of models’ clothes. And every forecast is made at the same origin from the same window, so the errors are paired and a differential is a difference of two numbers computed from identical information.

Two and a half, against the other multiplicities #

The figure of two and a half is worth setting beside the other two overlapping searches this collection has measured, because the three are far apart.

Twenty analyses of one dataset are worth about 16.5 independent chances — 83% of their nominal count. Ten looks at a growing dataset are worth 4.1 — 41%. Eight forecasting rules on one series are worth 2.5 — 31%.

So a table of rules is the most redundant of the three by some way. The reason is immediate: twenty analyses differ in what they compute, ten looks differ in how much data they have, and eight rules are eight functions of the same history, differing only in how they weight it.

Every variant is behind before the search begins #

The mean differential of each row is negative. It is not slightly negative on some rows and slightly positive on others, which is what a fair coin between nine equally good models would look like: it is negative on all eight, at every window length, in every draw.

That is not a defect of the simulation. It is what a nested null does. A model with more coefficients estimates more numbers from the same rows, and every one of those estimates carries noise which the forecast then carries out of sample. The population forecasts are identical; the fitted forecasts are not, and the extra one is fitted worse.

The size of it can be written down. A regression with $q$ parameters fitted on $n$ rows and used to forecast a fresh observation has expected squared error

$$\sigma^2\left(1 + \frac{q}{n}\right)$$

to leading order, so a variant with one coefficient more than the benchmark loses

$$\mathbb{E}[\bar d] = \frac{\sigma^2 (q_0 - q_1)}{n} = -\frac{\sigma^2}{n}$$

per origin. Nothing in that expression is about the series. It does not know the persistence, it does not know which lag the variant added, and it does not know that the added regressor is a lag rather than the weather. It is a statement about the shape of the table — how many coefficients each row has, and how many rows of data each fit was given — and it is available before any data exists.

With a window of 80 observations and a largest lag of 9, each fit gets 71 rows, so the prediction is −0.01408 per origin. Counted over two thousand tables, the eight variants come in between −0.01447 and −0.01675, averaging −0.01572: the same number, thirteen per cent larger.

The thirteen per cent is worth a sentence, because it is what an honest agreement between a closed form and a count looks like when the closed form is a leading term. Multiply the measured displacement by the number of rows and the product should be exactly σ² if the expression were exact. It is 1.4081 at 31 rows, 1.1159 at 71, 1.0296 at 151 and 1.0286 at 311. The first term is doing the work and the remainder is a higher-order term going the way a dropped positive term goes. The site’s usual standard is that two routes to a number agree to several digits; here they agree to a stated order, and saying which is the difference between a check and a flattering coincidence.

What the displacement does to a table read the usual way #

Now read the table the way a table gets read. Compute a t statistic for each row — with the long-run variance in the denominator, because neighbouring origins overlap — take the largest, and report its p-value.

At a nominal 5%, over two thousand tables in which every variant is a variant of the truth, that procedure rejects 4.4% of the time.

Which looks like a table that needs no correction at all, and is not. One variant stated in advance, with no search anywhere in the procedure, rejects 3.5%. So the search over eight is costing something. It is being paid for by the displacement: the winner has been selected out of eight, which pushes the rate up, and every row is behind by σ²/n before the selection happens, which pushes it down. The two effects are of comparable size at these settings and their sum is close to the nominal level.

A rate that is right because two errors of opposite sign happen to be the same size is not a procedure. It is an arithmetic accident, and it can be shown to be one by removing either error on its own.

The repair that makes it worse #

The nested case has a known repair and this site has already measured it: Clark and West’s adjustment, which adds back the squared difference between the two forecasts and puts the mean of the differential where the null says it should be. Applied to a single nested comparison it takes the rejection rate from 67% to about its nominal level, which is one of the larger repairs in this subject.

Applied row by row to a table of eight, it takes the rejection rate at a true null from 4.4% to 22.7%.

Nothing has gone wrong with the repair. It has done exactly what it says: removed the displacement, leaving the multiplicity standing on its own with nothing to cancel it. What the 4.4% was hiding is that a table of eight nested variants read by its winner is a procedure with two large errors in it, and correcting one of them correctly makes the table worse. A reader who applies the recommended fix to each row of a specification search and reports the best result has moved from a procedure that was wrong for two reasons to one that is wrong for one, and has quadrupled its false positive rate doing it.

The two corrections that address the other error behave as they should. Bonferroni over the eight recentred rows rejects 2.8%, inside its level. One row stated in advance rejects 3.5%. Both of those are honest procedures; neither of them is what a table gets read as.

How many comparisons a table of eight is worth #

The multiplicity has a size, and it is the same measurement the accuracy search makes on its own set: compare the rate at which the winner of the table is called significant with the rate for one row stated in advance, and read off how many independent comparisons would produce that gap.

The answer here is 7.12 of a possible 8.

The identical measurement on eight moving averages of one series gives 2.44. Those eight are eight views of the same shocks — long windows contain short ones, and their errors correlate at 0.9 — so the largest of their eight statistics is barely the largest of two. Eight nested variants that each add a different lag are nearly eight separate chances, because what each one adds is nearly orthogonal to what the others add.

Two tables of eight rows, the same benchmark, the same nominal level, and the correct charge for having looked at all of them differs by a factor of three. A Bonferroni correction over eight is far inside its level in the first situation and nearly exactly right in the second. Nothing in either table says which situation it is, and the quantity that decides it — how much the candidates overlap — is not reported by any of the standard output.

It also means the reassuring half of the set field’s result does not transfer. There, a correction for the number of models was hopeless because the models were not distinct; here the models are distinct, so counting them is nearly the right thing to do — and the thing that goes wrong instead is the location the reference distribution is built around, which is the subject of the next essay.

The accident does not survive a longer comparison #

The 4.4% is one point of a curve, and the curve goes the opposite way from the obvious guess. Run the same table over thirty origins and the naive reading rejects 12.3%; over sixty, 4.6%; over a hundred and twenty, 0.9%.

The reason is worth having because it is the arithmetic of the two errors rather than a fact about this table. The multiplicity is a property of the set — eight chances to be lucky — and it does not change when the comparison gets longer. The displacement is a shift in the mean differential, and the statistic it produces is that mean divided by a standard error which shrinks as 1/√P. So the displacement’s contribution to every t statistic grows like √P while the multiplicity’s does not grow at all, and a table read naively becomes steadily more conservative the more origins it is given.

Two consequences follow, and the second is the one to carry.

A naively read table is not “roughly calibrated”. It is at 12% on a short comparison and at 1% on a long one, and the 4.4% in between is where the two errors happen to be the same size at this window length and this many origins.

And a procedure whose calibration comes from two errors cancelling cannot be tuned. Nothing about the design of the search moves both effects the same way, so there is no comparison length, window length or candidate count at which the cancellation is anything other than a coincidence that has to be checked each time. That is exactly what a reference distribution built by re-running the search does automatically, and what the two corrections that repair one error at a time cannot.

What the displacement is not #

It is worth being clear about what has and has not been shown, because “every variant is worse than the benchmark” is a sentence that can be read as something much stronger than it is.

It is not a claim that larger models forecast worse. The truth here is the benchmark; when the truth is not the benchmark, the variant that contains it forecasts better as soon as the coefficient it adds is worth more than the σ²/n it costs, and the essay after next measures exactly where that crossing is.

It is not a small-sample curiosity that goes away. It shrinks as 1/n, so it is always the same order as the thing being tested: the sampling standard deviation of the mean differential shrinks as 1/√P, and both go to zero, and their ratio does not go to zero in any way that makes the problem disappear.

And it is not an artefact of one-step forecasts, of AR models, or of squared error. Any comparison in which the larger model is fitted and the smaller one is nested inside it has the same term, with q₁ − q₀ in the numerator. The reason it can be written down here is that this table is a table of regressions with a fixed number of coefficients; the reason it matters here is that eight of them are being searched over at once.

What is claimed here, and what is not #

This field takes the comparison of a table of fitted models, and this essay takes the first half of it: what the fitting does to the comparison before anything has been searched.

What stays out and is named as a decision: model selection as a subject in its own right, which is a different question with its own field here — this table is not trying to identify the true model, it is trying to say whether the best of eight variants beats the benchmark, and those two questions have different answers on the same data. Selection after a search — what the coefficient of the winning variant is worth once it has been chosen — is the estimate-after-the-choice problem and is not re-derived here. And the whole apparatus of the denominator, the horizon and the dependence between origins is established where it was measured and used unchanged.

The boundary against the set field is the fitting. Everything there holds for candidates that estimate nothing; the moment a candidate has a coefficient in it, the displacement above is present and every one of that field’s reference distributions is built around the wrong point.

The checks, and the refusals that make them mean something #

Four claims are gated in this field’s library. The displacement is required to be negative for every variant at every window length, and its product with the number of rows each fit was given is required to lie between one innovation variance and one and a half — which is the closed form together with an explicit statement of what is left over, rather than a tolerance chosen to make them agree. The naive reading of the table is required to be near its nominal level and the recentred reading to be several times above it, which is the accident and its removal stated as a comparison the table cannot make about itself. And the effective number of comparisons is required to be above four and no more than the count, which is what says a correction for eight is nearly right here and hopeless one field along.

The refusal beside them is a reference distribution recentred at a location the null does not put it at. Handed a table of fitted variants, the resampling scheme that assumes only stationarity rejects far below its level and finds a genuinely better model far less often than a distribution that was told which model the benchmark was — and the check throws when it does, because a procedure that cannot reject at a false null is not conservative, it is broken in a direction nobody complains about.