The displacement is a parameter count

A regression with q coefficients, fitted on R rows and used to forecast a row it has not seen, has an expected squared error larger than σ² by a factor that depends only on q and R. That is the whole of this essay’s arithmetic, and everything else is what follows when it is applied to a table.

The derivation is three lines. Out of sample the error is the noise in the new observation plus the error in the fitted coefficients evaluated at the new row’s predictors. The two are independent, the first contributes σ², and the second contributes σ² times the expected leverage of a fresh row — which for Gaussian predictors is q/(R − q − 1) exactly. So

$$\mathbb{E}\big[(y^* - x^{*\prime}\hat\beta)^2\big] = \sigma^2\left(1 + \frac{q}{R - q - 1}\right),$$

whenever the model contains the truth.

Nothing in that expression is about any other model. It is a statement about one regression, and so the expected difference between two correct candidates is

$$\mathbb{E}[\bar d_{ij}] = \sigma^2\left(\frac{q_i}{R - q_i - 1} - \frac{q_j}{R - q_j - 1}\right),$$

whether or not either contains the other.

Nesting was the scaffolding, not the structure #

The field that first measured this displacement measured it on a ladder, where the benchmark is contained in every variant. In that setting nesting does two things at once, and it is easy to attribute the second to the first.

It guarantees the dimensions differ — a variant with one extra regressor has one extra coefficient. And it guarantees the population accuracies are equal under the null, because the extra coefficient is zero and the two models are then the same model.

The second is what a nested null means. The first is what the displacement is made of. Take a table of subsets instead, where two candidates of the same size share neither model, and the two come apart:

• Two candidates of the same size are not displaced against each other at all. The four one-predictor subsets sit at −0.00101, 0.00084 and 0.00088 against the benchmark, against standard errors of about 0.0019 — zero, and nested to nothing.
• Two candidates of different sizes are displaced by their difference in size, whichever one contains the other and whether either does. The six two-predictor subsets sit between −0.01130 and −0.01393, against a closed form of −0.01350, and exactly one of those six contains the benchmark.

That is the sentence this essay exists for. The displacement is a parameter count. Containment was the device that made the counts differ reliably, and once the counts are visible directly it has nothing left to do.

The exact form and the leading term, which are two different numbers here #

The earlier field quotes the displacement as σ²(q₀ − q)/R and shows its remainder shrinking as the window lengthens — a leading term with its error displayed rather than a tolerance chosen to make two routes agree. Over a table whose dimensions run from two to five, the leading term and the exact form are far enough apart to tell apart, and it is worth seeing how far.

The leading term is 7% short at one extra coefficient and 10% short at three, and the exact form sits inside the counting error everywhere. Fitted as a straight line through the counted points, the cost of one extra coefficient comes out at −0.01329 where the leading term claims −0.01250 — a slope that is 6% steeper than 1/R, because q/(R − q − 1) is convex in q and the table’s dimensions are spread out.

The window is the dial. One extra coefficient costs −0.01848 of mean squared error per origin at a window of sixty, −0.01350 at eighty, −0.00877 at a hundred and twenty and −0.00649 at a hundred and sixty; the leading term says −0.01667, −0.01250, −0.00833 and −0.00625. The two converge, as they must, and the place they disagree is exactly the place a specification search is usually run — a short window, because somebody wanted many origins out of a series they have.

What it does to a search that has not been corrected for anything #

The displacement is not a bias in an estimate. It is a real difference in expected out-of-sample accuracy, and it means that under a true null the candidates in a mixed table are already ranked.

Search the fifteen subsets by out-of-sample mean squared error, at a null where none of the four predictors is worth anything, and the winner is a one-predictor model 77.3% of the time. A coin over fifteen candidates would say 6.7%. The average winning model has 2.27 coefficients where the table’s average is 3.27.

Read one way that is model selection working: the smallest correct model is the best forecaster, and the search finds it. Read the other way it is a false-positive machine. An open reading of a nested ladder — the largest studentised difference over every pair — rejects 46.4% of true nulls, and the difference it is finding is real. It is simply not the difference anybody wanted to hear about.

This is why a multiplicity correction cannot repair the nested case. Bonferroni divides a level among comparisons; it does nothing to a shift in the mean of the statistic. Applied to the ladder’s twelve ordered pairs it takes 46.4% to 6.8%, which is still above the nominal level, while on a table of six equal-size candidates — where there is no shift, only a maximum — it takes 27.4% to 0.8% and over-corrects by a factor of six.

The remainder has a closed form too #

The leading term is reported as 7% short at one extra coefficient and 10% at three, and those two percentages are not two measurements — they are one expression evaluated twice.

Expanding q/(R − q − 1) as (q/R)(1 + (q + 1)/R + …) and differencing gives

displacement ≈ (q₀ − q)/R + [q₀(q₀ + 1) − q(q + 1)]/R²

so the exact form exceeds its own leading term by [q(q + 1) − q₀(q₀ + 1)]/R², and the relative shortfall of the leading term is

(q + q₀ + 1) / R

for one extra coefficient. At the benchmark’s two coefficients and a window of eighty that is 6/80 = 7.5%, and across three extra coefficients it is [5·6 − 2·3]/[80·3] = 10.0% — the two percentages the table reports, from an expression with no data in it.

The window sweep is the same expression read the other way. The exact and leading values differ by 10.9%, 8.0%, 5.3% and 3.8% at windows of 60, 80, 120 and 160, and multiplying each by its window gives 6.54, 6.40, 6.36 and 6.08 — a constant of about six, which is q + q₀ + 1 at the dimensions being compared.

So the remainder is not a residual to be watched shrinking; it is 6/R, and it is knowable before any comparison is run. That matters for the essay’s own point about where searches are run: at a window of sixty the leading term is eleven per cent light, and eleven per cent of a displacement that already decides three quarters of the search’s outcome is not a rounding.

It also says when the leading term is safe, in the form a practitioner can apply. The relative error is the sum of the two dimensions plus one, over the window — so a table whose candidates run from two to eight coefficients, compared over eighty rows, has a leading term that is 14% light at its widest pair and 7.5% at its narrowest. The approximation degrades exactly as the table gets wider, which is the direction that makes a search harder rather than easier, and the exact form costs one division to avoid it.

Two routes, and why neither was tuned to the other #

The agreement in the table above is worth one paragraph of scepticism, because a closed form matching a simulation is the easiest result in statistics to manufacture by accident.

The two routes here share nothing. The left column is a rolling out-of-sample comparison: a window of eighty rows, a fresh row forecast, the squared error recorded, the window advanced, sixty times, over three hundred draws of a fresh world. It is never told how many coefficients any candidate has. The right column is q/(R − q − 1) evaluated at two integers. It is never told what the predictors are, how they are distributed, how long the series is, or which candidates contain which.

There is no free constant between them. If the exponent on R were wrong the columns would separate as the window changes, and the window sweep in the previous section is that check run four times. If the dependence on q were wrong the columns would separate as the table’s dimensions spread, and the fifteen rows are that check run fifteen times. The tolerance is the counting error of the simulation, which is reported beside every cell rather than chosen.

Where this meets the information criteria #

A reader who has met an information criterion will recognise the shape of q/(R − q − 1): a penalty proportional to the number of parameters, with a coefficient that depends on the sample size. That is not a coincidence and it is worth saying exactly what the relation is, because it decides whether selecting by a criterion changes anything.

An information criterion penalises the in-sample fit by a multiple of q, in order to estimate what the out-of-sample error would be. The displacement measured here is that out-of-sample error, counted rather than estimated. So a criterion is an attempt to predict the number on the vertical axis of this essay’s first figure, and Akaike’s is the attempt calibrated to get it right on average.

Two things follow. Selecting by a criterion instead of by out-of-sample error does not remove the displacement, because the criterion is built to reproduce it — it removes the noise in estimating it, which is a different and smaller problem. And a criterion whose penalty is heavier than Akaike’s does not remove it either; it over-corrects, and the search then prefers models that are too small for the same reason this one prefers models that are small.

The one thing that does remove it is comparing candidates of the same size, which is available exactly when somebody is willing to say in advance how large the model is going to be. That is a strong commitment and it is the only clean way out of this essay.

What this costs an experiment that has already been run #

The displacement is small in absolute terms — a hundredth of the benchmark’s mean squared error per extra coefficient, at this window — and it would be easy to conclude that it is negligible. Two things say otherwise, and both are about what it is being compared with.

It is a shift in the mean of a statistic whose spread falls like 1/√P. The differential is averaged over P origins before anything is tested, so the displacement stays where it is while the noise around it shrinks. At sixty origins it is a fraction of a standard error and at six hundred it is several, which is why a longer out-of-sample period makes a nested table worse rather than better — a finding the earlier field measured directly and which follows from this arithmetic alone.

And it is compared with zero. A test of “these two forecast equally well” puts the null at zero and the truth at the displacement. The distance between them is the whole of the size distortion, and it does not matter that the distance is small in the units of the loss; it matters how large it is in units of the statistic’s own standard error, which is a different question with a different answer at every sample size.

So the practical reading is not “the displacement is negligible” or “the displacement is large”. It is that the displacement is a fixed quantity in a comparison whose noise shrinks, and any statement about whether it matters has to name the number of origins it was made at.

The one place the closed form stops #

Everything above is at a null where every candidate contains the truth. Away from it there is a second term and it is much larger.

If candidate i omits something that matters, its expected squared error is σ_i²(1 + q_i/(R − q_i − 1)) with σ_i² the variance of the errors the best version of that model can achieve — a quantity of order one rather than of order 1/R. So the differential between two wrong models has a fixed part, from the difference in what they omit, and a displacement, from the difference in what they estimate; and the fixed part dominates as soon as it is non-zero.

That is the regime a real specification search is usually in, and it means the displacement matters in exactly one situation: when the candidates are close in what they capture and differ in what they estimate. Which is the situation that produces a difficult table, and the reason anybody is running the search.

One number a practitioner can carry #

If none of the arithmetic above is worth remembering, one ratio is. Comparing a model of q₁ coefficients with one of q₀ on a rolling window of R rows, the larger one starts behind by

$$\frac{q_1}{R - q_1 - 1} - \frac{q_0}{R - q_0 - 1} \;\approx\; \frac{q_1 - q_0}{R}$$

of the benchmark’s own mean squared error, per origin. At R = 80 and one extra coefficient that is 1.4% of σ² — which sounds like nothing until it is set beside the difference the search is looking for, which in a table anybody finds difficult is of the same order.

The rule of thumb that falls out of it is the one experienced forecasters state without the derivation: a search over models of different sizes on a short estimation window will prefer small models, and the preference is not evidence. The number above says how much preference, and it says it in the units the search is scored in.

What is claimed here, and what is not #

This essay takes the displacement as a statement about parameter counts, and the claims are the exact closed form against counted values across four dimensions, the size of its leading term’s error, the window sweep, and the 77.3% preference for the smallest model at a true null.

What stays out and is named as a decision: non-Gaussian predictors, for which the expected leverage is not q/(R − q − 1) and the exact form becomes an approximation again; the fixed part of the differential away from the null, which is measured here only in the sign of its effect; and the reference distribution, which is the next essay.

The boundary against the nested field is the derivation. That the displacement exists, that Clark and West’s recentring removes it row by row, and what that repair does to a table’s false-positive rate are established there, together with the reference distribution a nested null needs and the multiplicity a table of nested variants carries. What is new here is that the quantity being repaired was never a fact about containment, and that the reading of the table is a separate lever which the same correction cannot move.

The checks, and the refusals that make them mean something #

Two claims are gated in this field’s library. Every candidate’s counted displacement is required to match the closed form to within three standard errors, across dimensions two to five and across a table where most pairs are not nested — which is the two-route check, and the two routes share no arithmetic at all. And every pair of candidates of the same dimension is required to sit at zero, which is the half that says the closed form is about counting rather than about containment and which would fail if the effect were nesting under another name.

The refusal that bears on this essay is the multiplicity correction applied to the ladder: Bonferroni is required to leave the open reading of a nested table above its nominal level, because a correction that succeeded there would mean the excess was a maximum over many comparisons rather than the shift this essay computes.