The displacement on a heavy tail

A correction. The 28% this essay reports at three degrees of freedom should read 22.6%, and the disagreement between its batches was about the fresh row rather than about the mean. Integrating the fresh row out gives batches that agree to a point and a half. An excess and its window has the measurement.

The displacement is a parameter count found that a correct candidate with more parameters is behind a correct candidate with fewer, out of sample, by an amount with nothing about nesting in it. A regression with $q$ coefficients fitted on $R$ rows of Gaussian predictors has expected out-of-sample squared error $\sigma^2\big(1 + q/(R - q - 1)\big)$, so the expected differential between two correct candidates is a difference of two such terms: arithmetic on two integers and a window length. Counted from rolling forecasts across a table of fifteen candidates, it matched to within its own standard error.

It named the one place that arithmetic rests on something other than arithmetic. The term $q/(R - q - 1)$ is the expected leverage of a fresh row — $E[x_0^\top (X^\top X)^{-1} x_0]$ — and that expectation is $q/(R - q - 1)$ exactly when the predictors are Gaussian. On any other predictors it is something else, and the closed form becomes an approximation again. The essay left out non-Gaussian predictors as a decision. This essay measures how far the approximation can be carried, and finds the answer is decided by one moment.

Five laws for the predictors #

The four predictors are drawn from five laws, each at mean zero and unit variance so that only the shape changes: the normal; Student’s t at eight, five and three degrees of freedom, whose tails grow heavier in that order and whose fourth moment is finite at eight and five and infinite at three; and a centred exponential, which is skewed with a fourth moment of nine. The errors stay normal and independent, every candidate is correct — every coefficient is zero — and the window is eighty rows, the earlier essay’s.

The displacement is measured two ways.

Counted. The earlier essay’s construction, unchanged: a rolling comparison over sixty origins of all fifteen candidates, each refitted on its window, and the average differential of each candidate against the smallest, regressed on the difference in their parameter counts. Fifteen hundred draws per law. This route never computes a leverage.

From the leverage. The predictors alone, drawn many times: eighty rows and one fresh row, the fresh row’s leverage under the regression on the constant and the first one to four predictors, averaged. Four independent batches of twenty thousand draws per law. This route never forecasts anything.

The two routes, and the Gaussian value #

The Gaussian closed form puts the displacement at 0.01386 of an error variance per parameter. On normal predictors the leverage route gives 0.01386 and the counted route 0.01382: the earlier essay’s agreement, and a check that both constructions are doing what they claim.

On the other laws the two routes agree with each other and part from the Gaussian value by an amount that grows with the tails. At eight degrees of freedom the leverage gives 0.01407 and the count 0.01379; at five, 0.01448 and 0.01419; on the exponential, 0.01487 and 0.01465. Against the Gaussian value those are 1.5%, 4.5% and 7.3% more by the leverage route. At three degrees of freedom the leverage gives 0.01780 and the count 0.01751 — 28% more than the Gaussian value, and the routes still within two per cent of each other.

So the closed form is a Gaussian fact that very nearly generalises. For predictors with a finite fourth moment the displacement is the parameter count to within a few per cent, and the few per cent are larger the heavier the tails or the stronger the skew.

Why the fourth moment #

The excess has a mechanism that says why it is the fourth moment and not the third. The leverage of a fresh row is a quadratic form in the inverse of $X^\top X$, and the inverse of a random matrix is larger on average than the inverse of its average — that is Jensen’s inequality for a convex function, and it is why even Gaussian rows give $q/(R - q - 1)$ rather than $q/R$. How much larger depends on how much $X^\top X$ varies from sample to sample, and $X^\top X$ is a sum of squares and products of the predictors, so its variance is a fourth-moment quantity.

Gaussian predictors have a fourth moment of three. The t at eight has 4.5, the t at five has nine, the exponential has nine with a skew besides, and the t at three has none at all. The excesses order the same way, with the exponential a little above the t at five — the two share a fourth moment, and the exponential’s skew probably adds to the spread of the cross products.

The excess grows with the model #

The per-parameter figure is an average over candidates of two to five coefficients, and the leverage route can be read one size at a time. On normal predictors the four expected leverages are 0.0257, 0.0391, 0.0531 and 0.0673, against the closed form’s 0.0260, 0.0395, 0.0533 and 0.0676 — within a per cent at every size, which is the agreement the route has to show before anything else it says is worth reading.

At five degrees of freedom they are 0.0263, 0.0405, 0.0550 and 0.0698: 1.3%, 2.6%, 3.1% and 3.3% above the closed form. At three they are 0.0285, 0.0448, 0.0644 and 0.0819: 10%, 13%, 21% and 21% above it. The excess is not a constant fraction of the leverage. It grows with the number of coefficients, so the displacement between a large candidate and a small one grows faster than their difference in parameter count says.

That has the same Jensen’s-inequality reading as the excess itself. A larger regression inverts a larger $X^\top X$, whose smallest eigenvalue is smaller and more variable, and the fresh row’s leverage is most sensitive to exactly that eigenvalue. Heavy tails make the smallest eigenvalue more variable still, and a candidate with five coefficients feels it more than one with two.

The counted route says the same thing in its own units. Under the t at three the mean differential against the smallest candidate is 0.0157 for a candidate one coefficient larger, 0.0334 for two and 0.0516 for three, against the closed form’s 0.0135, 0.0274 and 0.0416: 16%, 22% and 24% more. On normal predictors the same three are 0.0130, 0.0269 and 0.0413, each within a few per cent of the formula. Two routes, one counting forecasts and one never making any, agree that the excess rises with the gap.

For a search that ranks candidates of very different sizes this is the part that matters. A table whose candidates run from two coefficients to twelve on heavy-tailed predictors is not displaced by the closed form’s amount times ten; it is displaced by more, and by an amount the closed form gives no warning of.

When the mean will not settle #

The t at three is different in kind, and the leverage route shows how.

Under the four laws with a finite fourth moment the four batches agree to within two per cent of each other: 0.01379 to 0.01398 on normal predictors, 0.01437 to 0.01464 at five degrees of freedom, 0.01464 to 0.01515 on the exponential. At three degrees of freedom they run from 0.01638 to 0.02122, a spread of more than a quarter of their own mean, on eighty thousand draws in all.

The expectation exists — the predictors have a variance, and the fresh row’s leverage has a finite mean — but the leverage itself has no variance, because a fresh row’s leverage grows like the square of its own size and a t at three has no finite fourth moment. An average of draws with no variance converges, slowly and unevenly, and is carried by its rare extreme rows: a batch that happens to contain one fresh row far out in the tail moves by more than all its ordinary rows together. So the number exists and cannot be measured to the precision every other number in this essay has, and a forecaster on predictors this heavy-tailed should read the closed form’s displacement as a lower bound of uncertain size rather than as a correction.

The counted route agrees with the leverage route at three degrees of freedom, 0.01751 against 0.01780, and that agreement should be read with the same caution. Both routes average the same rare events, and fifteen hundred draws of a rolling comparison are no better placed to catch them than twenty thousand draws of the predictors alone.

The preference for the smallest model #

The earlier essay’s practical consequence of the displacement was a preference. Where every candidate is correct, the population ordering is by parameter count and nothing else, so the search’s winner is one of the smallest candidates far more often than a coin over the table would say: 77.3% of the time, against the 26.7% that four candidates of fifteen would get by chance.

Under the other laws it is 76.5% at eight degrees of freedom, 76.0% at five, 75.0% at three and 76.1% on the exponential. The displacement grows with the tails, and the preference does not: it moves by two points across laws whose displacements differ by more than a quarter.

The two quantities answer different questions. The displacement is a mean, and a mean is moved by the rare rows that make a heavy-tailed predictor’s leverage large. The preference is a probability that the smallest candidate’s average squared error is the lowest in the table, which is decided by the typical draw rather than the rare one — and in the typical draw a heavy-tailed predictor’s rows are, if anything, more concentrated than a Gaussian’s of the same variance, because the variance is being spent in the tails. The search is ranked by dimension under every law here, by about the same margin; what the tails change is how far the losing candidates are behind on average, not how often they lose.

What a reading of a search should carry #

When the benchmark is a candidate found that the reading of a table — which candidate is compared with which — moves the false-positive rate independently of the displacement, and that the displacement’s correction cannot touch it. The tails add a third lever with the opposite character: they move the displacement and leave the preference, and so the false positives that come from preferring the small model, nearly where they were.

So the closed form’s displacement can be carried to any predictors whose fourth moment is moderate, with an error of a few per cent that is always in the direction of making larger candidates look worse than the formula says. It cannot be carried to predictors whose fourth moment is infinite, and the failure there is not that the number is wrong but that no sample of the size a forecaster has can say what it is.

The line that one point drew is the same fact seen in a single regression rather than averaged over many: one row far from the others carries most of a fit’s leverage, and a law that produces such rows regularly is a law under which the average leverage is a poor summary of any particular fit.

The other places the same expectation is hiding #

The expected leverage of a fresh row is not a quantity only the displacement uses, and the measurement above says something about each of the others.

A penalty is a trace found Akaike’s charge of two a coefficient to be what an optimism trace collapses to on independent rows. That trace is a sum of leverages, and on Gaussian predictors its expectation is the parameter count plus the same small finite-sample correction the displacement carries. On heavy-tailed predictors the trace grows the way the leverage does here, faster for the larger candidates, so a criterion with a penalty linear in the count — a prediction of the hold-out that assumed the Gaussian expectation — under-charges the large candidates by an amount that rises with their size — in the direction that makes a criterion buy coefficients, and by a few per cent per coefficient wherever the fourth moment is finite.

A reference distribution is the case where the expectation can be avoided. The resamplings the next essay in this field compares redraw residuals and keep the predictors the sample actually had, so every resample is scored on the realised rows’ leverages rather than on their average. A reference distribution built that way is conditional on the design, and conditioning on the design is exactly what removes the heavy tail’s rare rows from the question: if the sample contains one, the reference distribution knows; if it does not, the reference distribution does not need to. That is a reason to expect the generated references to be more robust to the predictors’ tails than any correction derived from the expectation, and it is offered as a reason, not measured here.

And the corner the test is calibrated at is a statement about which configuration of the candidates a search’s critical value is computed at. The tails do not move that corner — every candidate is still correct, and the ordering is still by dimension — but they move how far the corner sits from the candidates’ real positions, which is the displacement, and so how much a test calibrated there gives away.

Rolling forecasts beside the leverage of a fresh row #

The counted route is the earlier essay’s rolling comparison on the fifteen-candidate table with all coefficients zero, eighty rows of window and sixty origins, fifteen hundred draws per law; its per-parameter displacement is the slope of each candidate’s mean differential against the smallest candidate on the difference in their parameter counts. The preference is the share of eight hundred searches, on the earlier essay’s seed, whose lowest mean squared error belongs to a two-coefficient candidate. The leverage route draws eighty rows and one fresh row of four predictors and the constant, and computes the fresh row’s leverage under each of the four nested regressions from the constant alone up, in four batches of twenty thousand on independent streams; its per-parameter displacement is the difference between the largest and smallest regressions’ mean leverage divided by three.

Student’s t at $\nu$ degrees of freedom is drawn as a normal over the root of an independent chi-square on $\nu$ over $\nu$ and rescaled to unit variance by $\sqrt{(\nu - 2)/\nu}$; the exponential is centred at its mean of one.

A Gaussian constant carried as far as the fourth moment #

The claims are that the out-of-sample displacement between correct candidates is the expected leverage of a fresh row, by two routes that agree to within two per cent under every predictor law here; that it exceeds the Gaussian closed form by 1.5%, 4.5% and 7.3% at eight and five degrees of freedom and on an exponential, and by about 28% at three; that at three degrees of freedom its estimate does not settle, four batches of twenty thousand disagreeing by more than a quarter; and that the search’s preference for the smallest model moves by two points across all five laws.

Not claimed: anything about predictors that are dependent on each other or in time, where the persistence of a design changes the optimism through a different route; anything about errors that are not Gaussian, which the earlier essay’s derivation never used; and anything about windows other than eighty rows, where the excess, being a correction of order one over the window, should shrink.

Still open: the excess as a function of the window #

The excess over the closed form is a correction of order $1/R$ to a quantity of order $q/R$, so it should shrink relative to the displacement as the window grows, and the rate it shrinks at should depend on the fourth moment. A sweep of the leverage route over windows from forty to three hundred and twenty under each law would say whether the excess at five degrees of freedom is the 4.5% measured here at every window or a number that halves with each doubling — and, at three degrees of freedom, whether a long enough window makes the estimate settle at all, which is the measurement that would turn the caution above into a rule a forecaster could apply to their own predictors’ kurtosis.