An excess and its window

The displacement is a parameter count gave an out-of-sample comparison a closed form. A correct candidate with one more parameter is behind by a fixed amount, $q/(R - q - 1)$ in its leverage, whatever else is going on. The displacement on a heavy tail noticed that this rests on one Gaussian fact, the expected leverage of a fresh row, and measured what becomes of it on other predictors at one window of eighty rows. A law with a finite fourth moment was a little over: 1.5% at eight degrees of freedom, 4.5% at five, 7.3% on an exponential. At three degrees of freedom, where the fourth moment is infinite, it reported 28%, and it noted that its four batches disagreed by more than a fifth with one another. It offered that as a caution. A forecaster with very heavy tails should not take the number on trust.

It left two things undone. The excess is a correction of order $1/R$ to a quantity of order $q/R$, so it should shrink as a share of the displacement as the window grows. And at three degrees of freedom it wanted to know whether a long enough window makes the estimate settle at all. A sweep over windows would say. It would also turn the caution into a rule a forecaster could apply to their own predictors’ kurtosis.

Running the sweep as described does not work, as it happens. And the reason it does not work turned out to matter more than the sweep.

A measurement with a chi-square in it #

The earlier route draws $R$ rows of predictors and one fresh row, computes the fresh row’s leverage $x_0' G^{-1} x_0$ with $G = X'X$, and averages. That is an honest estimate, and on heavy tails it is a very noisy one. At a window of 320 rows, with 12,000 draws, the excess reads 1.7% ± 0.6% on normal predictors, where the closed form says it is zero, and on three degrees of freedom it reads 21% at 160 and 5% at 320, with its batches ranging from 0% to 46%.

Nothing’s wrong with the estimator, it has just spending its effort badly. The fresh row is drawn independently of $G$, with mean zero and unit variance in each predictor. So averaging over the fresh row alone gives

$$\mathrm{E}[x_0' G^{-1} x_0 \mid G] = \operatorname{tr}(G^{-1}),$$

and no fresh row has to be drawn at all. All of the noise from a chi-square-shaped quantity disappears and nothing about the estimand has changed: this is the same expectation, written so that the part of it that can be done in closed form is. With that substitution, 500 draws in each of six batches give better precision than 12,000 did. On normal predictors the excess is now −0.04% ± 0.09% at 80 rows, −0.02% ± 0.08% at 320, and 0.00% ± 0.06% at 640. That is the closed form, confirmed to a tenth of a per cent where it should be, and the route can now be trusted to resolve the small numbers it needs.

The sweep #

The next figure is the excess at windows from 20 to 640 rows, on log scales, and the table beneath it holds the same readings.

Every line falls, and the three with a fourth moment fall at about the same slope. Each doubling of the window halves the excess, so the answer to the earlier question for five degrees of freedom is that 4.5% is not a property of the predictors. It is a property of the predictors at eighty rows. It is 2% at 160 and under 1% by 640.

At three degrees of freedom the line is visibly less steep. It falls by about a third per doubling rather than a half.

The rule #

If the excess falls as one over the window, then the excess multiplied by the window is a constant, and the constant has to come from the law.

The exponential is the cleanest: its excess times the window runs 5.58, 5.61, 5.76, 5.65, 6.19 and 5.45, level within noise across a thirty-two-fold range of windows, against an excess kurtosis of exactly six. So the rule is almost embarrassingly simple:

$$\text{excess} \approx \frac{\kappa}{R},$$

with $\kappa$ the predictors’ excess kurtosis. The $t$ on eight has $\kappa = 1.5$ and reads 0.83, 1.11, 1.43, 1.16, 0.91, 1.17: lower and noisy, and about right. The $t$ on five has $\kappa = 6$, the same as the exponential, and reads 2.14, 3.00, 3.15, 3.22, 4.02 and 4.46: well below six at short windows and climbing.

The difference between those last two is worth stopping on, because they share a fourth moment. The exponential has every moment there is. The $t$ on five has a fourth moment and no sixth, so although the rule’s limit is the same for both, its approach is not. Its fourth moment is itself dominated by rare large values, which a window of twenty rows usually does not contain, so the sample behaves for a while like a lighter law and only learns its true kurtosis slowly. So a measured excess kurtosis on a sample of two hundred rows of a heavy-tailed predictor will itself come out low, and that reads as the same effect from two sides.

A rule a forecaster can use, then. Compute the excess kurtosis of each predictor, and divide by the window. On 80 rows, a kurtosis of six means the displacement per parameter is about 7.5% over the closed form; at 320 rows, about 2%. An analyst using a penalty built on the Gaussian count is undercharging by that much, and at most windows anyone uses for a comparison that is small against the other things wrong with the comparison. A charge the sample can afford found dependence moving that same charge by a factor of several.

The rule also answers a planning question directly. How long does a window have to be before the closed form is good to a per cent? The answer is about a hundred times the excess kurtosis: 150 rows for a $t$ on eight, 600 for a $t$ on five or an exponential. Against that, a forecaster with sixty origins and a window of eighty has a Gaussian count that is right to about 7.5% on a skewed predictor, and that is not a serious error. The cases where it matters are short windows and many coefficients, since the excess applies to every parameter. A window of twenty rows on an exponential predictor is 28% over per parameter, and with five coefficients that is a difference between candidates of more than a tenth of an error variance. A short rolling window is exactly where the corner a test is calibrated at puts most of its attention, and it is where a Gaussian count is least safe.

Three degrees of freedom #

At three degrees of freedom the fourth moment does not exist, the rule has nothing to divide, and so the excess cannot be falling as one over the window. It does not. The excess times the window runs 8.9, 13.1, 18.1, 26.3, 37.4 and 48.8, climbing steadily. Over a 32-fold range of windows, the excess falls from 44.44% to 7.63%, a factor of 5.8, and $\sqrt{32} = 5.66$. So at this law the excess falls as about the root of the window, and that is a rate rather than an accident. The sample’s own kurtosis grows with the window here, since new large values keep arriving, and an excess proportional to the sample’s kurtosis over the window falls more slowly than the window grows.

So does a long enough window make the estimate settle? It does settle. It settles slowly: at 640 rows it is still 7.6% over, and on the earlier essay’s eighty rows it is 22.6%. Extending the same rate, a per cent would take something like forty thousand rows, which is to say that a law this heavy never reaches the closed form on any sample anyone has.

The batches disagreed about the row #

Here is the correction. The earlier essay read its batches’ disagreement at three degrees of freedom as saying that the leverage might have no stable mean, and it held the 28% loosely on that account. Its cautious wording was a fair response to what it could see. But an estimand’s mean and an estimator’s spread are different things.

With a fresh row drawn, four batches of 20,000 read 53.08%, 23.60%, 18.13% and 18.61%. With the fresh row integrated out, six batches of just 500 read 22.66%, 22.04%, 23.80%, 22.20%, 21.49% and 23.52%. They agree within about a point and a half, with forty times fewer draws.

So the instability was in the fresh row. A leverage is a quadratic form in that row, and on three degrees of freedom a square of a heavy-tailed variable has infinite variance, so a sample mean of leverages converges very badly: one batch can contain one enormous row that moves it by thirty points, which is what the first batch above did. The expected leverage, however, exists and is perfectly well behaved, since the fresh row has a finite variance and $G^{-1}$ is the rest. The right number at eighty rows is 22.6% ± 0.4%, not the 28% the earlier essay’s mean returned, and the excess of the fourth row in its table should read accordingly. The counted route, which also reported about a quarter over, has the same heavy-tailed row in it and its own averaging across origins to absorb it.

So the caution is misplaced in a useful way. It said: at three degrees of freedom a forecaster should not trust this number. A better version is: the number exists, can be computed, and a Monte Carlo that draws fresh rows will not find it. That is a lesson about simulating with heavy tails generally. Where the bootstrap lies concerns the same kind of statistic for a different reason. If part of an expectation can be done analytically, it should be, and on a heavy-tailed law that may be the difference between 500 draws and an estimate that does not converge in any budget.

It does not change the operational reading of a single rolling comparison, which is an average of realised fresh rows. A forecaster with predictors at three degrees of freedom has a comparison whose expected displacement is 22.6% over at eighty rows and whose actual displacement, in any one run of sixty forecasts, may be nowhere near that. A heavy-tailed row can move it by a great deal. Both of those are true, and the earlier essay’s caution is right about the second.

Why the fourth moment, and why divided by the window #

The earlier essay gave the mechanism in words, and this one can put it in a line. $G = R(I + E)$ with $E$ a mean-zero matrix of size about $1/\sqrt{R}$, so

$$G^{-1} = \tfrac{1}{R}\left(I - E + E^2 - \cdots\right), \qquad \mathrm{E}\operatorname{tr}(G^{-1}) \approx \tfrac{1}{R}\left(q + \mathrm{E}\operatorname{tr}(E^2)\right).$$

Each diagonal entry of $E$ is a sample mean of squares less one, whose variance is $(\kappa + 2)/R$; each off-diagonal is a sample mean of products of uncorrelated variables, whose variance is $1/R$ whatever the law is. So $\kappa$ enters exactly once per predictor column, at order $1/R$, and the rest is what the Gaussian has anyway. The constant column has no variance and adds nothing. That is the Jensen effect in the earlier essay’s description, and it is where the rule’s form comes from: a correction proportional to $\kappa$, over a window, per predictor.

That arithmetic has a limit that the measurements confirm. Per added predictor it contributes $\kappa/R^2$ against a displacement of $1/R$ per parameter: an excess of $\kappa/R$, which is the figure above. That is first-order. So the rule is not a fit, it is a one-line expansion, and the $t$ on eight’s and $t$ on five’s shortfalls at short windows are the expansion’s higher terms, whose coefficients need the moments those laws lack.

A word about the counted route, since it is not what was changed here. It runs a rolling comparison over sixty origins, and averaged across that many realised fresh rows its heavy-tailed noise is a real feature of what a forecaster would see, not an artefact to integrate out. So the fresh-row version remains the right description of one comparison’s reading, and the integrated one of its expectation. An analyst needs both, and each answers something different.

How the numbers were made #

Each batch draws 500 samples of $R$ rows of a constant and four independent predictors at mean zero and unit variance, forms $G$, and computes the trace of its inverse at two and at five coefficients. The excess per parameter is the difference between the two, divided by three, against the same difference from the closed form. Six batches at each of five laws and six windows give standard errors of about 0.1 of a point at 80 rows and above on the laws with a fourth moment, and of about 0.3 on the $t$ on three.

Five things are required. On normal predictors the excess must be within 0.6% of zero at 80 rows and above. On the exponential, the excess times the window must be within 1.5 of six at every window from 40 up. At three degrees of freedom, the same product must grow at least fourfold from 20 rows to 640, and the excess must fall by between a factor of four and of eight across that range. And the reading that the batches at three degrees of freedom still disagree by a tenth once the fresh row is integrated out is refused at 640 rows, where their spread is 1.5%.

The older route’s numbers are kept in the figure as they stood, since they are an accurate record of what that route computed. They are off by what they are off by because of one batch.

Still open: a sample’s own kurtosis, and predictors that depend on each other #

Reading the kurtosis off the sample. The rule needs $\kappa$, and a sample estimate is itself low on a heavy-tailed law at the windows this rule is for: that is the $t$ on five’s climb, read from the other side. Whether a plug-in with the sample’s own excess kurtosis, divided by the window, reproduces the true excess at the same window, which would be a happy cancellation, is one line more in the same simulation. It is the check a forecaster using this would want done first.

Correlated predictors. Everything here has independent predictors. With correlated columns the trace of $G^{-1}$ picks up a contribution from a small eigenvalue that the earlier essay already saw growing with the model’s size, and the Gaussian closed form itself changes. Whether the rule becomes $\kappa/R$ times a factor in the correlation matrix’s condition, or something less tidy, is the natural next sweep, and an order chosen on correlated columns is where it would bite.

What the window does #

This essay takes the excess of the expected out-of-sample leverage over the Gaussian closed form, as the window grows. Its claims are that with the fresh row integrated out the closed form is exact on normal predictors to about a tenth of a per cent; that on an exponential the excess is the excess kurtosis over the window, within noise, at every window from 20 to 640; that a law with a fourth moment and no higher ones reaches the same limit from below; that at three degrees of freedom the excess falls as the root of the window, from 44.44% at 20 rows to 7.63% at 640; and that the earlier figure of 28% at 80 rows should read 22.6%, because the batches that disagreed were disagreeing about the fresh row rather than the mean.

What stays out, as a decision: correlated predictors, named above; non-zero coefficients, where a misspecified candidate’s displacement picks up a bias term this ignores; and any correction to a criterion, since at these sizes the excess is small beside the things that already move a criterion’s charge. The boundary against the displacement on a heavy tail is that it fixed the window and read the laws, and this one reads the window and finds that the law only enters through one number divided by it.