A charge the sample can afford

A penalty that counts independent rows charged each coefficient twice its own variance inflation, $2(1 + \phi\rho)/(1 - \phi\rho)$, and two things came of it. The charge agreed with the optimism counted on fresh rows to within a point up to persistence 0.7. And past that it parted company, reaching 234 against a count of 119 at persistence 0.95 on the widest of fifteen candidates, while still being the best selection rule in the table by a wide margin.

That essay offered a cause for the parting, and it was a good story. A series of a hundred and twenty rows at persistence 0.95 cannot be optimistic about movements slower than a hundred and twenty rows, because those movements are not in the sample at all, and so the inflation, which is a statement about a long series, overcharges. It proposed the obvious repair: the inflation for a series of $n$ rows rather than an infinite one, a finite sum instead of a ratio. It predicted that this would match the counted optimism at every persistence, and asked whether that charge would recover the oracle’s regret, and so whether a penalty for dependent rows is a method or a diagnosis.

All of that can be measured in an afternoon, and the story turned out to be wrong in an instructive way. The finite sum changes almost nothing, the real cause is elsewhere, and the charge that does match the optimism selects worse.

Three charges and two counts #

The finite-sample inflation has a closed form:

$$\kappa_n = 1 + 2\sum_{h=1}^{n-1}\left(1 - \frac{h}{n}\right)(\phi\rho)^h.$$

At $n = 120$ and $\phi = \rho = 0.95$ the sum for a predictor is 17.9 instead of 19.5, and for the constant it is 32.7 instead of 39. Summed over the widest candidate, which holds the constant and four predictors, the charge goes from 234.1 to 208.8. The count is still 119.1. The repair that was proposed removes a quarter of the gap at most. Slow movements just are not where the difference lies.

The figure at the head of this essay has the five numbers together. Reading them from the top at persistence 0.95: the long-series inflation charges 234.1; the finite sum 208.8; a count on rows far from the sample comes to 159.0; twice the trace of the hat matrix against the errors’ correlation, on the sample’s own design, comes to 132.2; and the count on the rows straight after the sample is 119.1.

Those five numbers are what the essay is about, and two of them are new. The trace is the exact statement a penalty is a trace gave in general, $2\operatorname{tr}(H\Sigma)$, with $H$ the candidate’s hat matrix and $\Sigma$ the errors’ correlation: it is what the fit’s rows actually do, with no long-series step. The far count moves the fresh rows six hundred rows beyond the sample, where nothing in the sample’s errors can reach them. Each accounts for a separate piece of the distance.

The first piece is the design in the sample #

Start with what does not move. The constant’s charge in a fit holding only a constant is twice the finite sum, $2\kappa_n = 65.4$, and that is exactly the trace of that fit, since a single column of ones has nothing to share and no sample variation to be wrong about. So the finite sum is the right answer for a constant, and the long-series 78 is too high by about the 13 the old story expected. That part of the story was true. It is just small.

The rest of the fall is in the predictors. The finite sum charges a predictor $2\kappa_n = 35.9$ at 0.95 on a hundred and twenty rows. The trace on the same rows charges, per added predictor, 15.4. Four predictors at a difference of about twenty each is the eighty that separates 209 from 132.

Part of that is the column’s own reading. The long-series formula assumes each column has the population persistence, and a sample of this length, once demeaned, does not show it: the lag-one correlation of a persistent column at a true 0.95 averages 0.904 over a thousand samples. Put that in the formula and the predictor’s charge falls to about 26. The rest comes from the columns being in one another’s way. In a regression holding a constant and four slow predictors, each slope is fitted only from what the others have left of that column, and for a series whose variation is concentrated in a few slow movements that shared part is large. The formula treats each coefficient alone, and the trace does not.

This is the difference between a finite-sum correction and a trace. The finite sum acknowledges that the series is short, and goes on treating the columns separately, at their population persistence; the trace evaluates the actual projection on the actual columns. The essay on observations that repeat each other worked out the inflation for a mean, where a single column means there is nothing to share, and the formula’s trouble only starts when there are several. Writing the answer as “a sample cannot be optimistic about movements slower than its length” was close, but it put the effect on the wrong thing. What the short sample changes is the design, and the variance formula was never being asked about its design.

The second piece is that the next rows are next #

Now the far count. Fresh rows straight after the sample are not independent of it. At persistence 0.95 the errors forty rows ahead are still correlated with the sample’s last rows by about 0.13, and the rows right after by close to 0.95. A fit that has absorbed the sample’s recent level into its constant, or into a slow predictor, is partly right about the rows that come next, for a reason that has nothing to do with whether the coefficient is any good.

So the fresh-row count is lower than the expected optimism on genuinely new data. Moved six hundred rows off, where the errors share nothing with the sample, the count at 0.95 rises from 119.1 to 159.0, a difference of more than five standard errors. Up to about 0.8 the near and far counts agree within their noise; the adjacency only starts to pay at 0.85, and by 0.95 it is worth a quarter of the far count.

This means there is not one optimism to match. The question the essay inherited, “does this charge match the counted optimism at every persistence”, assumed there was. A forecaster deployed straight after the sample gets the near reading, and for the first forty rows has a true benefit from the adjacency. A rule scored on some later regime gets the far reading. The two differ by a third at 0.95, and neither is wrong. The trace, at 132.2, sits between them. It is closer to the near count at every persistence, and at 0.95 it is 11% above it.

That is a decent result for a closed formula read off the sample’s own columns. So the first half of the old essay’s closing question, whether a right charge can be had, has an affirmative answer: yes, within about a tenth, with no fitting.

What a selection actually reads #

An estimate of the optimism and a selection rule are not the same object, and the essay on what a criterion predicts has been insisting on that from the start. Candidates all share the constant, so the constant’s charge cancels from every comparison. What decides a selection is the charge per added predictor: the difference between one candidate’s total and another’s, over the number of predictors between them.

On that reading the trace is a remarkably good instrument. Its marginal charge is 15.4 per predictor at persistence 0.95, against 14.6 counted near and 19.6 counted far. Up the whole range it sits within half a unit of the near count: 3.2 against 3.2 at 0.5, 5.2 against 5.4 at 0.7, 7.4 against 7.7 at 0.8, 12.1 against 12.4 at 0.9. And the long-series inflation is charging 39 per predictor at 0.95, more than two and a half times what is counted. So almost all of the marginal disagreement is at the top of the range, and the design in the sample is the whole explanation for it.

So here is the thing to test. Replace the long-series charge, which overcharges every predictor by a factor of two and a half at 0.95, with the trace, which is right to within a unit.

The accurate charge selects worse #

The regret on the fifteen candidates, given the persistences:

The finite sum, as the optimism table suggested, makes no difference. And the charge that is nearly right on the optimism has almost twice the regret of the charge that is nearly twice too large. Read off the sample rather than given, the same order holds at the top: 0.065 for the long-series inflation with persistences read, 0.074 for the trace with its persistence read, and Kendall’s correction to the read persistence moves the trace only to 0.071. The residuals’ lag-one correlation at a true 0.95 averages 0.80, and at 0.83 after correction, so the read charge is working from an underestimate in either form, and fixing the underestimate does not reach the given-persistence figure.

The earlier essay reported the moderate version of this finding: at 0.5 and 0.7 the inflation is right on the optimism and selects worse than Akaike’s, which is half of it. It left that as an instructive oddity. Taken with the top of the range, it looks like a rule. Each charge selects better when it is not the optimism, and it is on different sides of the optimism at the two ends of the range.

A flat charge, swept #

Since only the charge per predictor matters, the direct way to see what a selection wants is to give it a single number per predictor and sweep that number.

The picture has two regimes and a pivot between them.

At 0.5 and 0.7 the regret rises with the charge. Akaike’s 2 is the best on this grid, at 0.020 and 0.029, and the penalty’s correct optimism at 3.3 and 5.8 already costs something. The best candidates at these persistences hold two or three predictors, one of which has a coefficient of 0.12, which a heavy charge leaves out.

At 0.8 the curve is nearly flat: 0.043 at a charge of 2, 0.036 from 12 up.

At 0.9 and 0.95 the regret falls steeply as the charge rises, to 0.040 at a charge of 48 and to 0.027 at 96. Beyond about fifty it stops moving altogether, and the reason is in the mean candidate size. By a charge of 48 the rule picks a candidate with one predictor on nearly every draw, whichever predictor fits best. The best candidate available at 0.9 and 0.95 is the one holding the single strongest predictor. So the rule has stopped selecting and started saying “one predictor”, and on these persistences that is exactly the right thing to say.

So it is now possible to say what the long-series inflation’s advantage was. It charges 39 per predictor at 0.95, which is high enough to pick the smallest candidate on nearly every draw: its mean candidate size at that setting is 2.08, counting the constant. The trace, charging a correct 15.4, picks 2.61. The overcharge wins because it is large enough to end the selection, and it just happens to be correct that the selection should end.

Why an accurate charge does not stop the wrong inclusions #

The world being fitted has a predictor with no coefficient at all, so the mistake can be counted directly: how often does each charge let that predictor in, and what does it cost when it does?

Each column says something. On independent rows every charge is the same charge and lets the empty predictor in on 17.3% of draws, which is close to $P(\chi^2_1 > 2) = 15.7\%$, the textbook rate. Akaike’s rises to 71% at 0.95, and that is the failure the earlier essay diagnosed.

The trace, charged the right amount on average, doubles its rate to 35.3%. A charge equal to the mean of a gain does not hold the rate of clearing it constant when the gain’s distribution changes shape, and under this much persistence the in-sample gain from a spurious slow predictor is occasionally enormous, in the way spurious correlations always are. The long-series inflation, by being much larger than the mean, happens to hold the rate nearly level, at about 22% all the way to the top. The inflation is a poor estimate of the mean gain and a good estimate of something like its upper quantile, which is the thing selection is working against.

And the last column is why it matters: each of those mistakes costs ten times as much at 0.95 as at independence, 0.079 against 0.008, because a persistent predictor with a noise coefficient carries that noise into every fresh row. So a rule’s regret is about the mistake rate times the mistake cost, and both are rising under the accurate charge.

At 0.5 the balance runs the other way. A mistake there is cheap, the live coefficient of 0.12 is worth having, and a rule that takes risks at 29% does better than one that does not. That gives a single account of both ends of the range.

Method or diagnosis #

The earlier essay asked whether a penalty for dependent rows is a method or a diagnosis. There is now an answer with some detail in it.

As an estimate of an optimism it is a method. Twice the trace of the hat matrix against an AR(1) correlation, read on the sample’s own design, is within about a tenth of what is counted on the next forty rows at every persistence, and the per-predictor charge is within a unit.

As a selection rule at high persistence it is a diagnosis. What it diagnoses is that the sample cannot support a model. A long-series inflation of 39 per predictor is a way of saying “do not add anything”, and it says so with very little regard for what the fifteen candidates are. A rule that always takes the candidate with one predictor beats it at 0.95, with a regret of 0.027 at a flat charge of 96 against the inflation’s 0.032. At that persistence, a reader can drop the whole apparatus and read the persistence itself.

Between those two, it is a modest loss. At 0.5 and 0.7 the correct charge gives up about 0.005 to Akaike’s: real, by more than two standard errors, and small.

For an analyst the practical summary is short. Read the persistence of the residuals of the widest candidate, remembering that it reads about 0.15 low at the top. Below about 0.7, use Akaike’s. Above about 0.85, keep the smallest candidate with a predictor that a clinician or an engineer actually believes in, and report the optimism from the trace, since that number can be trusted. In between, the choice does not matter much.

How the numbers were made #

The optimism sweep uses 1,500 draws per persistence from the world of the earlier essay, with four predictors with coefficients of 0.30, 0.20, 0.12 and 0, the design and errors both first-order autoregressions, and the candidate fitted on 120 rows. The near count scores it on the next 40 rows and the far count on 40 rows that begin 600 rows further on. Both are in units of that candidate’s own error variance, so that a charge can be read directly against them. The trace is computed on the sample’s own columns, with the correlation applied by a two-sided recursion, and averaged over the draws.

Selection uses 300 draws on the same seeds as the earlier essay’s table, which is why its Akaike and long-series readings reproduce that essay’s 0.1281 and 0.0318 to the digit. That agreement is the check that nothing about the world has shifted between the two.

Standard errors on the near and far counts are 4.0 and 6.1 at 0.95. Regrets at 300 draws have a standard error of around 0.005, enough to resolve 0.032 against 0.061 and not to say anything about 0.043 against 0.046. Three things are required to hold: the finite sum must still charge half as much again as the count at 0.95; the trace must be within a fifth of it; and the far count must exceed the near one by more than three combined standard errors. The reading that closing the gap to the optimism improves selection is refused: the trace must lose to the overcharge at 0.95, and it does.

Still open: a charge for the mistake rather than the mean #

The account above suggests a form rather than a number. If the long-series inflation is working as a quantile of the spurious gain, then the charge to write down is a quantile of it, not its mean: the point a coefficient-free predictor’s gain clears 16% of the time, say, which is computable by simulation from the sample’s own columns under a read persistence. Whether that holds the false-inclusion rate level at every persistence, and whether it reproduces the flat sweep’s pattern, Akaike’s-sized below 0.7 and very large above 0.9, without being told the persistence directly, is the measurement that would turn a two-regime summary into one formula. It would also say what two defects and one resampling could add, since a simulated reference for the gain is the same object as a reference for the statistic there.

There is also a question about the coefficient itself. Everything here sits on one fixed vector of coefficients with a smallest live one at 0.12. The pivot near 0.8 has to depend on that: a larger smallest coefficient would hold the correct-charge regime further up. Sweeping the coefficients at each persistence would say whether the pivot is a property of the persistence or of the signal against it.

Which optimism, and what follows from it #

This essay takes the optimism of a linear fit on persistent rows, and what a selection does with a charge for it. Its claims are that summing the variance inflation over $n$ rows changes the charge at 0.95 from 234.1 to 208.8 against a count of 119.1; that twice the trace on the sample’s own design gives 132.2, and within a unit per predictor at every persistence; that the fresh-row count depends on how far off the rows are, 119.1 straight after the sample and 159.0 far from it; that the accurate charge doubles the regret from 0.032 to 0.061 at 0.95; and that past a flat charge of about fifty the rule is holding one predictor on nearly every draw, which is best at 0.9 and 0.95 for this table.

What stays out, by decision: other dependence laws, since the trace here assumes a first-order correlation for $\Sigma$ and the earlier field’s moving average would need its own; non-nested table shapes; and a persistence read better than from the widest candidate’s residuals. The boundary against where the two searches cross is that it compared a hold-out with criteria on regret alone, and this one separates what a criterion estimates from what it selects, and finds them pulling apart. The boundary against how long a block a multiplier shares is that it fixes the reference distribution of a hold-out’s scores, which is untouched here.