A penalty that counts independent rows

Where the two searches cross found the one dial that changes which of two ways of choosing a model does better. Splitting the sample differently never let a rolling hold-out beat Akaike’s criterion; making the rows repeat each other did. With both the predictors and the errors autoregressive at the same coefficient, the criterion’s advantage narrowed, closed at 0.81 and reversed, and by 0.95 it was giving up 0.128 of forecast error against the best candidate where the hold-out gave up 0.109. The criterion failed in the direction its own derivation predicts: its winners grew from 3.31 coefficients to 3.90 while the best model shrank from four to two.

The essay named the repair and declined to build it. A criterion’s penalty is an estimate of how much an in-sample fit flatters itself, and that flattery is set by how many independent rows each coefficient is fitted from. Akaike’s charge of two error variances a coefficient assumes the answer is $n$. A penalty computed from an effective number of rows would be the honest one — but estimating that number from the rows being selected on is, the essay said, a second selection problem of the same shape as the first.

This essay builds it, measures what it costs to estimate, and finds that the charge it makes is nearly exactly right where it was supposed to be and wrong in an instructive way where it was not.

The charge, written down #

A regression coefficient fitted to a persistent column against persistent errors has a variance larger than the textbook one by a factor that has a closed form when both are first-order autoregressions. With the column’s lag-one correlation $\phi$ and the errors’ $\rho$, the coefficient’s variance is inflated by

$$\kappa = \frac{1 + \phi\rho}{1 - \phi\rho},$$

which is the number of rows the coefficient is fitted from divided by the number of independent rows it is fitted from — the observations that repeat each other worked out the same arithmetic for the mean. An in-sample fit’s optimism is twice the sum of its coefficients’ variances in units of the error variance, so the charge for a candidate is $2\sum_j \kappa_j$ over its coefficients, the constant included — a column of ones has $\phi = 1$, and so the constant carries the errors’ own inflation, $(1 + \rho)/(1 - \rho)$. At independence every $\kappa$ is one and the charge is Akaike’s.

Two versions are measured. One is given the persistences of the law. The other reads them off the sample: each predictor’s lag-one correlation from the predictor, and the errors’ from the residuals of the widest candidate, the one holding all four predictors. That is one estimate shared by every candidate. A persistence estimated from each candidate’s own residuals would make the penalty a function of the fit it is penalising, which is the second selection problem in its sharpest form, and reading it once from the widest fit is the way every whitened rule here estimates a nuisance it needs before choosing anything.

Only the predictors’ charges decide #

One part of the charge does no work, and it is the largest part. The constant’s inflation is the errors’ own, $(1 + \rho)/(1 - \rho)$, which is 39 at a persistence of 0.95 against 19.5 for each predictor. But every candidate carries the constant, so its charge is added to every score alike and cancels from every comparison. What decides a selection is the charge on each predictor, $2\kappa_j$, and the difference between two candidates’ charges is the sum over the predictors one holds and the other does not.

That makes it possible to say exactly where an older penalty stands. Schwarz’s criterion charges $\log n$ a coefficient, 4.79 at a hundred and twenty rows, where Akaike’s charges 2. Setting $2\kappa = \log n$ gives $\phi\rho = 0.41$, which with a shared persistence is 0.64. So Schwarz’s penalty is the honest charge for a persistence of about 0.64 on a hundred and twenty rows, too heavy below it and too light above it — which is why the earlier essay found it worst on independent rows, best of the three it compared from 0.75, and still best of them at 0.95, where the honest charge is eight times its own. It was never answering the prediction question; it happens to charge, at one persistence, what the prediction question asks for.

A penalty is a trace gave the general form: the optimism is the trace of the hat matrix against the error covariance, and the parameter count is that trace only when the covariance is a multiple of the identity. The inflation charge is that trace computed for the one case where it has a closed form a coefficient at a time.

Regret #

On independent rows the new penalty is Akaike’s, and it gives up the same 0.0193 — the version that reads the persistences gives up 0.0188, the difference being that it occasionally reads a persistence where there is none.

From persistence 0.8 up it is the best rule in the table, and by a growing margin. At 0.85 Akaike’s criterion gives up 0.0666, the hold-out 0.0594 and Schwarz’s 0.0575; the inflation penalty gives up 0.0496 reading the persistences and 0.0426 given them. At 0.9 the four are 0.1075, 0.0913, 0.0850, and 0.0656 and 0.0469. At 0.95, where every rule the earlier essay compared had collapsed to between 0.105 and 0.128, the inflation penalty gives up 0.0653 reading the persistences and 0.0318 given them — half of Akaike’s regret, and a quarter.

So the repair does what the earlier essay’s diagnosis said it would. The criterion was failing because its penalty counted rows, and a penalty that counts independent rows instead does not fail that way. What is left at 0.95 is not the criterion’s defect; it is the difficulty the earlier essay already recorded, that a series this persistent carries little information for anybody.

What the fit is optimistic by, counted #

The penalty is a claim about the optimism of an in-sample fit, and that claim can be checked without any selection at all: fit a candidate on the first hundred and twenty rows, use it on the next forty, and count by how much the in-sample residual sum flatters the fresh error, in units of the candidate’s own error variance.

On independent rows the candidate with all four predictors is counted at 11.6 and Akaike’s charges 10, the small excess being the finite-sample correction Akaike’s corrected form exists for. At persistence 0.5 the count is 20.5 and the inflation charges 19.3; at 0.7, 34.5 counted against 34.7 charged. Akaike’s charges 10 at every persistence. The one-predictor candidate tells the same story: 17.3 counted at 0.7 against 17.2 charged.

That is the two routes agreeing on a number neither was tuned to: one is a variance-inflation factor from the theory of autocorrelated regression, the other is an average of in-sample and out-of-sample errors with no theory in it at all. Up to 0.7 the inflation is not an approximation to the optimism. It is the optimism.

Past 0.8 they part. At 0.85 the count is 65.0 and the charge 74.3; at 0.95, 118.6 against 234.1 — twice what the fit is actually optimistic by. The inflation factor is a statement about a long series, and a hundred and twenty rows of a series at 0.95 is not long: its slowest movements are not in the sample at all, so they cannot be fitted, and the optimism that would come from fitting them is never realised. The charge in that region is too large, and the rule that pays it still beats every rule that charges too little.

The size of model each rule buys #

The best candidate has four coefficients at persistence 0 and 0.5, three at 0.7 and 0.8, and two from 0.85 up. Akaike’s criterion buys 3.31 at 0 and 3.90 at 0.95 — growing as the best size shrinks, which was the earlier essay’s finding. Schwarz’s buys between 2.65 and 3.45, growing in the same direction from a smaller start.

The inflation penalty given the persistences buys 3.01 at 0.5, 2.46 at 0.8 and 2.08 at 0.95: it shrinks with the best size and arrives at it. Reading the persistences, it buys 3.08, 2.67 and 2.71. It turns the right way and stops short, and the reason it stops short is in what it reads.

What a hundred and twenty rows can tell the penalty #

The predictors’ persistence is read well: 0.474 at 0.5, 0.764 at 0.8, 0.905 at 0.95. The errors’ is read from residuals, and it is read low: 0.437, 0.698 and 0.803. A fit takes some of the errors’ slow movement into its coefficients, so the residuals are less persistent than the errors were — the same attenuation the essay on a dependence fitted with the line found when it estimated the persistence and the regression together.

And the inflation is steep exactly where the reading is worst. At $\phi\rho = 0.9025$, which is 0.95 squared, a coefficient’s inflation is 19.5. At the values read off the sample, about 0.905 times 0.803, it is 6.3. A sixth of the persistence missing in the reading is two thirds of the charge missing in the penalty, which is why the version that reads the persistences gets half of the repair at 0.95 rather than all of it.

That is the second selection problem the earlier essay anticipated, and it turns out to be a smaller one than feared and a different one. It is not that the penalty chases its own fit — it was built so that it cannot. It is that the nuisance it needs is a persistence near one, where every estimator from a hundred and twenty rows is biased towards zero and every function of it that blows up near one inherits the bias many times over.

A charge that is right, and a selection that is not better #

There is one more line in the regret figure and it is the most instructive one. At persistence 0.5 Akaike’s criterion gives up 0.0196. The inflation penalty, which at 0.5 charges 19.3 against an optimism counted at 20.5 — nearly right, where Akaike’s charges half of it — gives up 0.0236 given the persistences and 0.0231 reading them. At 0.7 the gap is similar: 0.0294 for Akaike’s, 0.0348 and 0.0339 for the inflation.

So at moderate persistence a penalty that estimates the optimism without bias selects worse than one that underestimates it by half, by more than two standard errors of either regret. The penalty is right about the expected optimism of each candidate, and choosing the candidate with the smallest unbiased estimate of its risk is not the same as choosing well. Both selected-model sizes in the figure fall short of the best: at 0.5 the best candidate has four coefficients and the inflation penalty buys about three, because the smallest coefficient, 0.12, is worth having by a margin narrower than the noise in the comparison and a larger charge resolves that noise towards leaving it out.

A criterion is a prediction of the hold-out made the same point from the other side: Akaike’s penalty is an unbiased estimate of an optimism, and its virtue as a selection rule was always a further fact rather than a consequence. Here that further fact fails at moderate persistence and holds strongly at high persistence, and the honest summary is that a penalty for dependent rows is worth charging when the rows depend a lot and costs a little when they depend modestly.

What an analyst would do with it #

Put in the units an analyst meets it in: ten years of monthly data, a hundred and twenty rows, a predictor whose lag-one correlation is 0.8 and residuals whose lag-one correlation is 0.7. The predictor’s inflation is $(1 + 0.56)/(1 - 0.56) = 3.5$, so the honest charge for including it is 7.1 error variances where Akaike’s charges 2 and Schwarz’s 4.8. A predictor that improves the in-sample log-likelihood by three units — comfortably past Akaike’s bar and Schwarz’s — falls well short of the bar its own persistence sets. On independent rows that predictor would be kept by every rule here; on these rows only the rule that counts independent rows drops it, and on the evidence of the regret figure it is right to.

The penalty needs three things an analyst already has: the lag-one correlation of each predictor, the lag-one correlation of the residuals of the widest model, and the arithmetic above. It changes nothing on independent rows. From persistence 0.8 up it is the best rule here whether the persistences are known or read, and it halves the criterion’s regret at 0.95 even read off a sample that underestimates them.

What it does not do is reach the oracle, and the gap between the read and the given versions is the part an analyst cannot close from the sample alone. Two things would close part of it. A bias-corrected estimate of the residuals’ persistence — correcting the persistence priced the obvious one — would recover some of the missing charge, at the cost of a noisier penalty. And a penalty that is capped where the inflation formula outruns the counted optimism, as it does past 0.85 on a hundred and twenty rows, would charge less than the formula and more than Akaike’s, which is where the counted optimism is.

Regret on forty fresh rows, on the draws the hold-out was scored on #

Regret is scored on forty fresh rows that continue each drawn series, over three hundred draws at each persistence on the seed the earlier essay used, so the hold-out, Akaike’s and Schwarz’s rows of the regret figure are the earlier essay’s to every digit, and every rule is scored on the same draws. The fifteen candidates are every non-empty subset of four predictors with coefficients 0.30, 0.20, 0.12 and 0. The optimism is counted separately, over three thousand draws at each persistence, for the candidate holding the first predictor and the candidate holding all four, each in units of its own error variance, the variance of what the candidate leaves out added to the noise.

The inflation formula is exact for the variance of a coefficient on one first-order autoregressive column against first-order autoregressive errors, in the limit of a long series. The four predictors here are independent of each other, so each coefficient’s inflation is its own column’s.

A charge nearly right on the optimism that still selects worse below 0.8 #

The claims are that charging each coefficient twice its variance inflation charges the counted optimism to within a tenth up to persistence 0.7, and twice the optimism at 0.95; that the criterion with that charge is the best of five rules from 0.8 up, giving up a quarter of Akaike’s regret at 0.95 given the persistences and half of it reading them; that reading them loses the rest because the residuals’ persistence is read low where the inflation is steepest; and that at 0.5 and 0.7 the charge, though nearly right, selects worse than Akaike’s.

Not claimed: anything about reference distributions, which the resampling half of the field priced for the same two defects and which a penalty does not touch; anything about persistence that is not first-order — a dependence with a shape would need its own inflation — or about predictors correlated with each other, or about sample sizes other than a hundred and twenty.

Still open: the charge the sample can afford #

The counted optimism and the inflation agree up to 0.7 and part after it, and the parting has a cause the sample size controls: a series of $n$ rows cannot be optimistic about movements slower than $n$ rows. A charge that knew this — the inflation of a coefficient computed for a series of $n$ rows rather than an infinite one, which is a finite sum rather than a ratio — would match the counted optimism at every persistence. Whether that charge, read off the sample with the residuals’ persistence corrected for the fit, recovers the oracle’s regret at 0.95 and stops costing anything at 0.5, is the measurement that would say whether a penalty for dependent rows is a method or a diagnosis.