A criterion is a prediction of the hold-out

A specification search costs data. The field that prices one scores every candidate on a rolling hold-out: a window of rows fits each model, the rows after it are forecast, and the candidate with the smallest average squared error wins. Every row spent scoring is a row not spent fitting, and on a sample of a hundred and twenty rows the usual split gives away sixty of them.

There is a standard way of not paying, and it is old enough that most people have stopped asking what it does. An information criterion scores each candidate from the fit alone: the residual sum of squares, plus a penalty that grows with the number of coefficients. No rows are held back. The question this essay asks is what that substitution actually buys, and the answer needs an oracle — some way of saying which candidate a rule should have chosen — that is not itself an estimate.

The oracle is arithmetic #

Four predictors, coefficients 0.3, 0.2, 0.12 and 0. Fifteen candidate regressions, one for each non-empty subset. A candidate that leaves out some of the predictors is not wrong so much as incomplete: its errors in population have variance σ² plus the squared coefficients it omitted, which is a number anybody can write down.

And a regression with q coefficients fitted on w rows of Gaussian predictors, used on a fresh row, has expected squared error

ρ(S, w) = σ_S² · ( 1 + q / (w − q − 1) ), with σ_S² = σ² + $\sum_{j \notin S} \beta_j^2$

exactly. Not asymptotically and not to leading order: the inflation factor is the expectation of x′(X′X)⁻¹x over the design, and for standard normal predictors that expectation is q/(w − q − 1) with no remainder. So every candidate has a risk that can be computed, the best candidate is the argmin of fifteen such numbers, and what a selection rule costs is the difference between the risk of what it chose and the risk of the best available — computed rather than estimated, on every draw.

At a hundred and twenty rows the best candidate here fits 4 coefficients, at a risk of 1.034783. Neither rule is ever told that.

The site’s habit is a closed form beside every simulation, and this closed form has a second route. Refit every candidate on the same rows and score it on sixty fresh ones from the same world: the average of that is an estimate of the same ρ. The two orderings of the fifteen candidates disagree by at most 0.00247, against a standard error of 0.01067 on each estimate — which is the check, and it is what licenses using the closed form as the oracle for the rest of this field.

What sixty rows cost before any selection happens #

The comparison is usually framed as one of scoring accuracy, and there is a cost that arrives before any candidate is scored at all.

A hold-out fits each candidate on the rows before the window and scores it on the rows after. On a hundred and twenty rows split in half, every candidate is fitted on sixty, so every coefficient carries a standard error $\sqrt{2}$ times what the full sample would give — 41% larger, on every coefficient of every candidate.

A criterion fits on all hundred and twenty.

So the two rules are not two ways of choosing among the same fifteen models. They are two ways of choosing among fifteen models fitted at two different precisions, and the hold-out is choosing among the worse-fitted set.

And why the oracle being exact matters more than it sounds #

The design note in the figure’s caption — that the oracle is a closed form, so the only thing simulated is which candidate was selected — has a consequence worth spelling out.

On every draw where a rule selects the oracle’s own candidate, its regret is exactly zero and contributes nothing to the variance. The regret’s standard error over five hundred draws is therefore driven entirely by how often a rule mis-selects and by how much the mis-selected candidates differ, not by five hundred noisy readings of a fixed quantity.

That is why five hundred draws is enough here for differences that would need many thousands if the benchmark were itself estimated. It also means the standard errors are not comparable across rules: a rule that mis-selects rarely has a regret that is small and precisely measured, and one that mis-selects often has neither.

What the penalty is estimating #

The reason a criterion works at all is worth counting rather than citing, because it is the whole reason a criterion cannot escape the thing it replaces.

A fit’s own residual mean square is optimistic. The coefficients were chosen to make it small on these rows, so it understates the error variance by exactly (n − q)/n. And that same fit used on a row it has never seen is pessimistic by exactly 1 + q/(n − q − 1), for the same reason from the other side. Both are functions of the parameter count and the sample size and of nothing else.

Akaike’s penalty of 2q is an estimate of the gap between those two lines. So a criterion is not an alternative to the displacement a hold-out measures — it is the displacement, computed instead of counted. The interesting question was never whether it has the same bias. It has the same bias by construction. The question is what it does to the variance around it.

An average of sixty squared errors is a bad estimator #

Here is the part that is easy to miss. The hold-out is unbiased for the quantity it estimates, and the criterion is not, and the criterion still wins by a factor of two.

The reason is that a squared forecast error is a terrible thing to average. If the errors are normal, a squared error has a standard deviation of √2 times its own mean, so sixty of them average to something whose standard error is about 18% of the quantity being measured. The differences the selection has to resolve — the risks of the fifteen candidates run from 1.0348 to 1.1640 — are a few per cent apart. Sixty origins is not enough resolution to tell them apart reliably, and the selection is largely noise: the hold-out picks the best candidate 15.2% of the time.

The criterion has no such noise. It computes the optimism from a theorem rather than measuring it, and what is left over is only the noise in the residual sum of squares, which is an average of a hundred and twenty squared residuals rather than sixty and is not being asked to resolve a small difference from scratch. It picks the best candidate 26.2% of the time and gives up 0.01701 against the hold-out’s 0.03200.

A theorem is a measurement with no standard error. Where its premises hold, it is not merely cheaper than measuring — it is better, and the cheapness is a side effect.

Which criterion, and what each one is trying to do #

Three penalties are worth separating, because two of them are answering the same question and one is not.

Akaike’s penalty is 2q, and it is an estimate of the optimism above, valid as the sample grows. Regret 0.01701.

The finite-sample form adds 2q(q + 1)/(n − q − 1), which is not a refinement so much as the arithmetic the first one approximates: it is the penalty whose implied inflation is 1 + q/(n − q − 1) rather than 1 + q/n. On a hundred and twenty rows the two are close and it gives up 0.01738 — inside its own noise of the first, which is what a correction of the right size looks like when the sample is large enough not to need it.

Schwarz’s penalty is q·log n, which at n = 120 is about 4.8q rather than 2q. It is not estimating ρ at all. It is estimating which candidate contains the truth, which is a different question with a different answer, and it is here so that the difference is a measurement rather than a remark. It gives up 0.03137 — as much as the hold-out — and it chooses the smallest candidate in the table 46.2% of the time, against the finite-sample form’s 13.0%. On this world the smallest model is genuinely wrong, and a rule that likes small models is wrong with it.

That is not a criticism of Schwarz’s criterion. It is a statement that a criterion answers the question its penalty was derived for, and reading a consistency criterion as a prediction criterion is the same error as reading one reading of a search table as another.

The number that chose the model is not a number about the model #

The last measurement is the one with the most practical consequence, and it costs more than the entire difference between the two selection rules.

An analyst who runs a rolling hold-out ends with a table, a winner, and the winner’s own average squared forecast error. That number is very natural to quote. It is also a minimum over fifteen candidates, and it is biased downwards by exactly the amount that made it the winner.

The winner’s own hold-out average is 1.0314. Its true risk is 1.0668. The gap is 0.0354, which is 2.0 times the entire regret of the criterion it is being used to justify. Every conclusion of the form “the model was selected on out-of-sample performance and achieves 1.03” is off by more than the whole quantity the selection was optimising.

This is the winner’s curse arriving as a performance figure rather than as an effect size, and it has the same repair: the number that selected has to be separated from the number that reports. A hold-out cannot do both jobs with the same rows, which means the honest version of a hold-out procedure is a third split — and the third split is more data spent on the same question.

Fifteen candidates, and how few of them anybody can tell apart #

Both hit rates are low — 26.2% for the best criterion and 15.2% for the hold-out — and that is worth sitting with rather than passing over, because it says what kind of problem a specification search is.

The fifteen candidates have risks running from 1.0348 to 1.1640. Three of them are within a per cent of the best. On a hundred and twenty rows, a per cent of the error variance is far inside the sampling noise of anything computed from those rows, so most of the table is genuinely indistinguishable and no rule can be expected to find the argmin reliably. That is not a failure of either method; it is the resolution the sample has.

Which is why regret is the right quantity and the hit rate is not. A rule that misses the best candidate but lands on one of the three next to it has cost almost nothing; a rule that misses it for a candidate two coefficients smaller has cost a great deal. Schwarz’s criterion is the case in point: it finds the best candidate 9.0% of the time, worse than either of the others, and its regret of 0.03137 is worse in proportion — its misses are not near misses, because a heavy penalty misses in one direction. The average candidate it selects fits 2.65 coefficients against the finite-sample form’s 3.30 and against a true best of 4.

A selection rule should be judged by what its mistakes cost, not by how often it makes them, and on a table where most rows are ties those two are different measurements.

What grows and what does not, as the sample does #

The sample size moves everything in the figure, and it moves the two rules differently, which is the last thing to say before the premises come off.

Every risk in the table falls as n grows, because the inflation q/(n − q − 1) does; the gaps between candidates fall with it, so the absolute regrets shrink for both rules. That is uninteresting. What is interesting is that the ratio does not move much: the hold-out gives up about twice what the criterion does at eighty rows, at a hundred and twenty and at two hundred. The hold-out’s disadvantage is not a small-sample artefact that goes away with more data — it is structural, because at every sample size the hold-out is estimating from a fraction of the rows a quantity the criterion computes from all of them.

What does change with n is which candidate is worth having. At a hundred and twenty rows the best candidate fits four coefficients; at forty it fits three, because the third predictor’s coefficient of 0.12 is not worth the parameter it costs on a short sample. The right model is a function of how much data there is, which is obvious once written down and is the reason the phrase the true model does no work in any of this. There is no true model here in the sense a selection rule could find one. There is a best model at a stated sample size, and it changes.

What this does not say #

Three limits are worth stating plainly, because the result reads as a stronger endorsement of criteria than it is.

The world here satisfies the theorem. Independent rows, homoskedastic errors, a Gaussian design, a correctly specified linear model except for the coefficients each candidate omits. That is exactly the setting Akaike’s penalty was derived in, and a rule that uses a derivation should win where the derivation holds. The next essay takes the premise away and the ordering reverses.

The error distribution does not matter, and that is not obvious. The closed form uses E[(X′X)⁻¹] for a Gaussian design and says nothing about the shape of the errors, so ρ holds for skewed errors too, and so does the optimism the penalty corrects — it is a second-moment argument. Non-normal errors are not the premise that breaks this.

And a hold-out is doing something a criterion cannot, which is not visible here. A rolling comparison measures the actual forecasting procedure, including anything about it that no likelihood knows: a transformation, a rule for handling missing rows, a step where somebody looks at a plot. A criterion prices the model. Where the procedure and the model are the same object, as they are here, the criterion wins; where they are not, the criterion is not measuring the thing being deployed.

What is claimed here, and what is not #

This essay takes the price of two selection rules against a computable oracle, and the claims are the three regrets, the two closed forms the penalty sits between, and the size of the winner’s own optimism.

What stays out and is named as a decision: everything about when the ordering reverses, which is the next essay entire; anything about testing, since a selection has no error rate and the readings that do are the neighbouring field’s; and the construction of a reference distribution for any of it, which needs a resampling and is the third essay of this field.

The boundary against the search field is that nothing here is a test. There is no null, no critical value and no rejection rate; there is a rule that picks a model and a number saying how much worse that model is than the best one. The two fields share a table of candidates and share nothing else.

The checks, and the refusals that make them mean something #

Three claims are gated in this field’s library. The two routes to a candidate’s risk — the closed form and sixty fresh rows — are required to order the fifteen candidates the same way at two sample sizes, which fails if the oracle every other number is measured against is wrong. The two ratios in the optimism figure are required to match (n − q)/n and 1 + q/(n − q − 1) at every candidate, which is the claim that a penalty is the displacement rather than an escape from it. And the hold-out is required to be beaten at every split of the sample, which is the result and which no single split could show.

The refusal for this essay is the winner’s own hold-out average: the check computes it, computes what the winner is actually worth, and refuses the first as a report on the second. It fires at a gap of 0.0354 on a rule whose entire regret is half of that, which is the shape of every measurement in this field — the error that matters is not in the choosing, it is in what gets written down afterwards.