Where the two searches cross

The previous essay leaves an obvious question. It compared one criterion against one hold-out at one split — sixty rows to fit, sixty to score — and the criterion won by a factor of about two. A hold-out has a dial on it. Score on fifteen origins instead of sixty and the window is nearly the whole sample; score on ninety-five and the average is much less noisy. Somewhere along that dial, surely, the hold-out becomes worth what it costs.

It does not. And the sweep that shows it does not is worth running anyway, because the shape of the curve says why — and because the dial that does change the answer turns out to be one no analyst gets to set.

Two costs, and they point the same way at both ends #

Sweep the split with the sample held at a hundred and twenty rows. Each setting is the same comparison: fifteen candidates, both rules, regret against a closed-form oracle.

The hold-out’s curve is a U. At fifteen origins it gives up 0.04683; at sixty, 0.03117; at ninety-six, 0.04528. The criterion sits flat at 0.01743, because it has no split.

Both ends of the U are costs and they are different costs.

Too few origins is noise. Fifteen squared forecast errors averaged together have a standard error of about a third of what they are estimating, and the differences the selection has to resolve are a few per cent. At that end the hold-out is choosing nearly at random among the candidates that are close: it finds the best one 10.7% of the time.

Too many origins is the wrong question. Every origin taken is a row the estimation window no longer has. At ninety-six origins the window is twenty-four rows, and on twenty-four rows the best candidate is not the same candidate. The penalty a parameter carries is q/(w − q − 1), which is steeper the shorter the window, so a short window prefers a smaller model — and the winner gets refitted on all hundred and twenty rows before anybody uses it.

The second cost is the one that is easy to miss, so it is worth separating from the noise entirely.

Why sixty and sixty was already the best setting #

The sweep finds no better point on the dial, and there is a heuristic reason to expect that before it is run.

Moving the dial trades two things directly against each other. The score’s noise falls like one over the square root of the number of origins. The fit’s precision falls like one over the square root of the estimation window. And the two must sum to the sample: more origins is fewer rows to fit on.

So what a hold-out is trying to maximise is, to a first approximation, the product of the window and the number of origins — and for a fixed total, a product is maximised when the two halves are equal.

Sixty and sixty is the optimum of the trade the dial is on. Fifteen origins gives a window of a hundred and five and a product of 1,575; ninety-five origins gives a window of twenty-five and a product of 2,375; sixty and sixty gives 3,600.

The argument is crude — the two costs do not enter the regret with equal weight — and it is enough to explain the shape. The curve is flat near the middle because a product is flat near its maximum, and falls away at both ends because one factor is going to zero.

The part that has no data in it #

Which candidate is best at a window of w is a closed-form question. So is which candidate is best at a sample of n. They are not the same candidate, and the difference is computable before any experiment runs.

Below eighty rows the best candidate fits 3 coefficients rather than four, and choosing it costs 0.005852 at deployment. Below thirty it fits 2, and choosing it costs 0.037641 — which is larger than the criterion’s entire regret.

That is a bias, and averaging more origins makes it worse, because taking more origins is exactly how the window is made shorter. The hold-out’s two costs therefore trade against each other along the same dial and neither end escapes: the minimum of the U is a compromise between a noise that falls and a bias that grows, and at its best it is still 1.8 times the flat line.

The mechanism is visible in what the hold-out selects. As the split moves from twenty-four origins to ninety-six, the hold-out’s agreement with the oracle for the deployment sample falls from 15.0% to 4.3%, while its agreement with the oracle for its own window climbs from 15.0% to 37.7%. It is not getting worse at its job. It is getting better at a job nobody asked for.

The dial that does change the answer #

So the crossing is not in the split. Where is it?

The premise the criterion’s penalty rests on is that the rows are independent. The optimism of an in-sample fit is 2qσ² because each of the q coefficients is fitted using n independent pieces of information; when the rows repeat each other there are fewer independent pieces than there are rows, the fit is more optimistic than 2q says, and the penalty is too small for the fit it is correcting.

A hold-out never used that argument. It measures the optimism instead of computing it, and a measurement does not care whether the rows are independent.

Make both the predictors and the errors autoregressive at the same coefficient and sweep it. At ρ = 0 the criterion gives up 0.01927 against the hold-out’s 0.03581. At ρ = 0.85 they are 0.06657 and 0.05939 — the criterion is now behind. At ρ = 0.95, 0.12815 against 0.10866. The crossing is at ρ = 0.81.

And the criterion fails in exactly the direction the theorem predicts. Its average selected model grows from 3.31 coefficients at ρ = 0 to 3.90 at ρ = 0.95, while the true best model shrinks from four coefficients to two — because persistent rows carry less information, so the sample is effectively smaller and a smaller model is worth having. The criterion is buying coefficients at exactly the moment they get more expensive, and it is doing so because its penalty is counting rows.

It has to be said that both rules get much worse in absolute terms. At ρ = 0.95 the best rule in the table gives up 0.10470, five times the worst rule at ρ = 0. Persistence is not a problem that a better selection rule solves; it is a problem that makes selection harder for everybody, and the crossing is a change in the ordering of two bad options.

The right answer for the wrong reason #

The most instructive line in that figure is Schwarz’s.

On independent rows it is the worst of the three, at 0.03550 — as bad as the hold-out — because its penalty of q·log n is calibrated for a different question and is far too heavy for prediction. As persistence rises it becomes the best rule in the table, crossing at ρ = 0.75, and by ρ = 0.9 it is giving up 0.08495 where Akaike’s gives up 0.10754.

Nothing about it has become correct. The penalty is still answering which candidate contains the truth, and the answer to which candidate forecasts best is still a different one. What has happened is that the too-heavy penalty is now compensating for a too-small one: the optimism has grown because the rows repeat, and a penalty that was already excessive is accidentally closer to the excess.

Two errors of opposite sign is not a correct method, and the practical consequence is sharp. An analyst who observes that Schwarz’s criterion is doing better on their persistent series and concludes that consistency was the right criterion after all has drawn a conclusion that will not survive the next dataset — because the size of the compensation depends on the persistence, which is a property of the data rather than of the method, and there is no ρ at which the two errors cancel exactly.

The honest repair is the one this suggests and does not build: a penalty computed from an effective sample size rather than from n. That is a real construction and it is not in this field, because estimating an effective sample size from the same rows that are being selected on introduces a second selection problem with the same shape as the first.

Why persistence and not heteroskedasticity #

It is worth saying which premise was taken away, because two others were tried first and neither did anything.

A variance that depends on the design does not move the ordering. With the error variance made a function of the first predictor, both rules get worse and the hold-out still gives up 2.24 times what the criterion does, against 2.03 on a clean world. The optimism of an in-sample fit is a trace, and where the design’s heteroskedasticity is unrelated to which columns each candidate uses that trace is still about q — so the penalty is still the right size and nothing in the derivation notices.

Skewed errors do nothing either — the ratio is 2.15 — and the reason is sharper. The whole argument — the closed form for ρ, the optimism the penalty estimates — is a statement about second moments and about the expectation of (X′X)⁻¹ for a Gaussian design. Neither depends on the shape of the errors at all. A third moment has to reach the criterion through some route, and there is no route.

Errors correlated in time, with independent predictors, do almost nothing: 2.06. That one is more surprising and the mechanism is worth a sentence — the optimism is a trace of the hat matrix against the error covariance, and where the design is a fresh draw of independent rows the hat matrix is not aligned with the errors’ low-frequency structure, so the trace comes out near qσ² anyway. Persistent predictors with independent errors are the mirror image and do no more: 2.13.

It takes both — a persistent design and persistent errors — for the fit to be able to spend its coefficients tracking the errors’ slow movements, and that is when the optimism exceeds what the parameter count says. This is the same pairing that the resampling half of this field runs into, and that is not a coincidence: an optimism theorem counts independent rows and a bootstrap counts independent draws, and the same dependence takes both counts away.

The estimation window is not the only thing that shrinks #

One more consequence of the U’s right-hand end is worth drawing out, because it applies to any procedure with a hold-out in it and not only to this one.

Everybody knows a hold-out costs rows. What is less often said is that it costs them from the fit, and a fitted model’s risk depends on the window through q/(w − q − 1) — a term that is not linear. Going from a hundred and twenty rows to sixty does not double the inflation on a four-coefficient model; it takes it from 0.0348 to 0.0727, slightly more than double, and on a six-coefficient model from 0.0531 to 0.1132. The larger the candidate, the worse a short window treats it, which is precisely why a short window reorders the table rather than merely blurring it.

So the reordering is not a nuisance to be tolerated. It is a systematic preference for small models, introduced by the act of holding rows back, of a size that can be computed in advance. On this table it is worth 0.005852 at sixty rows and 0.037641 at twenty, and neither number involves any data.

The comparison that would make a hold-out look best — a very large sample, where the window cost is negligible and the averaging noise is small — is also the comparison in which the criterion’s penalty is most accurate. There is no regime in the independent-row world in which the hold-out’s two costs are simultaneously small and the criterion’s bias is not.

What an analyst actually has to decide #

The sweep gives three usable statements and it is worth separating them from the numbers.

The split is not the interesting choice. Whatever the answer is, it is not hold out a bit more or hold out a bit less — the curve is flat enough near its minimum and far enough above the alternative that moving the split cannot rescue the method. Time spent choosing a split is time spent on the wrong parameter.

The interesting choice is whether the theorem applies. Independent rows, and a criterion. Rows that repeat each other, and a hold-out — or a criterion with a penalty built for dependence, which is a different object from the one anybody’s software prints by default.

And how much the rows repeat is measurable before any selection happens. Nothing in this field requires the crossing point to be known in advance. What it requires is that somebody looks at the autocorrelation of the residuals before choosing which rule to trust, and that is one line of output that most specification searches never print.

A dial nobody sets #

There is one asymmetry between the two sweeps that is worth stating on its own, because it is what makes the second one useful and the first one not.

The split is a decision. An analyst chooses it, can change it, and can be advised about it. The sweep says that whatever the choice, this method is behind — so the advice is not about the split.

The persistence is a property of the data. Nobody chooses it, and it is not a defect to be repaired — a persistent series is what most series in economics, climate and epidemiology look like. So the second sweep is not advice about a parameter; it is a boundary on where a whole family of methods applies, and the boundary happens to be sharp enough to name: the two rules change places at ρ = 0.81 on this table, and Schwarz’s overtakes Akaike’s at 0.75.

Neither number is a constant of nature. Both depend on the table, the sample size and the coefficients, and a different specification search will cross somewhere else. What does not depend on any of that is the direction: the more a row repeats the one before it, the worse a penalty built by counting rows performs, and the better a method that measures instead of counting.

What is claimed here, and what is not #

This essay takes where the ordering between a criterion and a hold-out changes, and the claims are the U-shaped curve with no crossing in it, the closed-form window cost that no averaging removes, the crossing in the persistence at 0.81, and the direction the criterion fails in.

What stays out and is named as a decision: a penalty built for dependent rows, which is named above and not built because estimating an effective sample size on the rows being selected on is a second selection problem; anything about reference distributions or rejection rates, since nothing here is a test; and the resampling that a dependent world needs, which is the next essay and which fails for the same reason the criterion does.

The boundary against the order-selection field is that this one compares two methods on a fixed table of candidates, where that one compares candidates that differ in how many rows they can use at all. The shared-sample trap it measures is a precondition for anything here to be well defined.

The checks, and the refusals that make them mean something #

Two claims are gated in this field’s library. The hold-out is required to be beaten at every split measured and to be worse at both ends of the split than in the middle, which is the two-costs result and which no single split could show. And the ordering is required to reverse in the persistence, with the criterion’s average selected model growing as the true best one shrinks — which fails if the crossing were being explained by anything other than a penalty counting rows.

The refusal for this essay is a criterion compared across candidates fitted on different numbers of rows. Written as −2 log L rather than as the n·log(RSS/n) shorthand it is decisive: the constant n(1 + log 2π) cancels only when the sample is shared, and dropping twelve rows from one candidate hands it the comparison 100.0% of the time on samples where the same two candidates compared on the same rows go the other way 82.8% of the time. The criterion is reading a sample size, not a fit.