Normal Distribution
nothing is called 95% until it has been counted
Field

Scoring a search without spending data

A hold-out is expensive: every row spent scoring is a row not spent fitting. There are two standard ways of not paying — an information criterion, which predicts the hold-out from the in-sample numbers, and a resampling, which builds a reference distribution from one sample. Neither avoids what it replaces. The criterion's penalty *is* the displacement, computed rather than counted, and it beats a rolling hold-out at every split there is; both it and the resampling are undone by the same defect, which is rows that repeat each other, because both are counting independent things and there are fewer of those than there are rows.
What each rule gives up against an oracle that is arithmetic. Expected squared error of the candidate each rule selects, minus the expected squared error of the best candidate in the table, over 500 draws of 120 rows. Both quantities are closed forms — σ_S²(1 + q/(n − q − 1)) — so the only Monte Carlo here is over which candidate got picked. The hold-out spends half its rows measuring what the criterion computes, and pays 1.8 times as much for it. Schwarz's criterion is worst because it is answering a different question: which candidate contains the truth, rather than which one forecasts best.

A criterion is a prediction of the hold-out

A rolling hold-out spends half the sample measuring what a criterion computes from all of it. Against an oracle that is arithmetic rather than an estimate, the criterion gives up 0.01701 and the hold-out 0.03200 — and the number the hold-out reports for its own winner is optimistic by more than either.

 The crossing is in the dependence, not in the split. Regret of each rule as the design and the errors are made persistent at the same coefficient, scored on fresh rows because the closed form assumes exactly what is being taken away. An optimism theorem counts rows; when the rows repeat each other there are fewer of them than there are rows, the penalty is too small for the fit it is correcting, and the criterion starts buying coefficients it should not — its average winner grows from 3.31 coefficients to 3.90. The hold-out never used the theorem and overtakes at ρ ≈ 0.81. Schwarz's criterion, worst of the three on independent rows, is best on repeating ones — its heavier penalty is right for the wrong reason.

Where the two searches cross

The obvious dial between a criterion and a hold-out is how much of the sample to hold out, and moving it never changes the answer. The dial that does is one nobody chooses — how much each row repeats the one before it — and the two rules change places at about 0.81.

 Each repair is for its own defect, and one is for both. The 95% point of the statistic's own distribution in each world, against the mean 95% point of five reference distributions built from one sample. Where the error variance is a function of the design, the two resamplings that detach a residual from its row fall short and the two multipliers that keep it there do not; where the rows repeat each other it is the other way round. With both defects at once the blocked multiplier — drawn once per run of 5 rows, so the residual never moves and its neighbours share a sign — is the closest of the five, at 2.999 against a truth of 3.803. It is still short by 0.804, and that shortfall is the next figure.

Two defects and one resampling

Four resamplings, each the repair for one defect and wrong about the other. Put both defects in the same world and the statistic's 5% point is 3.8028, where the best of the four reaches 2.8326 — until a multiplier that stays on its own row and shares a sign with its neighbours reaches 2.9988.

 A bias against a variance, with the answer in between. How wrong one sample's reference distribution is, split into the two things it is wrong by. Sharing the multiplier over more rows keeps more of the dependence and closes the bias from 1.688 to 0.835; every row it is shared over also removes an independent sign from the 101 the sample started with, and the spread of the resulting quantile rises from 1.307 to 2.172. The distance a practitioner with one sample is actually exposed to is the two together, and it is smallest at ℓ = 5.

How long a block a multiplier shares

Sharing a sign over more rows keeps more of the dependence and leaves fewer independent signs to build a distribution from. The bias falls from 1.6885 to 0.8479 and the spread rises from 1.3073 to 2.1716, and the rejection rate walks straight through its nominal level on the way from 11.3% to 1.3%.

 A penalty that counts independent rows. Regret of five selection rules over fifteen candidates as the design and the errors are made persistent at the same coefficient, 300 draws at each of 7 settings, scored on fresh rows. Akaike's criterion gives up 0.0193, 0.0196, 0.0294, 0.0428, 0.0666, 0.1075, 0.1281; charging each coefficient its own variance inflation, with the persistences read off the sample, gives up 0.0188, 0.0231, 0.0339, 0.0375, 0.0496, 0.0656, 0.0652, and with them given 0.0193, 0.0236, 0.0348, 0.0374, 0.0426, 0.0469, 0.0318. The hold-out gives up 0.0358, 0.0365, 0.0393, 0.0453, 0.0594, 0.0913, 0.1087 and Schwarz's criterion 0.0355, 0.0295, 0.0331, 0.0397, 0.0575, 0.0849, 0.1047.

A penalty that counts independent rows

Akaike's penalty charges a fit two error variances a coefficient because each coefficient is fitted from n independent rows. When the design and the errors both persist, a row is worth less than a row, and the honest charge is twice the coefficient's variance inflation. On a hundred and twenty rows at persistence 0.7 that charge is 34.7 against an optimism counted at 34.5, where Akaike's charges 10. Charged it, the criterion's regret at 0.95 falls from 0.128 to 0.032 with the persistences given and to 0.065 when they are read off the sample — and at 0.5, where the charge is right and Akaike's is half of it, Akaike's still selects better.

 Three charges and two counts. The optimism of an in-sample fit of the widest candidate on 120 rows, in units of its own error variance, over 1500 draws, by persistence. Counted on the 40 rows straight after the sample: 11.1, 21.5, 35.6, 51.5, 65.5, 87.8, 119.1. Counted on rows 600 further on: 9.3, 18.3, 33.0, 50.6, 67.0, 96.1, 159.0. Charged by the long-series inflation: 10.0, 19.3, 34.7, 54.4, 74.3, 114.2, 234.1. Charged by the same inflation summed over the sample's rows: 10.0, 19.2, 34.2, 53.1, 71.8, 108.2, 208.8. Charged by twice the trace of the hat matrix against the errors' correlation on the sample's own design: 10.0, 18.7, 32.0, 47.5, 61.5, 85.2, 132.2.

A charge the sample can afford

The inflation charge for dependent rows runs to twice the counted optimism at persistence 0.95, and the reason offered was that a series of n rows cannot be optimistic about movements slower than n rows. Summing the inflation over n rows does almost nothing: 234 becomes 209, against 119 counted. Evaluating the same variance on the sample's own design gives 132, and moving the fresh rows away from the sample raises the count to 159. So the charge can be made nearly right, and when it is, the regret doubles from 0.032 to 0.061. A selection wants a much larger charge than the optimism, and past about fifty per predictor it is no longer selecting.

All essays