What each rule gives up against an oracle that is arithmetic

Expected squared error of the candidate each rule selects, minus the expected squared error of the best candidate in the table, over 500 draws of 120 rows. Both quantities are closed forms — σ_S²(1 + q/(n − q − 1)) — so the only Monte Carlo here is over which candidate got picked. The hold-out spends half its rows measuring what the criterion computes, and pays 1.8 times as much for it. Schwarz's criterion is worst because it is answering a different question: which candidate contains the truth, rather than which one forecasts best.

The hold-out's regret as the sample is divided, against the criterion's, which has no division to make. Too few origins and the average of 15 squared errors is mostly noise; too many and the window the candidates are ranked at has nothing to do with the 120 rows the winner is refitted on. The minimum of the two is at P = 60 and it is still 1.8 times the flat line. The sweep was built to find the split at which a hold-out becomes worth its cost, and on independent rows there is not one.

The 95% point of the statistic's own distribution in each world, against the mean 95% point of five reference distributions built from one sample. Where the error variance is a function of the design, the two resamplings that detach a residual from its row fall short and the two multipliers that keep it there do not; where the rows repeat each other it is the other way round. With both defects at once the blocked multiplier — drawn once per run of 5 rows, so the residual never moves and its neighbours share a sign — is the closest of the five, at 2.999 against a truth of 3.803. It is still short by 0.804, and that shortfall is the next figure.