The table decides when the sample settles it

A table of candidate regressions on dependent errors is compared after whitening, and the whitening has a tuning value — a window width or an autoregressive order — read off the residuals. Read once from the fullest candidate, it is shared; read per candidate, the candidates can disagree about it, and a disagreement can change which candidate the table selects. A rate times a size decomposed what the per-candidate rule costs into how often the candidates disagree and what a disagreement costs, and the field went on to find the quantity that carries the cost: how often the tuning list changes the winner. At 120 rows it was about an eighth, across list lengths, tables and laws.

What a longer sample settles found that eighth to be a fact about 120 rows. On every subset of four predictors, the window’s list changes the winner on 12.33% of draws at 120 rows, 2.58% at 240, 1.00% at 480 and 0.25% at 960; the order’s on 10.58%, 6.25% and 2.33% at 120, 240 and 480. The fall was carried by the share of disagreements that change the winner — the turnover — while the disagreement rate itself stayed put. And it named the obvious caveat. Everything there was one table: fifteen candidates spanning one to four predictors, whose criteria are far apart because their dimensions differ. The eighth that was not a constant had found a nested table’s turnover lower at 120 rows, so the prediction was that a nested table reaches a small product sooner, with the rate again not the factor that moves.

Half of that prediction is right, and the measurement on a third table says why the other half was the wrong question.

Three tables at four sample sizes #

The construction is the field’s: a predictive regression on four persistent predictors with coefficients 0.30, 0.20, 0.12 and zero, errors from a first-order autoregression at 0.8, the full tuning list for each parameter — eight window widths from none to thirty lags, thirteen orders from zero to twelve — and twelve hundred draws at each sample size, scored on an independent replicate. Three tables of candidates are run on it.

Every subset of four is the field’s table: fifteen candidates, from one predictor to four. The nested sequence is four candidates — the first predictor, the first two, the first three, all four — each differing from the next by one coefficient. Six subsets of two is every pair of the four predictors, so every candidate has the same dimension and they differ only in which two predictors they hold. The data are the same draws for every table; only the list of candidates the table is choosing among changes.

The nested table does what the prediction said. Its window changes the winner on 5.83% of draws at 120 rows — under half the full table’s 12.33% — and on 1.25% at 240, where the full table is still at 2.58%. By 480 rows the two are close, 0.67% and 1.00%, and at 960 they are level at 0.25%. Its order changes the winner on 5.83%, 4.00% and 2.25% at 120, 240 and 480, against the full table’s 10.58%, 6.25% and 2.33%. A nested table reaches a small product about one sample size sooner, and converges with the full table by four times the sample.

The table of equal-sized candidates does not settle. Its window changes the winner on 11.75% of draws at 120 rows, about what the full table does, and then on 3.75% at 240, 5.75% at 480 and 2.33% at 960 — nine times the full table’s 0.25% at the largest size, and above the full table’s figure at 240. Its order changes the winner on 9.42%, 6.67% and 4.42% at 120, 240 and 480, falling at half the full table’s rate. Quadrupling the sample takes the full table’s window from an eighth to a hundredth and takes the equal table’s from an eighth to a seventeenth.

So the sample size does not settle the per-candidate rule. It settles it for tables whose candidates come apart as the sample grows, and the table decides whether they do.

The rate, again, is not what moves #

The disagreement rate behaves on all three tables the way it behaved on one. The order’s barely moves with the sample: 31.25%, 28.83% and 30.42% on the nested table, 44.08%, 43.83% and 41.08% on the equal one, beside the full table’s 48.9%, 47.5% and 46.1%. The level depends on how many candidates there are to disagree — four, six or fifteen — and the sample size leaves it alone.

The window’s rate jumps, and it jumps the same way on every table: high at 120 and 480 rows, low at 240 and 960. On the full table it is 43.3%, 17.5%, 38.3% and 11.6%; on the nested, 22.6%, 8.4%, 19.5% and 4.8%; on the equal, 39.4%, 15.5%, 35.0% and 10.3%. Three tables sharing a zigzag is the evidence that the zigzag belongs to the list rather than the candidates. Two values the criterion cannot tell apart found the window’s disagreement concentrated on pairs of widths the criterion treats as live together, and which widths are live depends on the sample size: at some sizes two adjacent widths on the window’s geometric sequence straddle the best band and the candidates split between them, at others one width sits alone near it. That is a property of the sequence and the sample, which is why it is the same on any table.

The turnover is what the table decides #

The turnover is where the tables part. On the full table it falls from 28.5% at 120 rows to 2.6% at 480, the factor of eleven the earlier essay reported. On the nested table it falls from 25.8% to 3.4% over the same range — the same fall from a slightly lower start, which with its lower rate is why its product starts at half. On the equal table it falls from 29.8% to 16.4% at 480 and is 22.6% at 960: a disagreement among six candidates of the same size changes the winner about as often at 960 rows as at 240.

The reading this supports is about what separates candidates. On the full and nested tables the candidates differ in dimension, and a criterion separates them by an amount that grows with the sample: a candidate missing a real predictor falls behind by a margin proportional to the number of rows, and a candidate carrying a useless one is charged for it. A change of whitening moves every candidate’s criterion by about the same amount, so as the margins grow the change of whitening stops reaching across them, and a disagreement about the window leaves the winner where it was.

On the equal table every candidate has two predictors and the same charge, and the margins between them are the differences in fit between pairs of predictors — the pair holding the 0.30 and the 0.20 against the pair holding the 0.30 and the 0.12. Those margins grow with the sample too, but the candidates nearest the winner are near it because their fits are close, and a whitening that reweights the rows can move which pair fits best. The measurement is consistent with that reading and does not prove it: what it shows is that the share of disagreements that matter stays between a sixth and a third at every size on this table, while it falls by an order of magnitude on the other two.

What it costs #

The regret follows the product. On the nested table the window’s per-candidate cost is 0.00056 at 120 rows, 0.00010 at 240 and at or a hair below zero from 480 on; the order’s is 0.00060, 0.00030 and 0.00006. That is a table on which the choice between a shared and a per-candidate whitening has stopped mattering by 240 rows. On the equal table the window’s cost is 0.00364 at 120 rows and still 0.00097 at 960, three and a third standard errors from zero, while its order’s cost falls to nothing by 240 rows. So the equal table’s persistent winner changes are not free: through the window they still cost about a thousandth of prediction error at the largest sample measured.

The order on the equal table is worth a sentence because it goes the other way. Its turnover falls, from 21.4% to 10.8%, and its cost is within a standard error of zero at 240 and 480 rows even though its product is 6.67% and 4.42%. The order’s quarrels on this table change the winner between candidates whose replicate errors are almost the same, which is the equal table’s whole character: when every candidate is nearly as good as every other, changing between them is frequent and cheap. The window’s are the exception because the window’s quarrels are between whitenings that differ more — the geometric sequence’s adjacent widths double — and a quarrel’s cost is its width measured how that distance enters on the order’s list.

What this changes about the field’s result #

The earlier essay’s headline was that a longer sample settles the per-candidate rule, carried by the turnover. That stays true of the table it was measured on, and of the nested table, which gets there sooner. It is not true of every table, and the table on which it fails is not exotic: a comparison among models of the same size — which pair of predictors, which three of five — is the commonest shape a model comparison takes in practice, and it is the shape whose candidates a longer sample does not separate from the whitening.

How often it matters had found, at 120 rows, the product running from 17.6% to 1.5% as the candidates were moved further apart. The sample size here and the spacing there are two routes to the same quantity: how far apart the candidates’ criteria are relative to how far a change of whitening moves them. A longer sample raises the first on a table of candidates of different sizes and leaves it level on a table of candidates of one size.

What a comparison of equal-sized models should do #

The practical reading is narrower than the field’s earlier one and more useful for it. The earlier essay could say that at a few hundred rows the choice between sharing a whitening and tuning it per candidate hardly matters. On a table of candidates that differ in size that remains the advice, and the nested table makes it stronger: four candidates differing by one coefficient each are past the point where it matters by 240 rows.

On a table of candidates of one size it is not the advice. The window tuned per candidate is still changing the winner on one draw in forty-three at 960 rows and still costing about a thousandth of prediction error, three standard errors from nothing, where a shared window costs exactly that thousandth less by construction. The order tuned per candidate changes the winner more often still — 4.42% at 480 rows — and costs nothing measurable, because the candidates it moves between are nearly equally good. So for a comparison among models of one size the measurement supports sharing the window, and is indifferent about the order: the order’s disagreements are frequent and free, the window’s are rarer and are not.

That asymmetry between the two parameters has the same source as the quarrel’s width. The order’s list is a sequence of correct whitenings ordered by how many noisy coefficients they carry, and a quarrel between two of them is a quarrel between two nearly identical fits. The window’s list is a sequence of widths that double, and two adjacent widths are different estimates of the dependence; a winner chosen through one is chosen through a different picture of the errors than a winner chosen through the other. When the candidates are far apart, neither matters. When they are close, the order’s closeness to itself keeps its quarrels cheap and the window’s distance from itself does not.

None of this needs the truth to act on. Whether a table’s candidates are of one size is visible before any data are drawn, and that is the one fact the measurement says decides whether a longer sample will settle the per-candidate rule.

How the numbers were made #

Twelve hundred draws at each of four sample sizes for the window and three for the order, on each table, every table on the same draws; the per-candidate and shared rules run on each draw exactly as the field defines them, the candidates scored through their whitening without a charge for the tuning value, and regret measured on an independent replicate of the same process. The decomposition’s three quantities are counted per draw: whether any candidate’s tuning value differs from the shared one, whether the selected candidate differs, and the paired regret, which is exactly zero on every draw where nothing differs. The full table’s figures are the earlier essay’s run, unchanged; the nested and equal tables are new runs on the same seeds.

Checked on the measurement: at 960 rows the equal table’s window changes the winner more than four times as often as the full table’s, and at 240 rows the nested table’s less often than the full table’s; and the equal table’s window turnover at 960 rows is more than three times the full table’s at 480.

Still open: what separates candidates of one size #

The reading above says that candidates of one size are separated only by their fits, and that a whitening moves those fits by as much as a longer sample separates them. That is testable directly: on the equal table, at each sample size, the criterion margin between the winner and the runner-up under the shared rule, against how far a change of whitening moves the same margin. If the reading is right the ratio stays level across sample sizes on the equal table and grows on the other two. A second open question is whether charging the tuning value — the charge the volume a whitening moves found the order’s criterion needs once it is per candidate — separates the equal table’s candidates where the sample does not. Neither is measured here.

What is claimed here, and what is not #

This essay takes the field’s decomposition at four sample sizes on three tables of candidates. The claims are that the nested table’s tuning list changes the winner on 5.83% of draws at 120 rows for both parameters, falls to 1.25% for the window at 240 rows and meets the full table at 0.25% by 960; that the six equal-sized candidates’ window changes the winner on 11.75%, 3.75%, 5.75% and 2.33% of draws at 120, 240, 480 and 960 rows, nine times the full table’s figure at the largest size, and its order on 9.42%, 6.67% and 4.42%; that the disagreement rate does not carry the difference on any table — the order’s is level with the sample and the window’s zigzags the same way on all three; that the turnover does, falling by an order of magnitude on the full and nested tables and staying between 16% and 30% on the equal one; and that the equal table’s window still costs 0.00097 of regret at 960 rows while the nested table’s costs nothing measurable from 480.

What stays out, named as a decision: other laws, on which every earlier table moved; tables of other sizes and shapes beyond the three; and the mechanism proposed for the equal table, deferred above as the measurement that would test it.

The boundary against what a longer sample settles is that it measured the decomposition against the sample size on one table and this essay asks whether the table decides the answer. Against how often it matters, which moved how far apart the candidates are at a fixed size, this essay holds the spacing to what each table’s construction gives and moves the size.