What a longer sample settles

How often it matters found a number that refused to move. A table of fifteen candidate regressions, each choosing its own whitening from its own residuals before the table is compared, picks a different winner than a table that chooses one whitening for all on about an eighth of draws. Lengthen the list of tuning values and the candidates disagree more often while a smaller share of their disagreements decide anything, and the product — how often the list changes the winner — stays where it was. It stayed there across tables and laws too.

The essay’s last section named three things that might move it, and the first it could name was the most ordinary: the sample size. Every measurement in the field was at a hundred and twenty rows. A longer sample estimates the dependence better, so the fifteen candidates’ tuning choices should converge and disagree less; and it separates the candidates’ criteria, so a disagreement about the tuning value should change the winner less often. Both would lower the product. Which one does it was not measured, and the two guesses predict different things a practitioner could check: if the candidates agree more, the observable disagreement rate falls; if their disagreements stop mattering, it need not.

The decomposition at four sample sizes #

The construction is unchanged: a true null, fifteen candidates from every non-empty subset of four predictors, errors from a first-order autoregression at 0.8, the full tuning list for each parameter — eight window widths, thirteen sieve orders — and twelve hundred draws at each size. For every draw the field’s three quantities are recorded: whether the fifteen candidates disagree about the tuning value, whether a disagreement changes the candidate the table selects, and the regret of choosing per candidate rather than once for the table.

For the window, how often the list changes the winner is 12.33% at 120 rows, 2.58% at 240, 1.00% at 480 and 0.25% at 960. For the order it is 10.58%, 6.25% and 2.33% at 120, 240 and 480. The field’s eighth is a property of a hundred and twenty rows. At four times the sample the window’s figure is a twelfth of what it was and the order’s under a quarter.

The turnover carries it #

The field already had both factors in hand at a hundred and twenty rows. A list is not a rule measured the rate against the list’s length, and the quarrel that changes the winner showed that a disagreement costs something only when it reorders the table. What neither could say is which of the two a longer sample would move, and the answer is not symmetric.

The two factors behave completely differently.

The share of disagreements that change the winner — the turnover — falls steadily for both parameters. For the window it is 28.5% at 120 rows, 14.8% at 240, 2.6% at 480 and 2.2% at 960; for the order 21.6%, 13.2% and 5.1%. Between 120 and 480 rows the window’s turnover falls by a factor of eleven and the order’s by a factor of four.

The disagreement rate does not. For the order it is 48.9%, 47.5% and 46.1% at 120, 240 and 480 rows — flat within its own noise. For the window it is 43.3%, 17.5%, 38.3% and 11.6% across the four sizes: it does not fall steadily, it jumps.

So the guess that a longer sample makes the candidates agree about the tuning value is wrong, for the order outright and for the window in its usual form. A longer sample makes their disagreements harmless. The fifteen candidates still pick different whitenings about as often as before; what changes is that by four hundred and eighty rows the difference in whitening almost never reorders the table.

Why the rate does not settle #

Two values the criterion cannot tell apart found what sets the disagreement rate at a hundred and twenty rows. It is not how many values the list has; it is how many pairs of values on it the tuning criterion is nearly indifferent between. Adjacent sieve orders past the truth tie at about the same rate pair after pair, because each extra lag faces the same penalty and gains a likelihood that is roughly chi-squared on one degree of freedom. That argument does not depend on the sample size at all — a useless lag gains a chi-squared amount at any $n$ — and the order’s flat rate is what it predicts.

The window’s rate jumps for the same reason, read through a list that is not evenly spread. At a hundred and twenty rows two widths, four and eight lags, are nearly tied for the fullest candidate, and the candidates disagree whenever the list holds both. As the sample grows the criterion’s preferred width moves, and which pairs of widths on the list are nearly tied moves with it: at some sizes the preferred width sits between two list values and the candidates split between them, at others it sits on one and they agree. The rate is a property of where the criterion’s flat region falls relative to the list, and the sample moves that region along a list whose values are not evenly placed.

What does settle with the sample is the thing the turnover measures. A disagreement about the whitening changes the winner only if the fifteen candidates’ criteria are close enough that a slightly different whitening can reorder them. The expected reason it falls is that with more rows the candidates’ criteria spread further apart while a change of whitening moves each of them by less, which is the mechanism the turnover field measured directly on another dial; here only its consequence is measured, not the two spreads.

Where the window’s quarrel moves #

The window’s jumping rate has a visible trace in what the candidates quarrel over. On the draws where they disagree, the largest and smallest widths they choose are 4.14 lags apart on average at 120 rows, 4.08 at 240 and 4.12 at 480: on almost every disagreement the candidates are one step of four lags apart on the list — four against eight, or eight against twelve. At 960 rows the gap is 7.97 lags. The quarrel has moved up the list, to a pair of wider values eight lags apart, because the criterion’s preferred width has grown with the sample.

That is the moving flat region the previous section described, read directly. Between 240 and 480 rows the preferred width sits where four and eight are again nearly tied and the rate is high; by 960 it has moved past eight, towards a pair of values the list spaces more widely, and the candidates split less often. The order’s list is evenly spaced, its quarrels are between orders three apart on average — 2.99, 3.15 and 3.07 — and its rate does not jump.

None of this reaches the regret. Whether the candidates quarrel over four against eight or twelve against twenty, by 480 rows the quarrel almost never decides which candidate wins.

What tuning per candidate now costs #

The regret follows the product down. For the window it is 0.00368 ± 0.00051 at 120 rows, 0.00112 ± 0.00029 at 240 and 0.00016 ± 0.00010 at 480 — inside two standard errors of zero — and at 960 rows it is indistinguishable from nothing. For the order it is 0.00312 ± 0.00049, 0.00161 ± 0.00030 and 0.00011 ± 0.00006.

So the field’s practical conclusion — that tuning per candidate costs about four thousandths of regret whatever list is used — is a conclusion about short samples. By four hundred and eighty rows the cost is a twentieth of that and at the edge of what twelve hundred draws can see, and a practitioner with a sample of that length can tune per candidate or once for the table and get the same answer to within the noise.

The price stays and the occasions vanish #

The regret is a rate times a size, and the size has its own story.

On the draws where the winner did change, the regret was 0.0314 for the window at 120 rows, on 148 draws, and 0.0432 at 240 rows, on 31. At 480 rows it was 0.0174 on 12 draws and at 960 rows 0.0002 on 3. For the order: 0.0292 on 127 draws at 120 rows, 0.0255 on 75 at 240, 0.0051 on 28 at 480.

Read as a whole, a changed winner costs about the same — two to four hundredths — while there are enough of them to measure, and both its size and its frequency fall away together at the largest samples. The longer sample does not make a changed winner cheaper so much as make it rare, and the few that remain at 480 rows are the near-ties between candidates whose risks are close anyway, so changing between them costs little.

What a practitioner can see on one sample #

How often it matters pointed out that the disagreement rate is observable on a single sample: fit the candidates, let each choose its whitening, and see whether the choices agree. At a hundred and twenty rows that was a useful warning, because a disagreement changed the winner more than a fifth of the time. This essay says it stops being one. At 480 rows the candidates disagree on almost half of samples for the order and more than a third for the window, and a disagreement changes the winner on one in twenty or fewer.

The quantity a practitioner should read instead is just as observable and more to the point: run the table both ways, with one whitening and with a whitening per candidate, and see whether the winner changes. That is two comparisons of fifteen criteria, no simulation and no law, and it answers the question the disagreement rate was standing in for. On these designs at 480 rows it would say no on all but one or two samples in a hundred, whatever the candidates’ tuning choices look like.

Not the same as separating the candidates #

The turnover field turned a different dial: it scaled the predictors’ coefficients so that the candidates’ true risks moved apart, at a fixed hundred and twenty rows. There the turnover fell, as here, from 51.6% to 1.7% — and the disagreement rate rose, from 31.8% to 88.8%, because candidates that differ more leave residuals that differ more and choose more different whitenings.

The sample size does not do that. Under a true null the candidates’ residuals all contain the same truth, and more rows make each residual series a better estimate of the same dependence, so the candidates’ choices do not drift apart; the order’s rate is flat and the window’s moves only with where its list’s values fall. Both dials collapse the turnover; only one of them makes the candidates quarrel more on the way. That is a useful distinction for a reader of the field’s earlier essays: a disagreement rate that rises is evidence of candidates that differ, not of a sample that is too short.

How fast, and how far twelve hundred draws can see #

Each doubling of the sample divides the window’s product by 4.8, then 2.6, then 4.0, and the order’s by 1.7 and then 2.7. The window falls faster at first because its rate happens to drop between 120 and 240 rows as well as its turnover; the order’s falls more steadily because only its turnover moves. Over the whole range both fall by more than a factor of four per quadrupling of the sample.

The resolution of the measurement shrinks with the effect. At 960 rows the window’s list changed the winner on 3 draws out of twelve hundred, and every quantity conditioned on those draws — what a changed winner costs, for instance — rests on three numbers. The rate itself is measured only roughly there, a quarter of a per cent with a standard error more than half its size, and the conditional readings past 480 rows are illustrations rather than estimates, drawn and quoted as such.

The window at 240 rows deserves the same caution in the other direction. Its disagreement rate there, 17.5%, is the lowest of the first three sizes and drives the window’s large first drop; it rests on 210 disagreements and is not noise, but it is a feature of where the preferred width sits at that size, which the previous sections explain and which a list with different values would move.

What the field’s invariant was #

The field’s finding was that how often the list changes the winner is about an eighth and does not move with the list’s length, the table or the law. At a hundred and twenty rows that remains true. It was never a constant of the rule. It was a property of a sample short enough that fifteen candidates’ criteria sit within a whitening’s reach of one another, and the three things the field varied — the list, the table and the law — none of them change that.

The sample size does. The reading that results is simpler than the field’s and more useful: tuning a whitening per candidate matters in short samples and stops mattering in long ones, and the transition is fast — between 120 and 240 rows the window’s product falls by a factor of five. A practitioner with a few hundred observations can stop worrying about the choice; one with a hundred cannot, and for that practitioner every number in the field’s earlier essays still applies as it was measured, at the sample size it was measured at.

Still open: where the transition sits for other tables #

The candidates here are every subset of four predictors, a table whose criteria span about twenty units at 120 rows. A nested table, whose neighbours differ by one coefficient, is closer together, and the turnover field found — against its own guess — that its turnover is lower at 120 rows, because a change of whitening moves its criteria by even less than it moves the fifteen-subset table’s. The prediction from that is that the nested table reaches a small product at a shorter sample than the table here, and that the disagreement rate, again, will not be the factor that moves. Measuring it needs the same four sample sizes on the nested table and the six-subset table, and the order’s slower decline here — a factor of four to the window’s eleven — suggests the answer depends on the parameter as well.

The claim, and where it stops #

This essay takes the field’s decomposition at sample sizes from 120 to 960 rows. The claims are that how often the tuning list changes the winner falls from 12.33% to 1.00% for the window and from 10.58% to 2.33% for the order between 120 and 480 rows, and to 0.25% for the window at 960; that the fall is carried by the share of disagreements that change the winner, 28.5% to 2.6% for the window and 21.6% to 5.1% for the order, while the disagreement rate stays near 47% for the order and jumps between 12% and 43% for the window; and that the regret of tuning per candidate falls from about four thousandths to inside two standard errors of zero by 480 rows.

What stays out, named as a decision: other tables and laws, deferred above; a list chosen for the sample size, since the window list here is the field’s list at every size and its fixed values are part of why the rate jumps; and the power of a real effect, since every draw here is at a true null. The boundary against a rate times a size, which separated the field’s product into its factors at one sample size, is that this essay finds which factor a second dial moves.