Two values the criterion cannot tell apart

A list is not a rule measured how often fifteen candidate regressions disagree about a tuning parameter as the list it is chosen from is thinned. For a sieve’s order the rate climbed from nothing at two values to two draws in five at thirteen, and almost all of the climb arrived by the eighth value. For a band’s width it did something stranger: thinning the eight natural widths gave a sequence that changed direction five times in six steps. That essay read the order’s sequence as an evenly spaced scale and the window’s as a grid whose thinning makes a different instrument each time, and it ended on a count of how much of the effect each step switched on.

Both readings treat a list as a length. This essay treats it as a set, and asks a narrower question that the thinning sweep could not: which values on a list make the candidates disagree, and why those.

Fifteen candidates, fifteen residual series #

The rule being measured is the one the window field priced: a table of candidate regressions, each choosing its own whitening from its own residuals before the table is compared. The table here is every non-empty subset of four predictors — fifteen candidates, from each predictor alone up to all four together — and at a true null every coefficient is zero, so every candidate contains the truth and any preference among them is manufactured.

Each candidate leaves a residual series, and each residual series is handed the same tuning list. The fifteen series are nearly the same series: they differ only by the noise each candidate’s fitted coefficients absorbed, which at a hundred and twenty rows and a true null is a few per cent of the variance. So the fifteen tuning criteria are fifteen slightly perturbed copies of one curve, and whether they agree about its minimum depends on how sharply the curve has one.

That framing is what the rest of this essay tests. It predicts that disagreement should live where the curve is flat between two values on the list, wherever on the list that is, and not anywhere in particular relative to the true order — the same distinction the field on searches drew between a criterion that counts parameters and one that has to resolve between them.

Seventy-eight lists of two #

The cheapest list that can produce a disagreement has two values on it, and there are seventy-eight of them among the thirteen orders. Each candidate’s criterion is computed once per draw at every order, and any list’s pick is then read off the same numbers — the first strict minimum in list order, which is how the rule breaks ties. That reading was checked against the rule itself on 1,200 picks across four lists, all fifteen candidates and twenty draws, and agreed on every one. So all seventy-eight pairs are measured on the same 240 draws at a true null, with every candidate containing the truth, and a pair can be compared with any other pair on identical data.

The triangle has three features and the first is a row of zeros. Every pair holding order zero disagrees on no draw at all. Under the first-order autoregression at 0.8 these residuals carry, a whitening of order zero is so much worse than any whitening that the criterion never picks it, so a list of zero and anything is a list of one competitive value, and one value is one choice.

The second is the diagonal. Every adjacent pair of positive orders disagrees on between 17.1% and 25.0% of draws — 22.1% for one and two, 22.1% for two and three, 25.0% for eleven and twelve — averaging 20.9% across the eleven of them. The full thirteen-order list disagrees on 40.0%. On average a list of two adjacent orders produces more than half the full list’s disagreement on its own.

The third is the falloff away from the diagonal: one and three disagree on 14.6%, one and six on 15.4%, one and twelve on 5.0%. Two orders far apart are less often near a tie than two orders next to each other.

What the triangle does not show is any special status for the right answer. The law has one autoregressive coefficient, so order one is the truth and the criterion picks it most often. But eleven and twelve, both far past it and both wasting parameters, disagree at least as often as one and two. Disagreement is not about being near the answer; it is about being near each other on the criterion’s scale.

Where on the criterion’s scale #

The margin at which fifteen candidates come apart can be read directly. For the pair of orders one and two, each draw gives the fullest candidate’s criterion gap between them — the number a shared rule reads — and whether all fifteen candidates picked the same order.

Where the shared margin is under half a unit the candidates disagree on 67.7% of draws; between half a unit and one, 43.8%; between one and one and a half, 25.8%; between one and a half and two, 4.5%. The fifteen candidates’ own gaps for the pair differ from one another by 0.82 of a unit at the median draw. A shared margin smaller than that is a margin one of the other fourteen crosses; a larger one is not.

The distribution of the margin itself explains the size of the diagonal. Adding a second autoregressive lag that the law does not need gains a likelihood that is roughly chi-squared on one degree of freedom, and the criterion charges two units for it, so the gap is two minus that gain and can never exceed two. Most draws gain little: 134 of the 240 sit between one and a half and two, where almost no one disagrees. The draws that gain between one and three units put the margin within a unit of zero, and the chance of a gain in that range is $P(1 < \chi^2_1 < 3) =$ 0.2340 — close to the 20.9% the adjacent pairs average. That is a rough account, resting on a spread of about one unit that is measured rather than derived, but it is why every adjacent pair past the truth disagrees at about the same rate: each extra lag faces the same penalty and the same chi-squared gain.

A handful of draws sit the other way, with order two winning by two units or more — 12 of them, a third of which disagree. On those the law’s single lag has left real structure in some candidates’ residuals and not in others, which is a different source of disagreement from a near-tie and a rare one at this sample size.

The thinning sweep, read as sets #

With the pairs measured, the earlier sweep’s two sequences stop being puzzles. Each thinned list keeps its endpoints and takes values at equal steps through the full list’s positions, and what matters about each is which nearby pairs of competitive values it happens to hold.

For the windows the natural list is zero, one, two, four, eight, twelve, twenty and thirty lags, and under this law only two widths are ever seriously in contention for the fullest candidate: four and eight. The pair of four and eight alone disagrees on 38.8% of draws. Thinned to six values the list is zero, one, four, eight, twenty, thirty and holds both; thinned to seven it is zero, one, two, eight, twelve, twenty, thirty and holds eight without four. On twice the draws the earlier essay used, the sequence from two values to eight is 1.7%, 0.4%, 17.5%, 9.6%, 38.8%, 10.0% and 41.3%, and every thinned list above three draws in ten is a list holding both four and eight. The five reversals are the thinning rule dropping and restoring one of two values, which is exactly what a grid thinned by position does to a scale whose competitive values are not evenly spaced.

For the orders the competitive values are all the positive ones, and the sweep rises as the thinned lists keep more of them close together. At four values the list is zero, four, eight, twelve — three positive orders four apart — and disagrees on 13.8%. By eight values it holds two and three side by side and reaches 38.3%. At ten values the thinned list is zero, one, three, four, five, seven, eight, nine, eleven and twelve: it has lost order two, the value the most-picked order is most often tied with, and the rate drops to 31.3% before recovering. The even scale the earlier essay saw is even because the order’s competitive values are evenly spread, so thinning removes them in proportion; the window’s is a grid because its two competitive values are a single pair the thinning keeps or loses.

One candidate’s doubt, fifteen candidates’ disagreement #

If disagreement is a matter of near-ties, then a list’s disagreement rate should be predictable from how often the criterion is near a tie in the first place — and that can be read from one candidate alone. The fullest candidate is the one a shared rule reads. Its own pick varies from draw to draw, and how often it picks something other than its most common value is a measure of how decisively the criterion separates the values on the list.

Across the 90 order lists — the seventy-eight pairs and the twelve thinned lists — the two quantities have a rank correlation of 0.964. Across the 35 window lists — twenty-eight pairs and seven thinned lists — it is 0.949. The second quantity knows nothing about the other fourteen candidates; it is computed from one residual series per draw. That one candidate’s doubt about its own choice predicts fifteen candidates’ disagreement this well says what the disagreement is: the other fourteen are near-copies of the fullest, with slightly different residuals, and they part company exactly where the fullest candidate’s criterion is close to indifferent.

It also gives a practitioner an instrument that does not require running the expensive rule. Before choosing a tuning value per candidate, choose it once for the fullest candidate on bootstrap replicates of the residuals, and see how often the pick moves. A list on which it rarely moves is a list on which per-candidate tuning will rarely do anything.

What the length was standing in for #

The earlier essay found that the length of the list moves the disagreement smoothly and the cost of disagreeing not at all, and a later field separated the cost into how often a disagreement changes which candidate wins and what a changed winner is worth. This essay adds the piece that comes before both: what decides how often a disagreement happens.

The answer is the number of pairs of values on the list that the criterion cannot tell apart, weighted by how close each pair sits to a tie. Length correlates with that for the orders, because their competitive values are evenly spread, and does not for the windows, because theirs are not. A two-value list can disagree more than a four-value one — the pair of one and two disagrees on 22.1% of draws and the thinned list of zero, four, eight and twelve on 13.8% — and a list can grow by a value and disagree less, when the value it gains is never competitive and the thinning that made room for it broke up a close pair.

So the matched-length comparison the earlier fields relied on matched the wrong thing. Two lists of eight are comparable if they hold the same number of near-tied pairs, and a window list of eight and an order list of eight hold very different numbers of them. The comparison that was not made was declined because the lists differed in length; it would have been fair to decline it for this reason instead, and matching the lengths did not remove it.

What it changes about choosing a list #

A tuning list is usually built for coverage: enough values to reach wherever the answer might be, at a spacing fine enough not to miss it. This adds a second consideration for any rule that tunes per candidate.

Values the criterion never picks are free. Order zero adds nothing to any list here, and a window of thirty lags adds nothing unless it is paired with another value it can tie with. Reach costs nothing in disagreement.

Close spacing where the criterion is flat is expensive. Adjacent orders past the truth tie about one draw in five, pair by pair, and a list that holds many of them gives fifteen candidates many chances to part. If the purpose of the list is to locate a value, a coarse grid with a refinement step afterwards — choose on a coarse list, then refine around the pick on the shared candidate alone — keeps the reach and removes most of the near-ties from the comparison across candidates.

And the order of the values on the list breaks the ties. The rule takes the first strict minimum in list order, so an exact tie always goes to the earlier value; a list written in descending order would change which value wins a tie, though exact ties are vanishingly rare with continuous residuals and the near-ties this essay measures are not affected.

Still open: whether the cost follows the pairs #

Everything here is about how often the candidates disagree, which is the first factor in the decomposition the next field made. The second factor — what a disagreement costs in regret — was found there not to move with the length of the list. Whether it moves with the pair is not measured: a disagreement between orders eleven and twelve, both wasting parameters, could cost less than one between one and two, where the truth is one of the options, or the same, if the cost is set by whether the winner among candidates changes rather than by which orders are in play. The seventy-eight pairs make that a measurement rather than a guess, and it is the next one.

The other open question is the law. The window that has to be chosen and the eighth that was not a constant both found the window list’s behaviour moving with the law and the table. Under the autoregression at 0.8 the window list has exactly two competitive widths and the order list has eleven. Under a law with a shorter memory both lists would have fewer competitive values, and the prediction from this essay is that the window’s disagreement would collapse wherever four and eight stop being close — which would make its thinning sequence flatter rather than more erratic. That has not been run.

The claim, and where it stops #

This essay takes which values on a tuning list make candidate regressions disagree about it. The claims are that, among the seventy-eight two-value lists of sieve orders at a true null, every pair holding order zero disagrees on no draw and every adjacent pair of positive orders on between 17.1% and 25.0%, averaging 20.9% against 40.0% for all thirteen; that for the pair one and two the candidates disagree on 67.7% of draws where the shared criterion margin is under half a unit and 4.5% where it is between one and a half and two; that every thinned window list disagreeing on more than three draws in ten is one holding both four and eight lags, which accounts for the window sequence’s reversals; and that across 125 lists the disagreement tracks the fullest candidate’s own uncertainty with rank correlations of 0.964 for orders and 0.949 for windows.

What stays out, named as a decision: the cost of each pair’s disagreement, deferred above; other laws and sample sizes; and the shared candidate’s bootstrap, proposed as an instrument and not run. The boundary against the volume a whitening moves, which showed the order’s criterion needs its determinant term once it is per candidate, is that every order criterion here is the sieve’s likelihood with its charge, read for one candidate at a time, so the determinant plays no part in any number above.