A quarrel's cost is its width

A table of fifteen candidate regressions on dependent errors is usually compared after whitening, and whitening needs a tuning value — an autoregressive order, or the width of a covariance band — read off the residuals. Choose that value once, from the fullest candidate, and every candidate is scored through the same whitening. Choose it per candidate, and each is scored through its own; the candidates can then disagree about the value, and when they do the table may select a different winner and pay for it in regret, the shortfall of the chosen model’s prediction error against the best available.

Two values the criterion cannot tell apart asked which values on a tuning list make the candidates disagree, by reading every two-value list off the same draws. The answer was the pairs the criterion cannot separate — adjacent orders, two widths both live at once — and that a pair holding a value the criterion never picks is disagreed about on no draw at all. It left the second half of the question open, and named it. How often the candidates disagree is the first factor of what a per-candidate rule costs, and a rate times a size had shown the cost to be that rate times a second factor: the regret a disagreement carries. Whether that second factor depends on which pair is disagreed about had not been measured.

The plausible guess is that it does, and in a particular way. A quarrel between orders one and two, under a first-order law, is a quarrel between the truth and one wasted parameter. A quarrel between eleven and twelve is between two orders both wasting ten. If the cost of a disagreement is the cost of being wrong about the dependence, the first should be dearer. The measurement says the guess is wrong, and that something simpler is right.

Seventy-eight lists on one set of draws #

The setting is the one the field has used throughout: a first-order autoregression at a persistence of 0.8 and a hundred and twenty rows, fifteen candidate subsets of four regressors, and the sieve’s autoregressive orders from zero to twelve. A per-candidate rule reads each candidate’s own residuals and picks the order its criterion prefers from the list; a shared rule reads the fullest candidate’s residuals once. The cost is paired on the draw — the per-candidate rule’s regret minus the shared rule’s — and is exactly zero on every draw on which all fifteen candidates pick what the shared rule picks, because the two rules are then the same rule.

Seventy-eight two-value lists cannot each be run separately at the draw counts a conditional average needs, and they do not have to be. On one draw, everything the two rules read is a function of a candidate and a value: each candidate’s tuning criterion at each order, its score at each order through the fullest candidate’s whitening, and the regret of its fit at each order. Computed once per draw on the full list, any list’s shared pick, own picks, two winners and paired regret are a lookup. Read at the full list, that lookup reproduces the rule it stands in for draw for draw, with no difference at all, for both tuning parameters. Here it is read 78 times on each of 2,400 draws.

The twelve lists that hold order zero are disagreed about on no draw: the criterion never picks zero under this law, so every candidate picks the other value. They are left out of everything that follows. The other sixty-six are the figure.

The pattern is visible before any statistic is computed. Along the diagonal — neighbouring orders — the cost is about one thousandth everywhere: 0.00126 for one and two, 0.00155 for two and three, 0.00088 for eleven and twelve. Four orders apart it is about eight thousandths whether the list is one and five or eight and twelve. Eleven apart, one and twelve, it is 0.02436. The grid grows away from the diagonal and is level along it. The cost of a quarrel is set by how far apart the two values are, and the values themselves do not appear.

A line in the distance #

A straight line in the distance between the two orders explains 96.2% of the variation in cost across the sixty-six lists, at 0.00272 of regret for every order of distance. The scatter around it at each distance is about the size of the standard errors, which run from under half a thousandth for neighbouring orders to about four for one and twelve. And the lists holding the true order, order one, sit on the line with the rest: matched on distance, one and three cost 0.00597 against 0.00434 to 0.00549 for the other lists three apart; one and eight cost 0.01610 against 0.01508 to 0.01722 for the other lists seven apart. No list holding the truth is more than two and a half standard errors from the line.

So the guess was wrong in both of its parts. A quarrel between the truth and a wasted parameter is not dearer than one between two wasted parameters at the same distance, and a quarrel between two far-apart orders is not cheap merely because both are generous. The mechanism the line suggests is that a disagreement costs what the two whitenings differ by, and under a first-order law every sieve order from one up whitens correctly in expectation and differs from the next by one more noisy coefficient. Orders eleven and twelve whiten almost identically; one and twelve differ by eleven estimated coefficients, and the fit through one of them can be visibly worse than the fit through the other.

That is also why the truth does not matter here. Order one is the true law, but order two is a correct law too — it holds the first-order process with a second coefficient that is zero in expectation — and so is every order above it. Under this law the list never offers a wrong whitening, only noisier and less noisy correct ones, and the distance between two of them is the number of extra coefficients one carries.

Two factors that cross #

The cost of a per-candidate rule on a list is the rate at which the candidates disagree times what a disagreement costs. The line is about the second factor. The first moves the other way.

Neighbouring orders are disagreed about on 27.8% of draws on average, and one and twelve on 4.2%. That is the earlier essay’s finding read along the same axis: two orders the criterion cannot tell apart are the two it splits on, and two orders far apart are told apart easily. What a disagreement does when it happens moves the opposite way. Between neighbours, a disagreement changes which candidate the table selects on 10.1% of occasions; between one and twelve, on 51.5%. A quarrel about whether the whitening has eleven coefficients or twelve rarely reaches the model selection, because the fifteen candidates’ scores barely move; a quarrel about one coefficient against twelve reaches it half the time.

Their product — how often the list changes the winner at all — is 2.81% for neighbouring orders, rises to a peak of 4.71% five orders apart and falls back to 2.17% at the widest. And what a changed winner costs rises across the whole range, from 0.0106 to 0.0462. So the conditional cost’s line has two sources: a disagreement is more likely to change the winner, and a changed winner is more costly, as the distance grows. Neither of them depends on which orders the list holds.

How often it matters found the product — how often the list changes the winner — moving from 17.6% to 1.5% as the candidates were spread further apart. Here the candidates are fixed and the values are moved, and the product barely moves at all: it is the one number on this figure that does not depend much on the pair.

The line, on lists it was not fitted to #

A law fitted on two-value lists is worth something only if it says what happens on the lists a practitioner uses, which hold many values. Every evenly thinned order list, from three values to all thirteen, can be read off the same draws, and on each one the candidates’ disagreement has a width: the distance between the highest and lowest order any of them picked. If the cost is set by width alone, the pairs’ line should predict each list’s cost from its width.

It does. On the three-value list the candidates spread over six orders when they disagree and the cost is 0.01576 against a prediction of 0.01398; on the full thirteen they spread over 2.92 and the cost is 0.00651 against 0.00560. Every one of the eleven lists sits within two and a half standard errors of the line drawn from the pairs. That settles a result a rate times a size reported and could only half explain: the order’s conditional cost falls as its list grows, from 0.01100 at four values to 0.00637 at thirteen in that essay’s twelve hundred draws. A longer interval of integers lets the candidates disagree by less, and a narrower disagreement costs proportionately less. The list’s length acts on the cost only through the width of the quarrels it permits.

The window has no width law #

The same measurement on the window — eight band widths from no lags to thirty — gives a different answer, and the difference is the earlier essay’s warning about transport made specific.

Only eleven of the twenty-eight window lists are disagreed about on forty or more of the 2,400 draws; the other seventeen are pairs the criterion separates almost every time, most of them on every draw. On the eleven, the distance in lags explains none of the cost — a fitted line has an $R^2$ that rounds to zero. The widest quarrels are among the cheapest: one lag against thirty costs 0.00231, one against twenty 0.00249. The dearest are between widths the criterion treats as live together: four against twelve costs 0.02290, four against twenty 0.03542. Measured on the width of the band’s log-scale sequence, or on the number of list steps between them, the distance explains no more.

What the window’s quarrels follow is closer to the earlier essay’s account of its disagreement rate. A list holding one lag against thirty is disagreed about often — on 37.4% of draws — and changes the winner on 69.6% of those occasions, but the winner it changes to is nearly as good: a changed winner costs 0.0046 there against 0.0472 between four and twelve. A reading that fits these numbers, and is not itself measured here, is that the band at one lag and the band at thirty are both poor whitenings of a persistence of 0.8, in opposite directions — one leaves most of the dependence in, the other estimates thirty noisy autocovariances — so the candidate each prefers is one neither describes well, and moving between them costs little. Between four and twelve both are good whitenings, the candidates that prefer each are genuinely different, and choosing the wrong one costs ten times as much.

That is the opposite structure to the order’s, and it has a reason that can be stated. The order list is a sequence of correct whitenings ordered by noise, so the distance between two of them is a distance in noise and the cost follows it. On that reading the window list is a sequence of biased whitenings — a band too narrow leaves dependence in, a band too wide adds estimated noise — so two widths far apart can both be wrong by similar amounts and the cost of moving between them is small. The quarrel that changes the winner found on the window that the distance between the values disagreed about had nothing to do with the cost. That was true and is true here. It is not true of the order.

How the numbers were checked #

The lookup that makes seventy-eight lists affordable is checked against the rule it stands in for: read at the full list, it gives the paired regret the per-candidate rule produces when it is run directly, on every draw, for both parameters, to rounding. The decomposition is exact by construction — a draw on which no candidate disagrees has a paired regret of exactly zero — and the largest such residue on any list here is zero. The width law is checked on its own terms: a line in distance explaining more than nine tenths of the order’s cost, and every list holding the true order within two and a half standard errors of it. The refusal is the sentence the order invites about the window — that its quarrels too cost their width — and on the window’s eleven measurable lists it fails, with an $R^2$ of zero.

Every list is read on the same 2,400 draws, so two lists are compared paired rather than across seeds, and the standard errors quoted are those of a conditional mean over the draws on which each list is disagreed about: between 629 and 697 draws for the neighbouring orders, about a hundred for one and twelve.

Still open: a law under which a list holds a wrong whitening #

Everything about the order here rests on its list being a sequence of correct whitenings, which is a property of the first-order law. Under a five-period moving average no finite autoregression is correct — the window against a model essay found none below 83% error — and low orders are then wrong, not merely noisy. The prediction is that the order’s width law breaks there in the window’s direction: quarrels between two low, biased orders become cheap and quarrels near the order where bias and noise balance become dear. If that holds, the statement is not “the order’s cost follows its width” but “a tuning parameter’s cost follows the distance between its whitenings in noise, and only a list of correct ones orders them by that distance”. The moving-average run is the measurement, and it has not been made.

A second open question is the product. The rate falls and the turnover rises with the distance, and their product is nearly level. Whether that near-cancellation is the same invariant the field found across list lengths — a property of how far apart the candidates are rather than of the list — or a coincidence of this law is a question about the candidates, and the measurement that would answer it holds the pairs fixed and moves the table.

What is claimed here, and what is not #

This essay takes the regret of a per-candidate tuning disagreement, list by list, on every two-value list of the order and the window. The claims are that no list holding order zero is ever disagreed about; that across the sixty-six lists of positive orders the conditional cost is a line in the distance between the two orders at 0.00272 an order, explaining 96.2% of its variation, with neighbouring orders at about a thousandth and one and twelve at 0.02436; that the lists holding the true order sit on that line within two and a half standard errors; that the disagreement rate falls from 27.8% to 4.2% and the share of disagreements that change the winner rises from 10.1% to 51.5% across the range of distances; that the line, fitted on pairs, predicts the cost of every evenly thinned order list from three values to thirteen within two and a half standard errors from the width of the candidates’ disagreement; and that on the eleven window lists disagreed about often enough to measure, distance in lags explains none of the cost, the widest quarrels costing about two thousandths and four lags against twelve or twenty two to three hundredths.

What stays out, named as a decision: other laws, deferred above, on which the order’s list may hold wrong whitenings; other tables and sample sizes, since everything is one table at a hundred and twenty rows; and lists of more than two values chosen freely, which the thinned lists stand in for and do not exhaust.

The boundary against two values the criterion cannot tell apart is that it found which pairs are disagreed about, and this essay what a disagreement about each pair costs. Against a rate times a size, which separated the per-candidate cost into its two factors across list lengths, this essay holds the length at two and finds the second factor’s dependence on the pair — and, for the order, the reason the first essay’s sizes fell. And against the volume a whitening moves, whose determinant term is in every order score here, it changes nothing about the criterion and asks only what the criterion’s disagreements cost.