A covariance repaired

A whitening needs a covariance matrix, and the obvious estimate of a general dependence is not one. A covariance with no parameter showed the failure: take a residual series’ sample autocovariances, truncate them at a window $L$, build the Toeplitz matrix, and on most draws of a persistent series it has a negative eigenvalue and no Cholesky factor at all. The usual cure is a taper. Multiply the autocovariances by a Bartlett triangle and the matrix is positive semidefinite by construction.

The window that has to be chosen went further and turned the failure into a warning. A truncated rule’s regret cannot be read without its availability, because it exists only on the draws it finds easy. At a window of two lags at persistence 0.85 it exists on one draw in fifty and its regret is better than the oracle’s, which is arithmetic proof that the row was computed on a selected subset. So this field has always recorded such a rule as absent on a failed draw, and never replaced it with anything.

That was the right call for what it was measuring. But it left a question nobody asked. Between refusing a truncated estimate and tapering it there is a third thing, the one a numerical analyst would reach for first: find the nearest matrix that is a covariance, by flooring the eigenvalues, and use that. It exists on every draw, so it has no availability to read. It keeps the truncated estimate’s flat weights, which are what a taper gives up, and it moves the matrix hardly at all. It is the obvious way to get a truncated rule scored on all draws at last.

The measurement is short, and its result was not what the description above suggests.

The nearest matrix #

The repair replaces each eigenvalue of the truncated matrix by the larger of itself and a floor. In Frobenius norm that is the closest matrix whose spectrum sits at or above the floor. The result is not Toeplitz or banded any more, so it gets a dense Cholesky factor, and the rest of the rule is as before: one shared estimate from the widest candidate’s residuals, whitening every candidate by the same factor, and choosing by the whitened criterion.

The floor has to be something. The natural choice is a small positive number, the value a practitioner uses to keep a factorisation alive: say a thousandth of the diagonal, or a hundredth.

The next figure is regret against that floor, on the field’s standard table: fifteen candidate models, 120 rows, a persistence of 0.85, 120 draws, at four windows on a log scale. Least squares, with no whitening at all, gives up 0.089 on this table. A whitening told the true dependence gives up 0.0075.

At a floor of 0.001 the repaired rule gives up 1.56 at a window of four lags, 1.19 at eight, 0.87 at twelve and 1.71 at twenty. That is not a slightly worse estimate. It is between ten and twenty times what least squares gives up, on a rule whose entire purpose was to improve on least squares. At a floor of 0.01, it is 0.094, 0.098, 0.062 and 0.25, still no better than doing nothing. Only at about 0.05 does it pass least squares, and at about 0.2 it is as good as it gets: 0.0226, 0.0197, 0.0212 and 0.0202.

So there is a curve here with an interior minimum, over a parameter that was not supposed to matter, and its left arm covers two orders of magnitude.

How little the matrix moved #

The obvious first explanation would be that a small floor is a big repair. It is the reverse.

At a window of four lags and a floor of 0.001, the repair moves the matrix by 4.1%, measured as Frobenius distance over the matrix’s norm. It lifts 26 of its 120 eigenvalues. At a window of eight it moves it by 1.0% and lifts eleven. At a floor of 0.2, which is sixty times less harmful, it moves it by 8.8% at four lags and 4.5% at eight, and lifts 43 and 42. So the least-moved matrix does the most damage, by nearly two orders of magnitude.

That is not paradoxical once what a whitening does is put in view. A whitening multiplies by $\Omega^{-1/2}$. In the eigenbasis, that is dividing each direction by the square root of its eigenvalue. So the directions at the floor get multiplied by $1/\sqrt{0.001} \approx 32$, and the regression is thereafter dominated by whatever the data does along those few directions. Those directions were not chosen for any reason to do with the dependence. They are where the truncated sample sequence happened to go negative, which means noise.

So Frobenius distance is a measure in the wrong geometry. It is dominated by the large eigenvalues, because they are large, and it barely registers a change at the small end. An inverse is dominated by the small eigenvalues, for the same reason. The closest matrix in one geometry is not close in the other. The window essay said that “a whitening is not a sum” and that an inverse is sensitive to what a sum is not. Here the same claim has a figure on it, sixty to one.

The floor is not a safety margin. It is the largest multiplier the whitening is allowed. At 0.2, it is a multiplier of about 2.2. That is a shrinkage, of a kind, applied to the part of the spectrum where the estimate has no information, and the regret at each window falls until the floor stops only shrinking noise and begins flattening real signal, which is why the right-hand arm starts to rise past 0.4. By 0.7 most of the spectrum is at the floor, and the whitening has almost stopped doing anything.

The rule the availability was hiding #

There is a second thing this measures, which the field has wanted to know since it first noticed the availability problem: what is the truncated rule worth on the draws it was absent from?

The truncated rule scored only on the draws it can factor gives up 0.0072, 0.0194, 0.0257, 0.0176, 0.0289 and 0.0468 at windows of 2, 4, 8, 12, 20 and 30. It exists on 2%, 18%, 37%, 34%, 23% and 10% of draws at those windows. The earlier essay read its 0.0072 at two lags, a number below the whitening told the truth, as proof of a selection effect, and it could say no more than that.

The same weights at a floor of 0.2 give up 0.0258, 0.0226, 0.0197, 0.0212, 0.0202 and 0.0263. These numbers are nearly flat across the windows, and all of them are real, on every draw. At two lags that is 3.6 times the selected figure, which puts a size on what the selection was concealing. At eight lags it is lower than the selected figure. So the truncated rule is not uniformly flattered by its availability: at a window where it survives on a third of draws, it is not doing well even on those.

Setting that beside the taper, which is available on all draws by construction: 0.0396, 0.0275, 0.0178, 0.0150, 0.0135 and 0.0160. At the two shortest windows, the repair wins, by quite a lot at two lags. From eight lags on, the taper wins. The best of the taper’s windows, at 0.0135 and twenty lags, beats the best of the repair’s, 0.0197 at eight, by about a third.

That is a clean comparison with an interesting asymmetry. A taper at two lags has nearly thrown away the lag-two autocovariance it was given, at a persistence where that matters, so a repair that keeps both lags at full weight does better there. At twenty lags the taper has all the lags it needs, and it is the inverse’s noise that matters. The same weights are less kind then. And the fitted first-order whitening, told only the form of the dependence and given a single number to estimate, is below all of them, at 0.0120. Every window on both curves is paying for not knowing that form.

The third selection problem #

The earlier essay’s closing section named a window chosen from the data as “a third selection problem”. This essay has found a fourth, and it is one the textbook description hides completely. A repair has a floor. The floor ranges over nearly three orders of magnitude between harmless and catastrophic, and the catastrophic end is the one that looks most conservative, because a small floor sounds like a small intervention.

No gate or diagnostic would catch that. A repaired rule fits, it has a factor, its condition number is a number like any other, and its whitened criterion selects a model of about the right size: about four coefficients at every floor. Nothing about the selected model size distinguishes a floor of 0.001 from a floor of 0.2. Only a comparison on fresh rows does, and a practitioner choosing a model on 120 rows does not have those.

How would one choose it? The fit-by-minimum in the first figure is not available to anyone without the truth. A floor of 0.2 was found by running a simulation at this persistence, and it is a statement about this table at this persistence. Set against the other persistence measured here, 0.5, where a floor of 0.01 is available and truncation rarely fails: there the repaired rule at a floor of 0.01 gave up 0.0347, 0.0313, 0.0363 and on up to 0.0733 at thirty lags against a taper that stays near 0.028, and against 0.0356 for least squares. The same floor that is near harmless at a persistence of 0.5 at short windows does damage at longer ones, where a few more eigenvalues go below it.

The floor of 0.2 does not transfer perfectly either. At a persistence of 0.5 it gives up 0.0309, 0.0335 and 0.0354 at windows of four, twenty and thirty lags, against about 0.028 for the taper and 0.0356 for least squares. So what was a clear gain at 0.85 is close to nothing at 0.5. At a persistence that low the truncated sequence hardly ever needs repairing at four lags: no eigenvalue falls below 0.01 on the average draw, its smallest averages 0.27, and yet the floor of 0.2 lifts two or three of them anyway and changes nothing much. By thirty lags it is lifting thirteen.

So a practitioner needs a rule for the floor as well as for the window, and nothing measured here supplies one. It is the same trap the triangle that was not the multiplier found in a resampling and the residue that was the weights found in a likelihood: a choice made for numerical convenience turned out to be doing statistical work, and nothing printed it.

Which repair the field should use #

The direct recommendation is short.

A truncated covariance that will not factor should not be repaired with a small floor. It is the worst thing on the page, including least squares. A numerical library’s default regularisation, say a small multiple of the identity added to clear a factorisation, is in this class.

If the flat weights have to be kept for some reason, the floor belongs at something like a fifth of the diagonal. At this persistence that gives up 0.020 or so at all windows, which is three times the told whitening and well below least squares, and it is insensitive to the window, which may be worth something.

Otherwise use the taper, and a long window, as the earlier essay says. It reaches a lower floor, at the cost of being worse at the shortest windows, and it has no second dial. Its positive-definiteness is a property of the weights and not something added afterwards. That is what it was buying all along, more than anything to do with bias.

And do not read an availability rate as a virtue of the taper alone. Errors from a fitted model escaped this whole problem from another side, since a fitted autoregression’s implied covariance is positive definite by construction and has no floor to set.

And if the form of the dependence is known or can be assumed, that is better than all of it. What the form buys already put a price on that, and it is the same conclusion from another direction.

How the numbers were made #

The world is the field’s: four predictors with coefficients of 0.30, 0.20, 0.12 and 0 on tiered persistences, errors at 0.85 or 0.5, fifteen candidates, 120 rows to fit and a separate replicate to score, on 120 draws at each cell. The baseline is the best model available from least squares and from the whitening told the true persistence, as in the field’s earlier tables, so a change in the window does not move the benchmark.

The repair takes the widest candidate’s residuals, normalises the truncated sequence to a unit diagonal, finds all 120 eigenvalues by Jacobi rotations, floors them, rebuilds the matrix and factors it. Its Cholesky factor is stored in the same layout as the banded one, with a bandwidth of $n - 1$, so all of the whitening that follows is the same code. At a standard error of about 0.003 in the regret at these draws, and much more in the left-hand arm, where a single draw can have a very large whitened fit, the numbers on that arm should be read for their size and not their value.

Three things are required. At a floor of 0.001 the repair must move the matrix by under 6%, and regret more than twenty times what a floor of 0.2 does. The truncated rule must exist on fewer than 40% of draws at a window of four lags, and the repair on all of them. And the reading that the nearest covariance matrix whitens better than none at all is refused at the lowest floor, against least squares itself.

Still open: a floor from the sample, and a repair in the other geometry #

A floor that reads the sample. The right floor falls as the truncated sequence has fewer negative eigenvalues, and a plausible rule sets it from the spectrum’s own bottom: say at the size of the largest negative eigenvalue, on the argument that anything smaller than that is indistinguishable from noise. At a window of four lags, where the smallest eigenvalue averages −0.24, that comes close to the empirical 0.2. At twenty lags, where it is −0.19, it does too, and at thirty, −0.25. That is suggestive enough to be worth running as a rule, on the same table at both persistences. It has no free parameter.

Repair by the inverse’s geometry. Nearest in Frobenius is the wrong distance for this use. Nearest in Stein’s loss, or any distance that treats a small eigenvalue as carefully as a large one, gives a different correction for the same negative spectrum, and it is a shrinkage rather than a projection. It is more work per draw and it has its own parameter, and whether it removes the second dial or only moves it is worth knowing.

What this leaves of the truncated rule #

This essay takes the truncated autocovariance estimate, made positive definite by flooring its eigenvalues, on the field’s standard selection table. Its claims are that at a persistence of 0.85 a floor of 0.001 gives up 0.87 to 1.71 of regret at the four windows measured, against 0.089 for least squares, while moving the matrix by only 0.8% to 4.1%; that the same weights at a floor of 0.2 give up about 0.02 at every window; that against the taper this wins at windows of two and four lags and loses from eight on; and that the best of the taper’s windows beats the best of the repair’s by about a third.

What stays out, as a decision: a floor chosen from the sample, named above and not measured; other windows than flat and Bartlett, since a flat-top window has a different spectrum; and any repair applied per candidate, where the same second dial would meet the earlier essay’s second selection problem. The boundary against the window that has to be chosen is that it read a rule on the draws where it exists and refused to go further, and this one goes further and finds out why refusing was the better idea.