A covariance with no parameter in it

There is a rule that repairs an information criterion on rows that repeat each other, and it is one line long: scale the first row by √(1 − ρ²), replace every later row by itself minus ρ times its predecessor, fit by least squares, and apply the ordinary penalty. The essay that measured it found that it recovers most of what counting rows gives up, and that the second selection problem it introduces — ρ has to be estimated from the same rows the selection is running on — costs a small fraction of what the repair is worth.

Every word of that depends on ρ being a number. The transform is written for a first-order autoregression; it has one parameter; the estimate of that parameter is a single lag-one autocorrelation read off a hundred and twenty residuals. A criterion reads differences between candidate scores, so an error in a quantity every candidate carries cancels out of every difference, and one number carried by fifteen candidates is about as shared as a nuisance can get.

A real dependence does not come with a parameter. This essay replaces the named Ω with an estimated one and measures what that costs. The answer has three parts and the first is not about accuracy at all.

The obvious estimate is not a covariance matrix #

Take the residuals, compute their sample autocovariances γ̂(0), γ̂(1), γ̂(2) … and stop at some lag L. Fill a Toeplitz matrix with them, take its Cholesky factor, use the factor to whiten. That construction is what anybody writes first and it has a defect that is invisible until it is run.

The truncated sequence need not be positive definite. The whole sample autocovariance sequence — every lag out to n − 1, divided by n rather than by n − k — is the autocovariance function of an actual process, so it is positive semidefinite by construction. Zeroing everything past L destroys that guarantee, and on a persistent sample it destroys it often.

Sixty-five per cent is not a corner case. At the best window there is, the rule built on a truncated estimate does not exist on two draws in three — and does not exist is not what it looks like from the outside. A whitening that cannot be built returns nothing, and the natural thing to do with nothing is to fall back on least squares for that draw. A rule that does that is two rules reported under one name, and its average is an average over the draws whose residual sequences happened to be benign.

The repair is standard and is a hundred years old: taper the sequence instead of cutting it. Weight lag k by 1 − k/(L + 1) — the Bartlett window — and the result is positive semidefinite for every sequence, because it is the autocovariance of the block-averaged process. Parzen’s window is smoother and has the same guarantee. Both fail on none of three hundred draws at any of six windows, which is a guarantee nobody had checked on this site’s own arithmetic until it was checked here.

The price of not knowing the form #

With a whitening that always exists, the comparison can be made. Five rules, all of them fitting the same fifteen candidates on the same hundred and twenty rows, differing only in what they are told about the errors.

Least squares with the ordinary penalty gives up 0.08683. The rule told the true ρ gives up 0.00879, which recovers 89.9% of that. Estimating ρ from each candidate’s own residuals costs 0.00563 more, at 5.4 standard errors of the paired difference — the same cheap second selection problem the field’s first essay priced, arriving here as six points of the eighty-nine.

Estimating a whole covariance costs 0.00672 more again, at 5.6 paired standard errors, and lands at 0.02115 — 75.6% recovered where knowing ρ recovers 89.9%. So the two prices are the same size:

• knowing the dependence against estimating its parameter: six points;
• estimating its parameter against estimating its shape: eight points.

That is the sentence this field exists for. The obstacle the criterion essay named — a nuisance estimated from the rows being selected on — is real and small when the nuisance is one number. Take the parameterisation away and the same obstacle is larger than the first one, because what has to be estimated is no longer a number, it is a function.

Estimated once, or estimated fifteen times #

Which raises the question the first field could ignore. There is a choice about where a general Ω̂ is estimated, and it does not exist when the estimate is a single lag-one autocorrelation that every candidate can afford to compute for itself.

Estimating a tapered Ω̂ once, from the fullest candidate’s residuals, before any selection, gives 0.02115. Estimating it from each candidate’s own residuals gives 0.02482 — worse by 0.00367, at 4.3 paired standard errors. The rule with more information about each candidate is the worse rule.

The mechanism is the one the criterion essay stated and did not have to act on: a criterion reads differences, and only the part of a nuisance error that differs between candidates survives into a comparison.

Seven and a half standard errors of spread is not noise that averages out. It is a systematic difference between candidates: a candidate that omits a persistent predictor has that predictor in its residuals and reports a dependence that is too high, and a candidate that fits everything has residuals a projection has already smoothed and reports one that is too low. Each candidate is scored against a different error model, and nothing in the criterion knows.

A shared nuisance cancels; a per-candidate one is a second comparison hidden inside the first. With one number the systematic part is a few hundredths and the damage is invisible. With a whole covariance the estimate has as many quantities in it as the window has lags, and the damage is measurable.

What each rule buys #

The dimension a rule selects says the same thing from the other side, and it says it without any reference to regret.

Least squares selects 3.22 coefficients on average at this persistence; the rule told the true ρ selects 3.94; the tapered rule estimated once selects 3.85. The whitened rules buy larger models because whitening makes the candidates separate more sharply once they are estimated efficiently — the same direction the criterion essay found and for the same reason. The estimated-covariance rule follows the infeasible one to within a tenth of a coefficient, which is the sense in which it is the right rule badly executed rather than a different rule.

What is actually being estimated #

It is worth being precise about the object, because a covariance matrix sounds like fourteen thousand four hundred numbers and is not.

Ω is n × n and Toeplitz, so it has n distinct entries; truncated or tapered at L it has L + 1, of which the first is a scale that the criterion’s own residual sum already carries. So a window of eight is eight numbers against the parameterisation’s one, estimated from a hundred and twenty rows, and it buys the ability to be right about a dependence that is not a first-order autoregression.

On this design the dependence is a first-order autoregression, and the eight-number estimate still loses to the one-number one by eight points of recovery. That is the honest shape of the comparison: the general rule is paying for a generality the design does not need. What it would be worth on a dependence the parameterisation cannot represent is a different measurement and this field does not make it — the sample here is generated from an AR(1), so the rule that assumes an AR(1) is correctly specified, and a fair contest would have to be run somewhere the assumption is false.

Naming that as a decision rather than a finding is the point. The number this essay reports is the cost of generality where generality is not needed, which is the only place it can be measured cleanly, and it is a lower bound on nothing and an upper bound on nothing.

The first estimated number costs six times what the rest do #

The two prices are described as the same size — six points against eight — and dividing by what each one buys says something the pair does not.

Going from a named ρ to an estimated one is one number estimated, and it costs 6.5 points of recovery. Going from that one number to a tapered band at eight lags is seven more numbers, and it costs 7.8 points. Per number that is 6.5 against 1.1: the first estimated quantity costs close to six times what each subsequent one does.

That asymmetry is the field’s own mechanism showing up as arithmetic. The first number carries the whole of the problem the criterion field named — a nuisance estimated from the same rows the selection is running on, so its error is correlated with the thing being selected. Every number after it adds only ordinary estimation variance to a quantity that already has that defect, and ordinary estimation variance is cheap when it is shared across candidates.

So generality is not expensive; the first step away from knowing is. A band at sixteen lags rather than eight would, on this rate, cost another eight points — real, and smaller than the step from a named parameter to any estimate at all. That reframes the choice a practitioner faces: the decision worth agonising over is whether the dependence is known, and the width of the band is a second-order question.

How much of the candidates’ disagreement is systematic #

The per-candidate estimates run 0.6287 to 0.6716 against a per-estimate standard error of 0.0056, and the essay reads the spread of 0.0429 as systematic. It can be sized.

Fifteen independent draws with that standard error would have an expected range of about 3.47 standard errors, which is 0.0194. The observed range is 0.0429 — 2.2 times as wide — so at least 79% of the between-candidate variance is something other than sampling noise.

That is a floor rather than an estimate, and the reason is worth stating: the fifteen candidates are fitted to the same hundred and twenty rows, so their estimation errors are strongly positively correlated and the range noise alone would produce is far less than 0.0194. The systematic share is therefore well above four fifths, and the correct reading of the figure is nearer the spread is the systematic part than the spread is mostly systematic.

The practical consequence is the one the field draws and it now has a size attached. A shared nuisance contributes a common offset that cancels out of every difference a criterion reads; a per-candidate one contributes an offset that is four-fifths-and-more a function of which candidate it belongs to, and that part does not cancel at all. Estimating the nuisance fifteen times rather than once converts almost the whole of the estimation error from a term that vanishes to a term that competes with the criterion, which is why the rule with more information about each candidate is the worse rule.

The banded factorisation, and why it is an economy #

One implementation note that is not a detail, because it is what makes the measurement affordable at all.

Ω̂ is banded Toeplitz with bandwidth L, so its Cholesky factor is banded too, and the factorisation costs O(nL²) rather than O(n³). At n = 120 and L = 8 that is about eight thousand operations against five hundred and seventy-six thousand. This runs once per candidate per draw inside every measurement in this field, so the difference is between a sweep and an afternoon.

It is checked against a textbook dense factorisation of the full matrix, entry by entry and on the log determinant, and required to agree to a billionth. An economy that changes an answer is not an economy, and the only way to know which one this is, is to compute it both ways.

What an availability rate does to a table #

The row that is not there is the hardest kind of defect to see in a comparison, because a table shows what it computed and has no column for what it declined to.

A truncated sample covariance is not positive definite on most draws, so on most draws there is no Cholesky factor, no whitening, and no number to put in the table. Every honest implementation then does one of two things: it drops the draw, or it falls back to something that always works. Dropping is the one that produces the reading here — at two lags the estimate survives on 1.0% of draws and its regret reads 0.00718, better than the rule that is told the true ρ. That is not a rule beating an oracle. It is a rule being scored on the 1% of draws whose residuals were well-behaved enough to admit a factorisation, which are the same draws every rule finds easy.

The selection is on the quantity being reported, which is what makes it worse than ordinary missingness. The draws where a truncated covariance fails to be positive definite are the draws where the sample autocovariances are noisy, and those are the draws where whitening has the most to do and does it worst. So the survivors are not a random 1%; they are the 1% chosen, by the estimator itself, to flatter it.

The other option is no better and is harder to notice. Falling back to least squares on the 99% and reporting a single number for the rule is reporting two rules averaged under one name, with mixing weights that depend on the design and are nowhere in the output. The average will look moderate. It will be moderate. It will not be a description of anything a reader could implement.

So the availability rate is not a caveat to the comparison; it is the comparison, and it is why this table carries the rate beside every rate of return. And the general form applies well past covariance estimation: any row of any table computed on a subset that the row’s own quantity selected needs its subset size printed next to it, or the row is a claim about a population nobody has named. The tapered windows need no such column, because they are positive semidefinite for every sequence there is and failed on none of three hundred draws.

What is claimed here, and what is not #

This essay takes what it costs to estimate a dependence rather than name it. The claims are that a truncated sample autocovariance sequence is not a covariance matrix on 64.7% to 98.3% of draws, so the rule built on it is undefined there; that a tapered sequence is one for every sequence and fails on none; that estimating the shape rather than the parameter costs 0.00672 of regret at 5.6 paired standard errors, which is slightly more than estimating the parameter rather than knowing it; and that estimating the covariance once for the table beats estimating it per candidate by 0.00367 at 4.3 paired standard errors.

What stays out and is named as a decision: any dependence the first-order parameterisation cannot represent. Every sample here is generated from an AR(1), so the rule with the parameterisation is correctly specified and the general rule is paying for a generality this design does not reward. The comparison that would show the general rule winning is a different design and is not made here.

The boundary against the essay that built the whitening is that it is about whether the fit or the penalty was the problem, and this one is about what happens when the fit’s repair has to be estimated without a form. What a criterion charges for an estimated nuisance is a trace, and what a fitted model’s own errors do to the same measurement is this field’s other half.

The checks, and the refusals that make them mean something #

Two claims are gated here. The banded Cholesky is required to reproduce a dense one on every entry and on the log determinant to a billionth, which is what makes an O(nL²) economy an economy rather than an approximation. And the ladder is required to be in order — knowing beats estimating a parameter, which beats estimating a shape, which beats least squares — because the whole argument is about the size of the gaps and a reversal anywhere in it would mean something else was happening.

Two refusals bite. A truncated Ω̂ used without asking whether it is a covariance matrix is rejected, with the availability rate named: a rule reported on the draws it survived is two rules under one name. And a general Ω̂ estimated from each candidate’s own residuals is rejected, because the protection the parameterised version relies on is cancellation, and a nuisance with a different value for every candidate does not cancel.