A length read off the whole sequence

What choosing the length costs found that the block length is where the points are. Between two block windows at the best available length there are about two points of error in the implied variance of a mean; between that length and the best rule a practitioner could actually run there are about seven. The best rule was the plug-in, which reads the sample’s lag-one correlation and puts it through the algebra for a first-order autoregression.

That essay named the obvious improvement and set it aside: a rule that estimates the whole spectrum rather than one number off it. It would be a better estimate of the length the population wants, and it would be a second selection problem, because the part of the spectrum it reads has to be chosen too. Whether the first effect outweighs the second was left open, and so was the more basic question of when the plug-in’s one assumption matters at all.

This essay builds the rule and runs it under three laws: the field’s own first-order autoregression, a five-period moving average the plug-in misreads, and a long-memory law nothing in the field was designed for.

The plug-in’s one assumption, isolated #

The block-length algebra is general before it is specialised. A block estimate of the long-run variance has a bias that falls like one over the block length and a variance that grows like the block length over the sample, and balancing the two gives an optimum proportional to $(c\,n\,\alpha)^{1/3}$, where $c$ depends on the window and

$$\alpha = \Bigl(\frac{G}{\sigma^2}\Bigr)^{2}, \qquad G = \sum_k |k|\,\gamma(k), \qquad \sigma^2 = \sum_k \gamma(k).$$

$G$ weights the autocovariances by how far out they are and $\sigma^2$ is the long-run variance itself, so their ratio says how much of the dependence sits at long lags — which is what decides how long a block has to be to carry it.

For a first-order autoregression with coefficient $\rho$, $G/\sigma^2 = 2\rho/(1 - \rho^2)$ exactly, and that is the whole of the plug-in: estimate $\rho$ by the lag-one correlation and use the autoregression’s ratio. The general version estimates $G$ and $\sigma^2$ directly from the sample’s autocovariances, summed through a flat-top window — weight one out to half a cut-off lag, falling straight to zero at it — with the cut-off chosen by the sample. That is Politis and White’s automatic rule, and fed a known autoregression it returns the plug-in’s length exactly. It is the same formula with its assumption taken out.

The assumption bites when the lag-one correlation is a poor summary of the rest. A five-period moving average has correlations 0.8, 0.6, 0.4 and 0.2 at the first four lags and nothing after. Its $G/\sigma^2$ is 1.600. The plug-in sees 0.8 at lag one and assumes the geometric tail of an autoregression at 0.8, whose ratio is 4.444 — nearly three times too much dependence at long lags, and a block length about twice as long as the moving average wants.

Where each rule puts the length #

Under the autoregression the plug-in is where it should be: 14.9 against the 15.4 the law asks for at 120 rows, with a spread of 3.0. The whole-sequence rule is a little short, 11.2, and its spread is 6.4 — more than twice the plug-in’s, which is the price of estimating a run of autocovariances and a cut-off instead of one correlation.

Under the moving average the plug-in averages 20.4 against 10.7 asked for: twice the length, as the algebra predicted, with the same tight spread. The whole-sequence rule averages 13.4, near the target, with a spread of 11.1. At 960 rows the pattern is the same and sharper: the plug-in 42.1 against 21.4 asked for, the whole-sequence rule 21.3 with a spread of 4.8.

So the rule does the thing it was built to do. It removes the plug-in’s systematic error under a law the plug-in misreads, and pays for it in noise everywhere.

At a hundred and twenty rows that buys nothing #

The quantity a practitioner experiences is not the length but the error in the variance it produces, and that is where the gain disappears.

Under the moving average at 120 rows the plug-in’s error is 39.68% and the whole-sequence rule’s 40.11%, a difference of half a point at 0.4 paired standard errors: nothing. And neither is the best feasible rule. A block length of eight written into a protocol does better than both, at 34.22%.

Under the autoregression the whole-sequence rule is 45.33% against the plug-in’s 41.46% — 3.87 points worse, at 3.5 paired standard errors. It is paying the premium of its extra noise where the plug-in’s assumption is exactly true and gaining nothing back.

The reason is not that the target is wrong. Run every fixed length on the same draws and the best of them under the moving average at 120 rows is 10, at 32.88% — within a step of the 10.7 the algebra asks for. The asymptotic optimum is a good guide at this size. What the two estimated rules get wrong is different in kind and equal in cost.

The plug-in is wrong in the centre and steady: it lands near twenty every time, and a block length of twenty fixed in advance gives 39.29%, which is where the plug-in sits. The whole-sequence rule is right in the centre and unsteady: its lengths average near the target, but with a spread of eleven they run from two to thirty, and the draws at either end cost as much as the plug-in’s overshoot. Its error splits into a bias of 20.71 points and a spread of 34.40; the plug-in’s into 15.43 and 36.60; and the length of eight in a protocol, which has no length noise at all, into 25.55 and 22.80. At 120 rows the sequence rule exchanges a bias in the length for a noise in the length of the same price, and a fixed length close to the target beats both because it has neither.

These errors are measured against the sample mean’s actual variance at 120 rows rather than against the long-run variance the earlier essays used, because one of the three laws has no long-run variance. For the autoregression the two targets differ by about two per cent, and the plug-in’s 41.46% and the best length’s 33.90% here sit a point or two from the 43.07% and 35.81% the earlier essay reported against the other target, on other draws.

The premium falls and the payout grows #

The sample size changes both halves of the comparison, in the direction that favours reading the sequence.

Under the autoregression the premium is 3.87 points at 120 rows, 2.37 at 240, 2.48 at 480 and 1.15 at 960. Under the moving average the comparison is a tie at 120 — 0.43 — and then a gain: 1.72 points at 240, 3.01 at 480 and 3.77 at 960, the last at 6.3 paired standard errors.

Two things are moving, and they are the two costs the previous section separated. As the sample grows each autocovariance is estimated on more pairs and the cut-off is found more reliably, so the whole-sequence rule’s length noise falls — its spread under the moving average is 11.1 at 120 rows and 4.8 at 960 — and with it both its premium under the autoregression and its noise cost under the moving average. The plug-in’s error does not fall in the same way, because it is not noise: at 960 rows it still lands at twice the length the law wants, and the error curve has sharpened around its minimum. The best fixed length under the moving average at 960 rows is 16, at 15.79%; a length near forty, where the plug-in sits, is several points worse. At 960 rows under the moving average the plug-in, at 22.28%, is beaten by the rule of thumb’s ten at 19.74% and by the whole-sequence rule at 18.51%.

So the honest description of reading the sequence is an insurance policy. It costs a premium on every sample whose dependence happens to be geometric, and the premium shrinks with the sample; it pays out on samples whose dependence is not, and the payout grows. At 120 rows the policy is a bad buy. By a few hundred rows it is a good one under any law the plug-in misreads, and still a small loss under the one law it does not.

The rule’s own selection #

The whole-sequence rule replaces one assumption with one choice: the lag at which its window is cut. The cut-off is twice the first lag after which five consecutive autocorrelations sit inside a band of $2\sqrt{\log_{10} n / n}$ — the point after which the sample’s correlations are indistinguishable from noise.

Under the moving average the cut-off averages 8.3 lags with a spread of 6.0. The law has nothing past its fourth lag, so a cut-off near eight — window weight one to lag four, falling to zero at eight — is the rule finding what it should. Under the autoregression it averages 6.8, spread 4.3: the geometric tail falls inside the noise band by about the fourth lag at 120 rows, and everything past that is cut.

That is the second selection problem the earlier essay predicted, and it is where the extra spread comes from. A cut-off of two lags and a cut-off of fourteen give very different estimates of $G$, because $G$ weights the far lags by their distance, and the sample decides which of the two it gets by where its noisiest correlation happens to fall. Choosing a window from the data has the same shape in another field, and a dependence with a shape meets the same trade between a parameterisation that is wrong and a sequence that is noisy.

A law neither rule can serve #

Under long memory at $d = 4/9$ the comparison stops meaning anything, and the reason is worth stating because it is not about either rule.

The sample mean’s variance at 120 rows is $n\,\mathrm{Var}(\bar x) =$ 67.45 in units of the series’ variance — the correlations decay so slowly that they are still 0.58 at the twentieth lag, and they add up. Every rule’s error is more than ninety per cent of it, nearly all of that a shortfall: 97.50% for the plug-in, 96.90% for the whole-sequence rule, and 95.56% for the best length on each draw. The whole-sequence rule’s advantage is real, at 10.0 paired standard errors, and it is six tenths of a point inside an error of nearly a hundred.

What fails is the instrument the field measures. A block resample’s implied variance is built from the sample’s own autocovariances around its own mean, and under long memory those are attenuated so heavily that no block length on the grid — up to sixty-four — recovers more than a few per cent of the truth. And the window’s cut-off makes it worse rather than better: the long-memory law’s cut-off averages 9.9 lags, barely more than the moving average’s, because by the tenth lag the sample’s correlations are inside the noise band even though the law’s are not. A rule that reads the sequence cannot read past the point where the sample stops resolving it, and long memory is dependence that lives entirely past that point.

The covariance with no parameter reached the same wall from the other side, fitting a banded covariance where this field resamples. Neither construction is aimed at long memory, and neither should be quoted for it.

What a good fixed length is evidence of #

One reading of the 120-row table would be that estimating the length is not worth it at all, since a length of eight written into a protocol beat both estimated rules under the moving average. That reading does not survive the other law. Under the autoregression the same eight gives 44.84%, worse than the plug-in’s 41.46%, and the best fixed length there is 12 or 16, not eight. A fixed length is good exactly when it happens to be near the length the law wants, and the protocol writing it down does not know the law.

What the fixed length’s showing is evidence of is how much of the estimated rules’ error is the estimation. Under the moving average a length fixed at the target would have given 32.88%; the plug-in’s systematic error added about seven points to that, and the whole-sequence rule’s random error added about the same. At 960 rows the best fixed length gives 15.79% and the whole-sequence rule 18.51%: the estimation now costs under three points, where the plug-in’s assumption costs more than six. That is the same comparison the essay has been making, stated against the one benchmark a practitioner could in principle have — a length chosen in advance by someone who knew the answer.

What the field’s recommendation becomes #

The earlier essays left a recommendation in three parts: estimate the block length, use the taper, and do not believe the interval. This adds a qualification to the first part and nothing to the other two.

At a hundred and twenty rows, estimate the length with the plug-in. The whole-sequence rule’s noise costs more than its correction is worth there, under every short-memory law measured, and a fixed length of eight is competitive with both under a law with a short memory.

From a few hundred rows, read the whole sequence. The premium where the plug-in is right falls to about a point; the gain where it is wrong grows to several, and keeps growing.

And under long memory, use neither. The question is no longer which length but whether a block resample can represent the dependence at all, and on this measurement it cannot.

Still open: the sequence rule on the interval #

Everything here is measured on the implied variance, which is the instrument that makes four sample sizes and three laws affordable. The instruments field showed that the instrument a ranking is read on can reverse it: the taper wins on both readings a practitioner acts on at every rule, including two where the implied variance says it loses. The obvious next measurement is the same five rules read on a 95% interval’s coverage, under the moving average at 120 and 480 rows, where the variance says the two estimated rules tie at the first and separate at the second. If the coverage agrees, the recommendation above stands on the reading a practitioner acts on; if it does not, the variance is the wrong instrument for this choice, which would be the third time this line of fields has found that.

A second open question is the cut-off rule itself. Its constants — a band of two, five quiet lags — were set for estimating a spectrum, not for choosing a block length, and a cut-off tuned for the second job might keep most of the correction and lose much of the spread. That is a third selection problem stacked on the second, and it would have to be priced on draws it was not tuned on.

The claim, and its limits #

This essay takes what reading the whole autocovariance sequence is worth in choosing a block length, against reading the lag-one correlation alone. The claims are that the general block-length algebra fed the sample’s autocovariances through a flat-top window contains the plug-in exactly at a first-order autoregression; that under a five-period moving average the plug-in’s length is twice what the law asks for and the whole-sequence rule’s is near it, with more than twice the spread; that at 120 rows this buys nothing — 39.68% against 40.11% under the moving average, with a fixed length of eight better than both — and costs 3.87 points under the autoregression; that by 960 rows the premium is 1.15 points and the gain under the moving average 3.77; and that under long memory every rule, including the best length on each draw, is short of the sample mean’s variance by more than ninety per cent.

What stays out, named as a decision: other windows and other cut-off rules. The comparison is on the trapezoid, the field’s recommended window, and on Politis’s constants; neither is varied. And the interval, for the reason given above. The boundary against the length nobody has is that it compared the four rules under the law the plug-in was written for, and this essay asks what the plug-in’s one assumption costs under laws it was not.