A sequence that knows it was centred

An interval whose scale comes from blocks has a block length to choose, and the published rule chooses it by reading the share of the mean’s variance that a block of that length would find. A share read off the sample tried what looked like the obvious improvement. The published rule reads that share from a first-order autoregression at the sample’s lag-one value, and so it assumes a shape. Reading the share from the sample’s own autocovariances assumes nothing, so it ought to survive series nobody modelled.

It did not, at a hundred and twenty rows. It covered 92.5% to 93.5% on all four laws, worse than the wrong shape on every one of them. The essay traced that to one identity. A sample autocovariance is computed around the sample mean, the deviations from which sum to zero, and so

$$\sum_{|k|<n} \hat\gamma(k) = 0.$$

A centred series’ autocovariances describe no variance at all. The further out a share reads, the more of that identity it inherits, the smaller the variance it reports, and the shorter the block it chooses. Read to lag six the rule covers 93.7%, and read to lag forty, 83.9%.

The closing section pointed out that an exact identity has a computable effect. To first order every lag loses the same amount, about the long-run variance divided by $n$. A sequence corrected by adding that amount back would not sum to zero any more, so its share would not drift towards one as it was read further out. Whether that restores the sample-read rule to the geometric rule’s coverage at 120 rows, keeps its advantage on the moving average at 480, and makes the cut-off lag the minor choice it was expected to be would say whether a share read off a short series can be trusted at all.

Two of those have a clear answer now. They are not the same answer.

The correction #

If every lag has lost about $L/n$, with $L = n\operatorname{Var}(\bar x)$ the long-run variance being estimated, then the raw window sum $S = \sum_{|k| \le K} w_k \hat\gamma(k)$ is short by $cL/n$, where $c$ is the sum of the window’s weights. The corrected sum is $L$ itself, so

$$L = S + \frac{c}{n}L, \qquad\text{that is,}\qquad L = \frac{S}{1 - c/n},$$

which is a fixed point with a closed form, and no iteration is needed. The corrected sequence adds $L/n$ to every lag up to the cut-off.

The correction is small on a short window. With a cut-off of six and a flat-top window, $c$ is about nine and $c/n$ is under 8%. At a cut-off of forty it is about sixty-one, and half the series. So a short read is barely changed and a long read is changed a lot. That is the right shape for what it is correcting, and it also says where the first-order account will start to give out.

Everything else about the rule stays: the same flat-top window, the same share, and the same rule for finding the shortest length whose predicted coverage reaches the target, now run on the same draws as the uncorrected rule, so any difference is what the correction made.

The lag stops mattering so much #

The figure at the head of this essay is the first answer. It shows coverage at 120 rows, against how far out the sequence is read, as it stands and corrected, on three of the four laws.

Read to lag forty, the correction recovers about five points on each law, and at lag twenty-four, about three. At six it does nothing measurable, as expected. On each law the spread between the best and worst lag narrows by about half. The earlier essay’s phrase was that on 120 rows the second choice “is most of the estimate”, and after the correction that is no longer true: a practitioner reading at twelve, or at six, or even at twenty-four, is within about three points of a fairly flat line.

So the third thing the closing section asked has an answer: yes, the correction does make the cut-off lag a more minor choice, and by a good deal. It does not make it a matter of indifference. At lag forty there is still a five-point shortfall, and that is on a sample of a hundred and twenty.

On the first-order autoregression at lag twelve, the corrected rule covers 94.5%, which is on the target the rule is aiming at, and it does so at an interval width of 1.13 against the geometric rule’s 1.21. So on that one law and cut-off, the correction delivers something real: about the level the rule was aiming for, at a narrower interval than the published rule’s. Its corrected share there reads 0.910 at a block a fifth of the series, against the law’s own 0.873 and the raw reading’s 0.926.

What it does to the belief #

The correction’s direct target is the share itself, and that can be read without any interval.

On the autoregression the law’s own share at a block of twenty-four is 0.873. The raw reading’s share runs 0.934, 0.926, 0.930 and 0.932 across the four lags: far too high, and insensitive to how far out it is read. The corrected reading’s runs 0.931, 0.910, 0.881 and 0.812. So at lag twenty-four it is almost exactly right, and at lag forty it has gone past the truth, to a share below the law’s. The first-order correction overcorrects when $c/n$ is large. Where the correction is big enough to matter it is a little too big, and where it is small it is a little too small.

The moving-average law reads the same way: truly 0.926, raw 0.936 to 0.948, corrected 0.933, 0.933, 0.894 and 0.826. It is right at the short lags, where this law has all its dependence anyway, and below the truth from lag twenty-four. So the corrected reading now falls below the share at long reads, on every law, and that is why coverage at lag forty does not quite recover: it is no longer a share that is too high, it is a noisier sum of more lags with a larger factor multiplying its noise. A correction that divides by $1 - c/n$ divides the window’s estimation noise by the same thing.

Under a break, the identity was never the main problem. The law’s own share at that length is about 0.51, and every sample reading is far above it. The raw share runs 0.833 to 0.919, and the corrected one 0.695 to 0.917. A break’s variance comes from a long-run shift that a hundred and twenty rows barely resolve, and no correction of order $1/n$ gets near it. The earlier essay’s account stands on this point: a short series hides the break, and it is still hidden.

Which rule this repairs #

The earlier essay’s question was about one rule in particular, and it is not the rule that read to a fixed lag. It is the rule that picks its own cut-off from the sample’s correlogram, so that a practitioner has one less thing to choose.

The correction does not restore the self-cutting rule. It moves it by between a tenth and three tenths of a point. The reason is in how that rule works. It cuts off where the correlogram first looks like nothing, and on these laws at this length that is a short window, so $c/n$ is small and there is little to correct. The rule’s shortfall is not the identity’s doing. Its share is too high for reasons that the identity contributes little to: at a short cut-off, it reads the part of the correlogram it can see, and it stops before the part it cannot. The same reasoning accounts for the rule reading to six, at 93.9% and hardly changed at 94.0%.

The geometric rule’s advantage in coverage is also not simply a matter of a better share. It assumes a shape, it is wrong about most of these laws in a known direction, and that error takes it towards longer blocks. On the moving average it believes a share of 0.782 against the law’s 0.926, and it chooses a block of forty where the oracle chooses thirty-one, so that it over-covers at 96.4% at a width of 1.19 against 1.01. It has good coverage because it is wrong in the conservative direction. On that account the fairer comparison is not with its 96.4% but with the target of 94.5%, and there the corrected reading to twelve is much closer, at less width.

So here is the honest summary. If a fixed cut-off is acceptable, a corrected reading to twelve is the best rule on this page at 120 rows: within a point of the target on all four laws and narrower than the geometric rule on the two without a break. Nothing here says twelve is special, except that it is where the correction has room to work and has not begun to overshoot, on a sample of this size.

At four times the length #

At 480 rows the correction’s size has fallen by a factor of four, as it should, and the rules have closed up. On the autoregression, the geometric rule covers 95.0% at a width of 0.466, the self-cutting rule 94.0% at 0.452, and corrected also 94.0% at 0.453. On the moving average, the geometric rule 96.0% at 0.487, the self-cutting rule 95.0% at 0.413, and corrected 95.2% at 0.414.

That is the second answer the closing section wanted. The self-cutting rule’s advantage on the moving average at 480 is still there: fifteen per cent narrower, at the level. The correction leaves it alone, because a long sample does not need it. And it confirms what the shape of the correction says. It is a term of order $c/n$, and its main use is to protect a short series against a long read.

Reading to forty at 480 rows shows what is left of the lag problem at that size: 91.3% as read and 93.3% corrected on the autoregression, 92.8% and 94.2% on the moving average. So the correction still pays at this length if the read is long enough for $c/n$ to matter, which here it is about an eighth.

None of this touches the two other ways of choosing a length. A length read off the whole sequence reads a plug-in from a variance’s expansion, which sums a few low lags and so has little of this identity to inherit, and the oracle row here is a reminder of what is available: on the moving average at 480 rows, a known share gives 94.7% at a width of 0.410, and the corrected self-cutting rule is already at 95.2% and 0.414. That is about as near as an estimated rule gets to knowing the answer, and at that length no correction is doing it.

Can a share read off a short series be trusted? #

The closing question was whether a share read off a short series can be trusted at all. The measurement gives a better-qualified answer than the essay expected.

Its dependence on the cut-off lag can be repaired, at no cost, by one line of arithmetic. That was an error with a known form and it can be removed. Nobody should now read a sequence to a long lag without it.

Its level on a shape it can see is about right after that. A corrected reading to twelve lands within a point of the rule’s own target on all four laws at 120 rows.

On a shape it cannot see, nothing here helps. A break in a hundred and twenty rows looks to the sequence like ordinary persistence, and a correction of order $1/n$ cannot supply what is not in the correlogram. The geometric rule’s coverage there is an accident in its favour, and the length nobody has is a warning that a block length on so short a series wanders by more than this correction moves anything.

And a rule reading its own cut-off does not benefit. That is a reason to prefer fixing the cut-off, if the correction is available.

This also leaves a question about the block weighted inside itself, since a tapered block’s own attenuation interacts with the same sequence, and a shrink towards the answer has shown that tapered windows have a separate issue about their level. The whole of this essay is on a rectangular overlapping scale.

How the numbers were made #

The four laws are the earlier essay’s: a first-order autoregression at 0.7, a five-term moving average, and two series with a break in the level, at 120 rows, with the first two also at 480. Fifteen hundred draws at 120 rows, and 600 at 480, run the rule on the raw and corrected sequences at four cut-off lags and at the self-cutting rule’s own, on the same draws, along with the geometric rule and the oracle holding each law’s true share. An interval’s scale is the overlapping block estimate at the chosen length, with its $t$ on the block scale’s nominal degrees of freedom. The standard error on a coverage near 94% is 0.6 of a point at 120 rows and about 1.0 at 480.

Four things must hold on the autoregression at 120 rows. The correction must recover more than three points at lag forty and more than two at twenty-four. At lag forty the raw share must lie above the law’s own and the corrected one below it. The spread in coverage across lags must fall to under three quarters of what it was. And the claim that the correction brings the self-cutting rule to within a point of the geometric rule is refused.

Still open: a second-order term, and a rule that needs no cut-off #

The overshoot. The correction assumes every lag has lost the same amount. It has not quite: a lag’s loss depends on how far it is from the origin, with a term in $k/n$ that a first-order argument drops, and an exact expectation of $\hat\gamma(k)$ under a stated shape is available. A correction using a lag-dependent amount would address the overshoot, at the cost of needing a shape to compute it from, which is what the sample-read rule was trying to avoid. Whether a one-step refinement, using the corrected sequence’s own shape, removes the overshoot at lag forty without reintroducing an assumption is a calculation that can be made here.

Choosing the cut-off for the corrected rule. The self-cutting rule picks a short lag because a raw sequence’s long lags cannot be read. A corrected sequence’s long lags can, a little, so a cut-off rule built for it might read further out and buy more on the moving average at no cost in the identity. That is a different rule, and it would not be surprising if it did better than a fixed twelve. It has not been written.

What the identity accounts for #

This essay takes the share of a mean’s variance read off a sample’s autocovariances, corrected by the fixed point $L = S/(1 - c/n)$. Its claims are that at 120 rows the correction moves coverage at a cut-off of forty from about 84.5% to about 89.5% across the laws, and narrows the spread between lags by about a half; that it brings the first-order autoregression’s reading to lag twelve to 94.5% on a target of 94.5%, narrower than the geometric rule; that it moves the believed share past the truth at lag forty, because a first-order correction overshoots at a large $c/n$; that it moves the self-cutting rule by at most three tenths of a point, because that rule’s cut-off is short; and that a break’s share stays out of reach at either reading.

What stays out, as a decision: long memory, where a sequence’s sum is not what this correction assumes and the essay on it declined to measure either; tapered scales; and other scales than the overlapping one. The boundary against a share read off the sample is that it found the identity and blamed it, and this one removes the identity and finds that it accounted for the lag but not for the rule.