What the correction assumes

The essay that proposed counting a band’s width in pairs ends by naming what it could not explain. Its correction is arithmetic — Σ w(k)(1 − k/n) instead of Σ w(k), no fitted parameter, computable before any data exist — and it is stated for a window in general. Measured on four windows, it repairs one. Its flattening factor is 10.008 for the plain Bartlett window, 2.111 for that window squared, 1.884 for its cube and 1.348 for Parzen.

A correction with nothing in it to fit cannot be right for one shape and wrong for three by having been fitted to the one. So something else decides, and this essay is about what. The four windows are the ones assembled so that a claim about windows would not rest on one window, which is why the failure is visible at all.

The correction is one number #

The first thing to do with an arithmetic correction is to write it as arithmetic.

Everything in this field is read as a rise above the narrowest band in the sweep, so what matters is not the weight a band of width $L$ carries but the weight it adds over the base. Write $W(L) = \sum_{k \le L} w(k)$ for the summed weights and $P(L) = \sum_{k \le L} w(k)(1 - k/n)$ for the pairs count. Then

$$\frac{P(L) - P(L_0)}{W(L) - W(L_0)} = 1 - \frac{\mu(L)}{n}, \qquad \mu(L) = \frac{\sum_{k \le L} k\,w(k) - \sum_{k \le L_0} k\,w(k)}{W(L) - W(L_0)}$$

and $\mu(L)$ is the mean lag of the weight the band adds. That is the whole of the correction. It multiplies the charge per unit of width by $1/(1 - \mu(L)/n)$ at every width, and it does nothing else at all.

The quantity is not the window’s mean lag. It is the mean lag of the increment, and for the plain Bartlett window at a base of one lag it comes out at 2.3333 at four lags and 11.0000 at thirty, against a nominal width of thirty. As a fraction of the width that is 0.3667, and the same reading for the squared window, the cubed one and Parzen is 0.2772, 0.2236 and 0.2567.

Those four numbers are the correction. A window that piles its weight into the short lags adds weight the sample knows a great deal about, so there is little to discount and $1 - \mu/n$ stays near one; a window whose weight reaches out to long lags gets a larger discount. Squaring a Bartlett weight pulls it towards the short lags, and cubing it pulls it further, which is why the three sit in the order they do.

What it supplies, against what has to be accounted for #

Across the plateau — four lags to thirty — the correction multiplies the profile’s fall by a factor that is entirely determined by those mean lags:

$$\text{supplied} = \frac{1 - \mu(4)/n}{1 - \mu(30)/n}$$

On a hundred and twenty rows that is 1.07951, 1.05798, 1.04556 and 1.05392 for the four windows. It reads no data; it is four divisions.

What has to be accounted for is the reciprocal of the fall the measurement actually shows. The charge per unit of summed weight falls from four lags to thirty by a factor of 0.9363 on the plain window, and the corresponding reciprocals for the four are 1.06800, 1.23819, 1.17105 and 1.60049.

Set the two columns beside each other and the answer is immediate. The correction accounts for 1.0108 of what the plain Bartlett window needs, 0.8545 of the squared window’s, 0.8928 of the cube’s and 0.6585 of Parzen’s.

The two orderings are opposite, and nothing connects them #

The table above is not a story about a correction that is slightly too small for three windows. It is worse than that, and the reason is in the ordering.

The correction is largest for the plain window and smallest for the cube, because that is what the mean lags say. The curvature to be accounted for is largest for Parzen and smallest for the plain window, because that is what the optimism sweep says. The two are ordered opposite ways, so the window that gets the biggest correction is the window that needs the smallest one.

There is no mechanism linking them, and that is the point rather than an oversight. $\mu(L)$ is a property of where a window puts its weight, computable with no data in the room. The fall in the measured charge is a property of how the optimism of a plug-in covariance grows as the band widens, which is a fact about the likelihood, the law and the sample. Two quantities can be ordered opposite ways and still cross somewhere, and the plain Bartlett window is where these two cross.

Stated that way the finding is not that the correction works on one window. It is that the correction has a size, the curvature has a size, and on one of the four they are the same size. Everything the earlier reading called a repair is that coincidence.

The plateau was stated in one window’s units #

Before accepting the coincidence there is a range problem to clear, because it could produce the whole table on its own.

This field’s plateau starts at four lags, and the reason it starts there is a measurement: at two and three lags the charge per unit of width is above the plateau on both scales, and the correction is nearly nothing there because $1 - k/n$ is nearly one. That reasoning is sound, it is reported in every figure rather than buried, and it was made on the plain Bartlett window — the one the essay that first read the profile measured.

Four nominal lags is not the same band for the four windows. In summed weights it is 2.000 for the plain window, 1.200 for the squared one, 0.800 for the cube and 1.376 for Parzen. So four lags of the cubed window is a narrower band than two lags of the plain one, which is a width this field excludes from its own plateau for being dear. Three of the four windows are being read partly on the very region the plateau was defined to exclude.

The repair is to state the range in the units the correction is a function of. Every one of the four windows has exactly five widths on the existing grid whose effective width lies between 2 and 8 summed weights — the plain window at 4, 6, 8, 12 and 16 lags, the squared one at 8 to 24, the cube at 12 to 30, Parzen at 6 to 20 — so the four can be read over the same range of the same quantity at the same number of points, off the same measurements, with nothing re-simulated.

What the matched range shows, which is not what it was expected to show #

The matched range does move the table, and it moves it in a direction nobody was looking.

On the five widths from two to eight summed weights, the plain Bartlett window’s charge per unit of width sits 0.0349 from a constant, against 0.6323 over the whole plateau. Its residuals there are +0.33, +0.12, +0.07, +0.04 and −0.22 standard errors — a profile a constant describes to within a third of one standard error at every point. And the correction’s flattening factor over that range is 1.016 rather than 10.008.

So the curvature the whole field was built to explain is not spread across the plateau. Over the full range the residuals run +0.64, +0.54, +0.61, +0.82, +0.79, +0.53, −0.01, −1.56, and the last of those is the thirty-lag band on its own. The three widest bands carry 53.3% of the total misfit between them, and they are the three the other windows never reach: the cubed window’s widest band anywhere on the grid is 7.26 summed weights, which is inside the range where the plain window’s profile is already flat.

That reframes the original finding without rescuing it. The correction is worth a factor of ten on the plain window because the plain window is the only one that reaches the widths where there is anything to correct. Read over the range the four share, the four flattening factors are 1.016, 2.385, 2.529 and 1.292 — the spread across windows falls from 7.42 to 2.49, so matching the range does bring them nearer each other, and it does so by taking the outlier away from the window that was supposed to be the success.

The other three are not repaired either. On the matched range the squared window still sits 0.6482 from a constant on the summed weights, the cube 0.4706 and Parzen 2.8440. Matching the range relocates the finding; it does not generalise it.

Is the form right and only the size wrong? #

There is one more way the correction could be substantially right, and it is worth testing rather than dismissing, because it is the reading a sympathetic reader would reach for.

Suppose a lag’s contribution to the optimism really is discounted by how few products went into it, and only the strength of the discount is mis-stated. Then the profile would be described by

$$P_p(L) = \sum_{k \le L} w(k)\,(1 - k/n)^p$$

for some exponent $p$, with $p = 1$ the pairs count and $p = 0$ the summed weights. A single mechanism mis-stated in size gives one exponent for all four windows. Four different mechanisms give four.

Fitted to the same measured rises, with the rate profiled out at each exponent and each width weighted by the reciprocal of its own squared standard error, the four windows want 0.792, 2.749, 3.564 and 4.893. Read over the matched range instead they want 0.499, 2.695, 2.671 and 7.607.

They do not agree, and they do not come nearer to agreeing when the range is matched — the spread goes from 4.102 to 7.107. A discount whose exponent has to be six times larger for one window than for another is not one discount applied with the wrong strength. It is a curve being fitted, which is the thing the essay that put four shapes against each other already priced, arriving here from a different direction and reaching the same place.

The plain window’s own exponent is worth a sentence. At 0.792 it is below one, which says the pairs count slightly overshoots on the plateau — and it is the same overshoot the flattening reports as a fall of 1.0108 rather than of exactly one.

The sample size, which was named as a prediction #

The strongest test available is the one the field that priced these charges left explicitly as a prediction rather than a measurement: the correction’s size is $1/n$ and nothing else, so shortening the sample should make it larger, and the curvature it accounts for should grow to match.

Halve the sample. At sixty rows the correction supplies 1.17687 on the plain window against 1.07951 at a hundred and twenty — a little more than twice the effect, exactly as $\mu/n$ requires. The curvature grows too: the charge per unit of summed weight now falls by a factor of 0.7517 across the plateau, so what has to be accounted for is 1.33027.

But it grows faster. The correction accounts for 0.8847 of the plain window’s fall at sixty rows, against 1.0108 at a hundred and twenty, and for 0.8174, 0.8507 and 0.7039 of the other three. At sixty rows no window is repaired, the plain one included.

That is the measurement the whole essay turns on. If the coincidence at a hundred and twenty rows were a mechanism, it would survive a change of $n$, because both sides of it would move together. They do not: the correction’s size moves as $1/n$ and the curvature moves faster, so the crossing is a point rather than a line. The correction is the right size for one window at one sample size.

The three assumptions, and which one fails #

It is now possible to say what the correction assumes, which was the question.

That a lag’s contribution to the optimism is proportional to the weight the window gives it. That is the derivation of the summed weights and it is exact for the object it is about: a plug-in tapered autocovariance is a linear shrinkage, and the effective dimension of a linear shrinkage is the sum of its derivatives. Nothing here disturbs it.

That the contribution is then discounted by the share of the sample’s products the lag was built from. That is the arithmetic above, and it is checked as arithmetic — against a closed form at every width and every sample size, and against a call that reads no data.

That the constant of proportionality is the same at every lag. This is the one nothing has ever checked, and it is where the correction fails. If the optimism per unit of discounted weight were the same at every lag, the four fitted exponents would be one number; they span a factor of six. Something about a long lag makes it dearer, or cheaper, than its own discounted weight says, and the amount depends on the shape of the window in a way $\mu(L)$ does not capture.

The reading a practitioner should take is unchanged in direction and much narrower in scope. Counting a band’s width in pairs is the right correction to make and it is not large enough to be the whole of the curvature. It moves every window and every law the right way — that much the sweep across the four laws already establishes — and on one window at one sample size it happens to land exactly.

What a coincidence costs, in the currency anybody spends #

None of this changes the width a charge picks by very much, and it is worth saying so before the qualification is read as an alarm.

The essay that priced the four derived charges finds them picking widths within a few per cent of each other and delivering errors within two per cent of the gap any of them leaves. A correction whose size is right on one window and eighty-five per cent right on the next is a difference well inside that. What is at stake is not the width; it is what a reader is entitled to conclude from the agreement, and the field that measured how little the width decides is the reason that distinction is worth drawing at all rather than a reason to stop drawing it.

The difference matters in one specific place. A correction that is a mechanism transports — to another window, another sample size, another grid — and a correction that is the right size once does not. The claim available at the end of the essay that proposed it was the first kind, and the measurement here says it is the second.

That is also why the exponent fit is in this essay rather than left as a remark. It is the only test on the table that could have distinguished the two, because it asks whether the four windows are describing one thing badly or four things each in their own way, and it answers plainly.

What the shapes of the windows still do not explain #

Two things this essay names and does not settle.

What makes Parzen different in kind. Parzen is the outlier under every reading here: it needs 1.60049 where the correction supplies 1.05392, it sits 2.8440 from a constant on the matched range against the other three windows’ 0.0349, 0.6482 and 0.4706, and its charge per unit of width is above one at the narrow end where every other window’s is below. Its mean added lag, 0.2567 of its width, sits between the squared window’s and the cube’s, so the quantity that orders the correction does not order Parzen’s behaviour at all. Whatever separates it is not the mean lag and is not measured here.

And what the rate is a property of. Read over the matched range, the charge per unit of pairs comes out at 0.8067, 0.8300, 0.8231 and 0.9658 — three windows agreeing to within three per cent and one nineteen per cent above them. A field that could say why the first three agree would be able to say why the fourth does not, and this one can say only that they do.

What stays out #

Three things this essay could have measured and did not.

A grid wide enough to put every window on the same effective widths at both ends. The matched range is two to eight summed weights because that is the widest band the existing grid gives all four windows five points in. Taking the cubed window out to fifteen summed weights would need bands of about sixty lags on a hundred and twenty rows, which is half the sample, and the comparison would then be between a band and something that is no longer a band.

A second law for the window comparison. Everything above is a first-order autoregression. The sweep that varied the law shows the correction improving every one of the four and finishing the job in one, but it varies the law on one window; the window-by-law grid is not measured, and the claim that the ordering of the four windows is the same under long memory is a guess.

And a third sample size. Two points establish that the curvature grows faster than the correction and cannot say how much faster. A third would turn the direction into a rate, and the rate is what a reader who wanted to know when the coincidence recurs would actually need.