The residuals are not the errors

Every resampling in this subject starts by fitting something and keeping what is left. The residuals stand in for the errors, get shuffled or multiplied or blocked, and are added back to a fitted null to produce a sample that is supposed to look like one the world could have produced.

The substitution is stated as an approximation and is usually discussed in terms of size: residuals are a little smaller than errors because a fit absorbs some of them, the shrinkage is 1 − h for the row with leverage h, and the standard repair divides each residual by √(1 − h). That repair is correct about size and this essay is about something else. A fit does not take a uniform bite out of the errors. It takes the part of them that looks like its own columns, and when the columns are slow the part it takes is the slow part.

What a projection removes #

Write the residuals as (I − H)ε. If H were a scalar multiple of the identity, the residuals would be a scaled copy of the errors and would have exactly the same autocorrelation. It is not: it is the projection onto the candidate’s column space, so what it removes is whatever component of ε happens to lie in that space.

For a design of independent rows the column space points in no particular direction relative to the errors’ shape, and the residuals inherit the errors’ autocorrelation nearly intact. For a design whose columns are persistent, the column space is a low-frequency subspace, and the low-frequency part of ε is exactly what gets projected out.

So the residuals are smoother than the errors, and increasingly so at longer lags, because the component being removed is the one that survives longest.

Measured on a hundred and one rows with the design and the errors both autoregressive at 0.7, the errors have a first-lag autocorrelation of 0.6622 and the residuals 0.6234. That is a shortfall of six per cent, which does not sound alarming. The lags after it are the point:

By lag three the errors are at 0.2775 and the residuals at 0.2257 — a shortfall of nineteen per cent. By lag five, 0.0988 against 0.0607, a shortfall of thirty-nine per cent. By lag six, 0.0509 against 0.0210, a shortfall of fifty-nine per cent. The residual series does not merely carry less dependence; it carries dependence with a shorter memory, which is a different defect and matters more.

It is worth putting a number on how much of the shortfall the projection accounts for. The benchmark being fitted here has an intercept and one column, so it removes a two-dimensional subspace from a hundred and one rows — a fortieth of the degrees of freedom. That is what produces a fifty-nine per cent shortfall at lag six. The effect is not proportional to how much the fit takes; it is proportional to how well aligned what it takes is with what is being measured, and a slow column is perfectly aligned with a slow autocovariance.

The corollary is that a larger benchmark does not have to be worse. What matters is whether the extra columns are persistent. Adding a white-noise regressor to the benchmark removes another dimension and takes almost no dependence with it; adding a near-unit-root one removes another dimension and takes a great deal. The number of parameters is not the diagnostic. The persistence of the columns is.

The repair that repairs the other thing #

Dividing each residual by √(1 − h) is the standard correction, and everything about the way it is described suggests it should help here. It is derived precisely from the fact that a projection removes something, it is applied precisely to the residual it removed something from, and it restores exactly the variance a residual is missing.

It does nothing to the dependence. The rescaled residuals have a first-lag autocorrelation of 0.6210 against the plain residuals’ 0.6234 and the errors’ 0.6622, and the same at every other lag: 0.3822 against 0.3844, 0.2240 against 0.2257, 0.1248 against 0.1257. It moves the number in the wrong direction by a thousandth.

The reason is that leverage is a per-row quantity and dependence is a between-row one. Scaling row t by a factor that depends on row t alone cannot put back a correlation between t and t + 3; only something that knew what was removed from both rows could, and h_t does not. The correction is exactly right about the thing it was derived for and silent about the thing that looks like it.

This is the shape of finding this collection keeps running into: a correction that is precisely right about a quantity, carried one step to a neighbouring quantity because the derivation reads as though it were about both.

It is worth being clear that the correction is not merely unhelpful here — it is unhelpful and harmless, which is the combination that keeps it in use. It restores the residuals’ variance, so any statistic that is a function of their scale is improved by it; it leaves their autocorrelation exactly where it was, so any statistic that is a function of their dependence is neither improved nor damaged. A practitioner applying it to a blocked resampling has done something correct and has not touched the thing the blocking is for.

An alignment factor that grows with the lag #

The essay’s point that the effect is about alignment rather than about how much a fit takes can be sized, by comparing the shortfall with what dimension counting alone would predict.

A projection removing two of a hundred and one dimensions should, if it were pointing nowhere in particular, take about 2% of anything. The shortfalls it actually produces are 6%, 19%, 39% and 59% at lags one, three, five and six.

So the alignment factor — how much more the projection takes than its dimension count says — runs 3, 9.5, 19.5 and 30 across those four lags, rising nearly in proportion to the lag itself.

That is the mechanism as a rate rather than as a description. A slow column is aligned with the slow part of the errors, and the further out the lag the more exclusively the surviving autocovariance is that slow part — so the same two-dimensional bite takes three per cent of the first lag and thirty per cent of the sixth, out of a subspace that has not changed.

It also gives the corollary about column persistence a form a practitioner can apply. Adding a column does not add 1/n of shortfall; it adds 1/n times an alignment factor that is a property of the column’s own spectrum against the lag being read. A benchmark’s cost to a dependence measurement is its dimension times its slowness, and only the first of those appears in any model summary.

The leverage correction supplies minus six per cent of the repair #

The correction is reported as moving the number by a thousandth in the wrong direction, and putting that beside the size of the hole says how completely it misses.

At the first lag the residuals are 0.0388 below the errors. The rescaling moves them a further 0.0024 down. So the standard repair supplies −6.2% of the repair required — it is not a partial fix, it is a small step away from the target.

The block-length consequence is the same arithmetic at the end of the pipeline. The rule asks for 5.47 where the errors ask for 6.30, a shortfall of 13% at eight standard errors — and since a block length is an integer, 5.47 rounds to five and 6.30 rounds to six. The systematic thirteen per cent becomes a whole block, which is the unit the parameter is actually spent in, and a whole block at these lengths is a fifth of the dependence a resample keeps.

That is the practical residue. Not that the residuals are a poor stand-in for the errors in general — at the first lag they are within six per cent — but that the one quantity a practitioner reads off them is read at a lag where the shortfall is large, and is then rounded to a grid coarse enough to turn the shortfall into a decision.

The parameter this actually breaks #

A resampling that keeps dependence has a length in it. A block bootstrap takes runs of ℓ consecutive residuals; a blocked multiplier draws one sign per run of ℓ rows. Either way ℓ decides how much dependence survives, and choosing it is a real decision with an interior optimum — too short and the dependence is thrown away, too long and the reference distribution is built from too few independent things.

Nobody chooses ℓ by looking at the errors, because nobody has the errors. The rule of thumb is to look at where the autocorrelation of what is available has decayed, and what is available is the residuals.

Take the smallest ℓ whose lag-ℓ autocorrelation is under a tenth as the rule. From the errors it gives 6.30. From the residuals it gives 5.47. The gap is 0.835, with a standard error of 0.105 on the paired difference — eight standard errors, so it is not an artefact of the particular threshold or the particular sample.

Under a tenth of a block, in a parameter where the essay that measured its cost found the bias at ℓ = 1 to be twice the bias at ℓ = 5. So the size of the error is not the alarming part either. The alarming part is the direction: the residuals understate the dependence, so the rule asks for a shorter block, so the resampling keeps less dependence, so the reference distribution falls further short of the truth — which is the failure the block length was being tuned to reduce.

Every step of that chain is small and every step points the same way.

What a shorter block costs, in the units the field already has #

The chain above ends in a reference distribution that falls further short, and it is worth converting that into the field’s own currency rather than leaving it as a direction.

A block length is a trade between two errors. Short blocks throw dependence away, which biases the reference distribution’s critical value downwards; long blocks keep it and leave the distribution to be built from ⌈n/ℓ⌉ independent signs, which makes its own quantiles noisy. The essay that measured the trade found the bias falling from 1.6885 at ℓ = 1 to 0.8479 by ℓ = 5 and then stopping, and the spread rising monotonically from 1.3073 to 2.1716 across the same sweep.

So an error of 0.835 in ℓ lands in the steep part of the bias curve rather than the flat part — between five and six is precisely where the bias has stopped falling and the spread has not yet risen much, which is why the optimum is there. Moving off it by five-sixths of a block is not catastrophic and is not free, and it is a systematic move rather than a noisy one: every practitioner using the same rule of thumb on the same kind of series makes the same error in the same direction.

Why this is not the whole shortfall #

It would be neat if this were the explanation for the residue that the blocked multiplier leaves, and it is not. The essay that splits that residue takes the residuals away entirely — hands the same construction the true errors, which exist because the null being resampled under is β = 0 — and the reference distribution gets further from its own truth, not closer.

That is worth stating here rather than leaving to the next essay, because it is easy to read this one as an accusation and it is not. The residuals’ smoothness is real, it is measurable, it biases the tuning parameter in a known direction, and it is not the reason a multiplier resampling falls a quarter short of the truth. Two true statements about the same object, one of which explains something and one of which does not, and telling them apart takes a measurement rather than a mechanism.

What the smoothness does explain is a ceiling. A multiplier resampling produces something whose autocovariance is the residuals’ multiplied by the multiplier’s own, so the residuals’ dependence is an upper bound on what any of these constructions can carry — and the residuals’ is below the errors’ at every lag by the amounts above. 91.3% of the errors’ first-lag dependence is the most that any choice of ℓ could reach, before the block length has been chosen at all.

There is a second reading of the ceiling worth having, because it says which of the two shortfalls is worth attacking. The residuals reach 91.3% of the errors’ first-lag dependence; the best block length measured reaches 87.1% of the errors’, which is 95.4% of the residuals’. So of the 12.9% that is missing at the first lag, 8.7 points are what the fit took out before any resampling started and 4.2 points are the taper a finite block length imposes. The second is a dial with a known optimum. The first is not a dial at all, and it is the larger of the two.

That reverses the natural order of attention. A practitioner tuning a block length is working on the smaller of the two gaps, and the larger one is fixed by the choice of benchmark rather than by any tuning parameter — which is a choice made for other reasons entirely, usually before anybody has thought about a reference distribution.

What a fit takes, in one sentence #

The general statement is worth separating from the arithmetic, because it applies to more than resampling.

A residual is what is left of an error after the part of it that resembles the model has been removed. Everything a residual is used for inherits that, and the inheritance is invisible whenever the model’s columns have no particular shape. On independent rows, a fit’s columns are a random direction and a residual is a faithful if slightly shrunken copy of an error. On a persistent design the columns are not a random direction, and the copy is faithful about its size and systematically wrong about its memory.

Diagnostics built on residuals are the obvious other place this matters. A test for serial correlation in the errors, run on the residuals of a fit whose regressors are themselves serially correlated, is run on a series that has had some of the correlation taken out of it — and it is a test for the presence of a thing, so it fails in the direction of finding less.

What is claimed here, and what is not #

This essay takes what a fit does to the dependence of what it leaves behind, and the claims are that the residuals are less persistent than the errors at every lag the errors have anything at, that the shortfall grows with the lag from six per cent at the first to fifty-nine per cent by the sixth, that the leverage correction moves it by a thousandth in the wrong direction, and that a block length chosen from residuals comes out 0.835 shorter than one chosen from errors, eight standard errors apart.

What stays out and is named as a decision: any repair for the dependence a fit removes, because the obvious candidates are per-row and the defect is between rows, and a construction that knew what had been removed from two rows at once would need Ω — which is the thing being estimated; and the question of whether this is the shortfall a blocked multiplier leaves, which is measured in the next essay and answered no.

The boundary against the field that built the table of resamplings is that it asks which defect each resampling survives and this one asks what the residual it is handed is missing before any resampling happens.

The checks, and the refusals that make them mean something #

Two claims are gated in this field’s library. The residuals are required to be less persistent than the errors at every lag where the errors still have anything, which is a claim about a mechanism rather than about an average and would fail if it were a coincidence of one lag. And the block length chosen from the residuals is required to be shorter than the one chosen from the errors by more than two standard errors of the paired difference.

The refusal for this essay is a block length read off the residuals. It is not a hypothetical: it is what every practitioner does, because the errors are not available, and it asks for a shorter block than the dependence needs on the one series guaranteed to be too smooth.