Residuals that keep their own variance

The reference distribution for a search over a table has to be generated rather than resampled from the differentials. An earlier field measures why: resampling rows of the observed loss differentials recentres each column at its own sample mean, which under a null with a displacement in it is the displacement rather than the boundary, and the procedure that was supposed to be conservative stops rejecting anything at all.

Generating means simulating from a fitted model, and simulating from a fitted model means producing errors. The construction that field uses draws the benchmark’s residuals one at a time and with replacement, which is the obvious thing to do and throws away everything about them except their marginal distribution.

This essay is about what that costs, and the answer is not the one the reflex says.

Four resamplings, and what each keeps #

One at a time. Draw residuals independently with replacement. The marginal distribution survives — its spread, its shape, its outliers — and every relationship the residuals had with anything else is destroyed.

Runs of neighbours. Take blocks of consecutive residuals and lay them end to end. Whatever correlation exists between an error and the one before it survives, and so does clustering in their sizes; the tie between a residual and the row it came from does not.

Its own row, signed. Multiply each residual by an independent ±1 and leave it where it is. The residual stays attached to its own predictors, so a variance that is a function of the design survives exactly; the correlation between neighbouring errors is destroyed, because independent signs uncorrelate anything; and so is the skewness, because a symmetric multiplier symmetrises everything.

Its own row, two-point. The same, with Mammen’s multiplier — the two roots of the golden ratio, weighted so that the multiplier has mean zero, variance one and third moment one. It keeps everything the signed version keeps and the skewness as well.

Nothing keeps all four properties. Which one is right is a statement about the world, and the world is what a bootstrap was reached for in order not to have to state.

Three properties, and no construction holds all three #

The four descriptions above are four combinations of the same three things, and setting them out that way says what is on offer and what is not.

A residual carries a marginal distribution — its spread, its shape, its outliers. It carries a tie to its neighbours, which is whatever serial correlation the errors had. And it carries a tie to its own row, which is the design point it was produced at and therefore whatever variance that row was entitled to.

Drawing one at a time keeps the first and destroys the other two. Drawing runs of neighbours keeps the first two and destroys the third. Multiplying in place by ±1 keeps the third and destroys the second — and does something to the first that the others do not: a symmetric multiplier forces the resampled marginal to be symmetric, so the residuals’ third moment goes with it. A multiplier shared along a block keeps the second and the third, at the same cost to the first.

So the three properties are not independently purchasable. No construction here keeps all three, and the one that comes closest buys the two ties by giving up the shape, which is exactly the trade the two-point multiplier later in this essay exists to soften.

That is worth having before any of them is measured, because it says what a comparison between them can possibly be about. It cannot be about which construction is more faithful in general. It is about which of three fidelities the statistic in hand actually reads.

The statistic that makes the choice matter #

There is a version of this comparison in which none of it matters, and it is worth disposing of first.

If the statistic is studentised — a differential divided by its own long-run standard error — then it is already approximately pivotal, and the reference distribution’s job is to supply the multiplicity and the dependence between comparisons, not the scale. On this table all four resamplings do that about equally well.

White’s reality check is not stated that way. Its statistic is the largest mean differential, √P·d̄, with no denominator. That is deliberate: on a set of many candidates the studentisation is itself noisy and can produce a maximum that is large only because one column’s variance was badly estimated. The price is that the reference distribution has to supply the scale as well, and then what the resampling keeps about the errors’ variance is the whole of the test.

Everything below is about that statistic, on the six two-predictor subsets of four predictors, whose least favourable configuration is attained by construction so that a measured rejection rate is a measurement of the reference distribution and of nothing else.

Where each one goes wrong #

The sharp measurement here is not the rejection rate — at a hundred and twenty draws each cell carries about 2.0 points of counting error — but the critical value, which is an average over draws and is precise. Each reference distribution produces a 5% point; the statistic has a true 5% point across the same draws, which no procedure has to estimate because these are simulations of a known world. The gap between them is the mechanism, and the rejection rate is its noisy consequence.

Errors of constant variance and symmetric shape. The truth’s 5% point is 1.167 and all four resamplings put theirs above it: 1.373, 1.351, 1.259 and 1.324. All four are conservative, none is wrong, and the choice makes no difference worth the name.

A variance that is a function of the design. The truth’s 5% point moves to 2.132, because a row with a large predictor value carries a large error and the maximum over a table is dominated by those rows. The two resamplings that detach a residual from its row do not follow: they land at 1.556 and 1.535, short by 0.576 and 0.598. The two that keep it attached land at 1.965 and 2.119 — the second is short by 0.014, which is a seventh of one per cent of the value it is estimating.

Skewed errors. The truth’s 5% point is 1.283. Here the ordering half-reverses: drawing residuals one at a time keeps the skewness and the symmetric multiplier throws it away, so the signed version is at 1.329 and the two-point version at 1.384 against the iid draw’s 1.437. But every one of the four is above the truth, so all four are conservative and the defect costs size rather than level.

The block bootstrap repairs nothing here, and that is the finding #

Three earlier fields on this site reach for a block bootstrap, and they are right to: the loss differentials from overlapping forecast origins are dependent, the dependence is over time, and a block resampling is what carries it.

Here it is worthless. Its 5% point tracks the iid draw’s to within 0.022 in the world where the variance depends on the design and to within 0.009 where the errors are skewed, and in the heteroskedastic world it rejects where the signed multiplier does not on 5.8% of draws and never the other way round.

The reason is not that block resampling is a bad idea. It is that the defect it repairs is not the defect that is present. Dependence between origins arises here from the overlapping estimation windows, and the generated reference distribution reproduces that on its own — it re-runs the whole rolling scheme on each simulated series, windows and all, so the correlation between origin thirty and origin thirty-one is built rather than resampled. What is left over for the resampling to carry is the relationship between a residual and its own row, and a block bootstrap destroys exactly that.

A repair inherited from a neighbouring problem is not a repair. This one has been correct everywhere else it has been used on this site and is worth nothing in the one place its cousin is needed, and no gate would have found that: it never fails, it is simply conservative in a world where something else is anti-conservative.

Why the multiplier is the only one that can carry it #

The identity underneath the middle column is one line, and once it is written the result stops being a horse race.

A wild bootstrap sets εₜ = êₜ·vₜ with vₜ independent of everything and E[v] = 0, E[v²] = 1. Then conditional on the data, E[εₜ² | row t] = êₜ², which is an estimate of σ²(xₜ) — the conditional variance at that row, not the average variance. So the simulated world has a variance function, and it is the one the residuals report.

An iid draw sets εₜ = $\hat{e}_{J(t)}$ with J uniform, so E[εₜ² | row t] is the average of êₜ² over all rows, whatever row t is. The simulated world is homoskedastic by construction. A block draw is the same statement with a little local memory.

That is why the two multipliers succeed and the two draws fail, and it also says where the multipliers will fail: whenever the thing that matters about the errors is a relationship between rows rather than at a row. Serial correlation in the levels is the obvious case, and a wild bootstrap destroys it completely.

So the honest summary is a table rather than a winner, and the practical rule is the boring one: resample in the direction the errors are structured in. What makes it hard is that nobody knows which direction that is, which is why the reflex to reach for whichever bootstrap the last problem used is so strong.

What the rejection rates say, once their error bars are on #

The critical values are the mechanism and the rejection rates are what a practitioner is actually exposed to, so it is worth putting the second beside the first with the counting error attached.

In the world where the variance depends on the design, at a nominal 5%, the four rates are 5.8%, 8.3%, 3.3% and 2.5%, each carrying about two points of error. Read one at a time those bars are wide enough that only the extremes separate. Read paired — every resampling run on the same draws, so the difference between two of them is a matched quantity — they are unambiguous: the block resampling rejects where the signed multiplier does not on 5.8% of draws and the signed multiplier never rejects where the block one does not. The iid draw beats the multiplier the same way on 2.5% and is never beaten. There is no draw in the whole experiment on which the ordering runs the other way.

That is the right way to read a comparison of procedures on shared draws, and it is worth stating because the temptation is the opposite: to run each procedure on its own simulation, report two rates with overlapping intervals, and conclude that the difference is not established. The difference is established; it was the design of the comparison that was hiding it.

In the world with skewed errors the same paired reading gives the mirror image at a smaller scale: the iid draw rejects where the signed multiplier does not on 3.3% of draws and never the reverse, which is the skewness the symmetric multiplier threw away, arriving as size.

The two-point multiplier, and what its third moment is for #

Mammen’s multiplier is the least familiar of the four and the one this essay ends up recommending, so it is worth saying what it is and why it has the form it has.

It takes two values: −(√5 − 1)/2 ≈ −0.618 with probability (√5 + 1)/(2√5) ≈ 0.724, and (√5 + 1)/2 ≈ 1.618 with the remaining probability. Those numbers are not aesthetic. They are the unique two-point distribution with mean zero, variance one and third moment one, and the third condition is the whole reason to prefer it to a coin.

A wild bootstrap’s simulated error is êₜ·vₜ, so its third moment at row t is êₜ³·E[v³]. With a symmetric multiplier E[v³] = 0 and every simulated error distribution is symmetric, whatever the real ones look like. With Mammen’s, E[v³] = 1 and the simulated third moment is êₜ³ — the residual’s own skewness, kept in place along with its size.

The cost is that the multiplier is not symmetric, so a single simulated error is a lopsided thing; the benefit is visible in the third column of the table above, where the symmetric version’s 5% point is 1.329 against the two-point version’s 1.384 and the truth’s 1.283. Both are conservative there, so nothing breaks — but the direction is the one the arithmetic predicts, and it is the direction that would matter in a world where the errors were skewed and the reference distribution were not already conservative.

What a diagnostic would have to show #

An analyst who wanted to choose between these on evidence rather than on doctrine would need to see the residuals’ second moment as a function of the design, which is a regression of êₜ² on the predictors — a thing anyone can run and almost nobody does before choosing a bootstrap.

Two cautions about it are worth having. The regression is on squared residuals, so it is noisy: detecting the dependence used here takes a good deal more data than fitting the mean model did. And a null result is not evidence of homoskedasticity so much as evidence that the check was underpowered — which is the standing problem with every diagnostic that licenses a simpler procedure.

Given that, the defensible default is the one that costs the least when it is unnecessary. Mammen’s multiplier is conservative in both of the worlds where it is not needed — its 5% point sits 0.157 and 0.102 above the truth’s — and it is the only one of the four that is right where it is. That is an unusually clean recommendation for this site to be able to make, and it is clean only because the comparison was run in three worlds rather than one.

What is claimed here, and what is not #

This essay takes what a resampling of the residuals keeps, and the claims are the twelve critical values against three truths, the paired disagreement rates, and the identity that says why only a multiplier can carry a variance that depends on the design.

What stays out and is named as a decision: serial correlation in the errors’ levels, which is where a block resampling would earn its keep and which this construction does not have; a studentised statistic, for which the whole comparison collapses and which has its own defect the reality check was stated to avoid; and the choice of block length, which is a dial nobody here turned because the resampling it belongs to is worth nothing in this problem at any setting.

The boundary against the nested field is the null. That a reference distribution has to be generated rather than resampled from the differentials, and what the row bootstrap does when it is not, are established there. What is new is that generating one requires a second decision nobody was making — and the reading of the table and the displacement inside it are two further levers that this one is independent of.

The checks, and the refusals that make them mean something #

Two claims are gated in this field’s library. The two resamplings that detach a residual from its row are required to over-reject where the variance is a function of the design and the two multipliers to hold their level, which is the field’s central result and would fail in both directions if the mechanism were being described backwards. And the block bootstrap’s reference distribution is required to sit within a fixed fraction of the iid one’s, which is the measurement that says it is repairing something that is not there.

The refusal for this essay is a reference distribution simulated from the candidate the search selected rather than from the benchmark. Under that reference the winner is the truth by construction, every rival is misspecified, and the procedure finds a genuine improvement a small fraction as often as the honest one.