An interval that knows the band was maximised

What fitting them together buys maximised a regression’s likelihood over its coefficients and an eight-lag band of error autocorrelations at once, and found the joint fit worth having: under a five-period moving average it beat the two-step plug-in by 0.00767 of selection risk at 3.3 paired standard errors. It named, and left out, what the joint fit does to the uncertainty it reports. The interval the joint fit reports measured it: on the white predictor, whose true coefficient is zero and on which every other rule is honest, the joint fit’s 95% interval covers 71.4% to 85.3% across four laws, because it reports nearly the standard error of a rule that knows the covariance and delivers about half of that rule’s advantage. Counting the eight lags it estimated in the degrees of freedom buys one or two points.

That essay named two standard repairs and measured neither. A sandwich replaces the reported variance with one that reads the whitened residuals’ own dependence, so that a band which whitened too eagerly is caught by what it left behind. A parametric bootstrap redraws the regression from the fitted band, refits the joint maximum on every redraw, and reads the spread of the redrawn coefficients, pricing the maximisation by repeating it. Which of the two restores 95%, and whether either does so without giving back the risk the joint fit earned, was the question it left.

This essay builds both, on the same regressions. Neither restores 95%, and the reasons they fall short are different and instructive.

The regressions and the six intervals #

The setting is the earlier essay’s: fifteen candidate regressions on four predictors whose persistences run from 0.9 to zero, a hundred and twenty rows, errors from four laws each with a first-order autocorrelation of 0.8 — a first-order autoregression, a five-period moving average, long memory, and a break in the persistence. The interval is for the widest candidate’s coefficients, read on two of them: the white predictor, with no persistence and a true coefficient of zero, and the most persistent one, at 0.9.

Six intervals on each regression:

• the two-step band, plugged in and reported as if known;
• the joint fit, as it reports itself;
• the joint fit with White’s sandwich on its whitened residuals;
• the joint fit with a sandwich over four lags, Newey and West’s rule at this length;
• the joint fit with a parametric bootstrap: twenty-four redraws from the fitted coefficients and the fitted band, each with its band maximised afresh from its own plug-in;
• the regression whitened by the law’s own covariance, which is entitled to treat it as known.

Two hundred regressions per law, the first two hundred of the earlier essay’s draws, so its intervals reappear here on the same samples.

Under the moving average #

On the white predictor the two-step band covers 95.5% and the law’s own covariance 92.0%. The joint fit as reported covers 72.0% — the earlier essay’s 71.4%, on these two hundred draws.

White’s sandwich covers 71.0%. The four-lag sandwich covers 71.0%. Neither moves the interval at all.

The bootstrap covers 85.5%. It closes more than half of the gap between the joint fit and the promise, and it does it by reporting a larger standard error: 0.0244 on average where the joint fit reports 0.0167, against an actual spread across regressions of 0.0315. The ratio of what the estimate delivers to what the interval reports falls from 1.738 to 1.216. The oracle’s ratio is 1.096, the two-step’s 0.956.

Why the sandwich reads nothing #

A sandwich standard error is a correction for what the model got wrong about the errors, read off the residuals the model leaves. If the whitening were too aggressive or not aggressive enough, the whitened residuals would carry dependence or heteroskedasticity, and the sandwich would see it and widen the interval.

The joint fit’s whitened residuals carry neither, and the reason is the fit. The band was chosen by maximising the likelihood of exactly these residuals under exactly this whitening, and the likelihood of a Gaussian model is largest when its whitened residuals look like independent noise of constant variance. So the maximisation spends its eight free lags making the residuals look white, and a sandwich that reads the residuals’ remaining dependence finds the dependence the maximisation was paid to remove. The four-lag sandwich covers no better than White’s because there is nothing at lags one to four to read.

The check before the standard error found a residual-based check missing a lag-one correlation of 0.2 four times in five on a hundred and twenty rows. Here the sandwich is not merely weak; it is reading residuals that were fitted to pass it. Its failure is a statement about the order of operations: a diagnostic computed after a fit that optimised the diagnostic’s own target measures the fit’s success, not the model’s adequacy.

Why the bootstrap reads half #

The bootstrap does what the sandwich cannot: it repeats the maximisation. Every redraw is a regression the joint fit has never seen, with errors from the fitted band, and the joint fit is run on it from scratch. The spread of the redrawn coefficients therefore includes the variability the maximisation adds — the coefficient moving as the band is chosen differently on each sample — which is precisely what the reported standard error leaves out.

What it cannot repeat is the first maximisation. Every redraw comes from the band this sample’s maximisation chose, and that band was fitted to make this sample’s residuals look white: it is the over-eager band, the one the earlier essay found sitting further towards the edge of the admissible region than the plug-in. In the bootstrap’s world that band is the truth, so the redraws are regressions on which the maximised band is right, and the joint fit’s reported precision is closer to honest there than it is in the world the sample came from. The bootstrap prices the maximisation’s variability around the fitted band and not its bias towards one.

That leaves a gap of seven to nine points, and its size is the price of one maximisation that the bootstrap takes as given.

Under all four laws #

The pattern holds under every law. As reported the joint fit covers 80.0%, 72.0%, 82.5% and 77.0% under the autoregression, the moving average, long memory and the break. The four-lag sandwich covers 79.0%, 71.0%, 81.0% and 76.0% — a point lower everywhere, because it adds estimation noise to a correction that has nothing to correct. The bootstrap covers 88.0%, 85.5%, 88.0% and 89.0%. The law’s own covariance covers 93.0%, 92.0%, 93.0% and 93.0%.

So the bootstrap repairs part of the gap under every law, between five and thirteen points, and leaves the interval at 85.5% to 89%, four to six and a half points behind the oracle on these draws. The sandwich’s is zero under every law.

The ratios say the same thing in the interval’s own units. The joint fit’s delivered spread is 1.36 to 1.74 times what it reports; with the sandwich, 1.39 to 1.75; with the bootstrap, 1.07 to 1.22. The two-step band’s ratios are 0.95 to 1.05 and the oracle’s 1.03 to 1.10.

The persistent predictor #

On the predictor with a persistence of 0.9, every rule that estimates the covariance is short, as the earlier essay found, because a slightly wrong covariance moves the variance of a persistent predictor’s coefficient a great deal. The joint fit covers 78.0%, 75.5%, 83.5% and 75.5% under the four laws, the two-step band 83.0%, 85.0%, 85.0% and 85.5%, and the oracle 93.0%, 94.0%, 90.5% and 93.5%.

The bootstrap lifts the joint fit to 82.5%, 87.5%, 87.5% and 84.0% — the two-step band’s level or a little above, which on this column is as far as any feasible rule here reaches. The sandwich again does nothing, at 79.0%, 75.5%, 82.0% and 76.5%.

On this column the bootstrap is not repairing the joint fit’s own defect alone. Every estimated covariance is short here, the two-step band by twelve points and the joint fit by seventeen, because the persistent predictor’s variance depends steeply on the covariance — the observations that repeat each other worked out how steeply. The bootstrap brings the joint fit up to the two-step band and no further, which says it has removed the maximisation’s share of the shortfall and left the share every estimated covariance carries.

Repeating the procedure, and when it matters #

Two bootstraps have now asked the same question of two different procedures and got opposite answers, and the difference is the point. A bootstrap that repeats the design asked whether a resample after an adaptive experiment should repeat the adaptive procedure, and found that it made no difference: the design converged so fast that repeating it added a variability that was nearly absent. Here, repeating the maximisation is the whole of the repair, and a resample that does not repeat it — the warm-started version, which barely moved off the sample’s maximum — reproduces the reported standard error and repairs nothing.

The two procedures differ in how much the estimate depends on what they choose. A sequential design that lands on nearly the same settings every time contributes nothing to the estimate’s spread. A band of eight free autocorrelations chosen to maximise a likelihood moves with every sample, and the coefficient moves with it, because the plug-in and the maximum are different points and the distance between them is itself random. So the rule is not that a bootstrap must always repeat the procedure. It is that it must repeat whatever part of the procedure the estimate depends on, and finding out which part that is requires running both versions — which is what the warm start, by accident, did.

What a practitioner would report #

The joint fit is worth using when its estimate is better, which nothing in the fit can guarantee on its own and the earlier essay measured case by case. When it is used, its reported standard error should not be printed. The bootstrap that re-maximises on each redraw is the least bad of the intervals measured: it costs twenty-four joint fits per regression, it is still a fifth too narrow, and it says so in no output a practitioner sees. The two-step band’s interval is honest and wide. A report that gives the joint estimate with the two-step band’s interval is conservative and defensible; one that gives the joint estimate with its own interval is wrong by twenty points; one that gives it with a sandwich is wrong by twenty points and looks corrected.

The joint fit was worth fitting because its estimate is tighter. Under the moving average the white predictor’s coefficient spreads by 0.0315 across regressions under the joint fit and 0.0421 under the two-step band — the earlier essay’s gain, as a standard deviation. The bootstrap reports 0.0244 for the joint fit, so its 95% interval is about 0.096 wide; the two-step band reports 0.0438, an interval about 0.172 wide. The bootstrapped joint fit’s interval is the narrower by more than two fifths and covers 85.5% against the two-step’s 95.5%.

Neither is the interval a forecaster wants. One is honest and wide; the other is narrower than the estimate justifies by a fifth of its width. What the bootstrap establishes is that the joint fit’s advantage is not imaginary — its estimate really is tighter and the bootstrap’s interval is still the narrower one when made nearly honest — and that the last part of the honesty is the part a resampling built on the fitted model cannot supply.

Why the band has to be a family for any of this to work #

Both repairs presuppose something a family before a fit had to establish first: that the band is a parameter, with a likelihood over it, rather than a plug-in. The sandwich is defined for any fitted model and never asks; the bootstrap needs the band to be re-estimable on each redraw by the same procedure, which is only possible because the procedure is a maximisation over a stated family. A plug-in band could be recomputed on each redraw too, and the two-step band’s honest interval is what that recomputation would confirm. The joint fit’s difficulty is specific to the step that makes it better — the search over the family — and the bootstrap is the one repair that contains that step.

That also explains why counting the eight lags in the degrees of freedom, the earlier essay’s first repair, bought so little. Counting charges for the dimension of the family, which is the same on every sample. What the interval misses is how far the search moves on each sample, which is a property of the likelihood’s shape around its maximum and changes from sample to sample. A correction that does not depend on the sample cannot track a cost that does.

Two hundred regressions and a band maximised again on each redraw #

Two hundred regressions per law, the first two hundred of the earlier essay’s seeds. The joint fit maximises the concentrated likelihood over the eight band coefficients by coordinate ascent from the tapered plug-in. The sandwiches are computed on the whitened design and residuals with Bartlett weights at lags zero and four. Each bootstrap redraw adds the fitted band’s Cholesky factor times fresh normals, scaled by the fitted error variance, to the fitted mean, and maximises its band from its own plug-in with the full search; the interval is the joint estimate plus or minus 1.96 times the redrawn coefficients’ standard deviation over twenty-four redraws.

A first version of the bootstrap started each redraw’s search at the sample’s own maximum with a short step, which is cheaper and is wrong: the search barely left its start, the redrawn coefficients barely moved, and the bootstrap standard error came out at the joint fit’s own — 0.0173 against 0.0177 on a trial run of thirty regressions — so it appeared to repair nothing. Starting each redraw where the sample started, at its own plug-in, the standard error is 0.0255 on the same thirty. A bootstrap that does not repeat the maximisation is not pricing it, and the warm start was a way of not repeating it.

Coverages on two hundred regressions have standard errors of one and a half to three points, so the bootstrap’s gain of five to thirteen points is two or more of them and differences between the laws of two or three points are not.

A sandwich changes nothing, and re-maximising closes most of the gap #

The claims are that a sandwich standard error on the joint fit’s whitened residuals, with or without lags, leaves its interval’s coverage where it was under every law here; that a parametric bootstrap that maximises the band again on every redraw lifts the white predictor’s coverage from 72.0%–82.5% to 85.5%–89.0% and the ratio of delivered spread to reported standard error from 1.36–1.74 to 1.07–1.22; and that the remaining shortfall is consistent with the redraws coming from the maximised band rather than the true one.

Not claimed: anything about bootstraps with more redraws, which would make the standard error less noisy and should not change its expectation; anything about a bootstrap that resamples residuals rather than drawing them from the fitted band; and anything about the risk the joint fit earns, which an interval does not change.

Still open: redrawing from a band that was not maximised #

The bootstrap’s remaining gap is attributed to its world: it redraws from the band the sample’s maximisation chose. The obvious test of that attribution is to redraw from a band the maximisation did not choose — the two-step plug-in, which is less eager, or a band shrunk partway from the maximum back towards it — and maximise again on every redraw. If the attribution is right, a bootstrap world whose band is less over-fitted should produce a larger spread and close more of the gap; if the coverage does not move, the shortfall belongs to something else, such as the redrawn errors being Gaussian where the laws here are not. That comparison, on the same regressions, would say whether an honest interval for a joint fit can be had from one sample at all.