A set calibrated at the estimate

A search for where a regression breaks reports a row, and a confidence set for the break point reports every row whose profile likelihood is within some threshold of the best. A break that is there measured what that threshold has to be when the break is real — a shift added to the response from row 60 of 120, on errors from a first-order autoregression at 0.8 — and found the conventional two log-likelihood units holding the true row on 12% to 69% of samples. The threshold a 95% set needs is the 95th percentile of the deficit at the true row, and it was 13.54 units at the largest break and 20.27 at the smallest, at a fixed window. No one number serves every break size; a threshold of 21 units that is honest everywhere holds thirty to sixty-eight of the seventy-three candidate rows.

The essay named the construction that could track the break’s size. Simulate the profile’s deficit at the estimated break, under the estimated shift and the estimated dependence, and take that simulation’s 95% point as this sample’s threshold: a parametric bootstrap of the deficit. Its threshold would move with the break the sample shows, and its sets should be narrower at large breaks and no wider at small ones. Whether it holds 95% when everything it simulates from is itself estimated was the question, and the essay said it was the same transport problem the charge that is not a sum met when a threshold calibrated under one law was used under another.

The construction, and its four versions #

For each sample: run the break search at the fixed window, take its top row $\hat b$, fit the regression’s five coefficients separately on each side of $\hat b$, fit a first-order autoregression to the residuals of that split fit, and then simulate 120 responses on the sample’s own regressors from those coefficients, that break and that persistence. On each simulated response, run the break search and record the profile’s deficit at $\hat b$ — how far below its best the profile sits at the row the simulation put the break. The 95th percentile of the 120 deficits is the threshold; the set is every row within it of the sample’s own best.

The same construction can be run with any of its ingredients replaced by the truth, and that is how the shortfall is taken apart. Simulated at the truth: the true break row, the true coefficients and shift, the true persistence of 0.8. True persistence: the estimated break and coefficients, with 0.8 for the errors. True break: the true row and coefficients, with the estimated persistence. And one feasible repair, persistence matched: the persistence chosen by simulation so that a split fit on simulated data reports, on average, the residual correlation this sample reported. Three hundred samples at each of five break sizes, from 0.3 to 2 standard deviations.

Simulated at the truth, the construction is what it was meant to be. It covers 95.0%, 93.7%, 93.7%, 95.0% and 94.0% at the five break sizes, within its own standard error of about a point and a quarter everywhere. The idea is sound: a threshold read off a simulation of the deficit at the right break, under the right dependence, gives a 95% set at every break size, which no fixed threshold does.

Simulated at the estimate, it covers 64.0%, 66.0%, 67.7%, 70.7% and 75.7%. That is better than the two-unit set — 12.3% at the smallest break, 66.7% at the largest — and much worse than its promise. A set calibrated from the sample’s own estimates misses the true row on a quarter to a third of samples.

The one number that does most of it #

The persistence read off the split regression’s residuals averages 0.61 to 0.63 across the five break sizes. The errors have 0.8. The gap is not a small-sample bias of the usual kind — the bias of a lag-one correlation at 120 rows is a few hundredths — but the regression: ten coefficients fitted to 120 rows of persistent errors, on regressors that are themselves persistent, absorb a share of the errors’ slow wander into the fitted values, and what is left in the residuals is less persistent than the errors were. A simulation that draws its errors at 0.62 simulates a world where the profile’s deficit at the true break is smaller and steadier than it is, and hands the sample a threshold that is too low.

The thresholds show it directly. Simulated at the truth, the median threshold is 19.9 units at the smallest break and falls to 14.8 at the largest — the tracking the construction promised, a quarter lower at a large break than a small one. Simulated at the estimate it is 9.2 falling to 5.3: less than half the truth’s at every size. Told the true persistence but estimating the break and coefficients, it is 18.4 falling to 12.6, close to the truth’s. The threshold is mostly a function of the persistence it is simulated at.

Two routes to the threshold a set needs #

The truth’s thresholds are worth reading against the earlier essay’s, because they are the same quantity reached two ways. A break that is there measured the threshold a 95% set needs directly: draw samples with a real break, compute the profile’s deficit at the true row on each, and take the 95th percentile across samples. Here the threshold is computed per sample, by simulating from the true model on that sample’s own regressors, and the median across samples is reported. The first is one number per break size; the second is a distribution of numbers per break size, one for each set of regressors. They agree where they should: 19.9 units here against 20.27 there at the smallest break, 14.8 against 13.54 at the largest, the per-sample route slightly higher at the large break because a median over regressors is not a pooled percentile.

That agreement is what makes the shortfall interpretable, and it is checked rather than assumed: the two routes share no simulation and no samples. The construction reproduces the earlier measurement when it is given the truth, so its failure from the sample is a failure of what it is given and not of how it uses it.

Residuals are not errors #

The persistence’s bias is not new to this collection; it has been found twice in other fields and it is worth putting the three findings side by side. The fit that takes the memory out showed that a candidate regression’s residuals report less dependence than its errors, by an amount that is arithmetic rather than noise, and that the candidate with the most coefficients loses the most. A dependence fitted with the line repaired it by estimating the dependence jointly with the regression rather than from its residuals. Here the fitted model has ten coefficients — five on each side of the break — on regressors that are themselves persistent, which is the configuration in which the residuals lose the most, and the loss is from 0.8 to about 0.62.

The consequence is sharper here than in either earlier field because a profile likelihood’s deficit is a quantity whose size is set by the dependence. At a persistence of 0.8, the observations that repeat each other put the information in a fifty-row series at about six independent observations; at 0.62 it is about twice that. A simulation run at 0.62 simulates a sample with more information about where the break is than the real one has, and hands it a set sized for that. The matched persistence is the indirect-inference version of the joint fit the dependence field recommended, applied to a quantity that field did not need.

Taking the shortfall apart #

Estimating only the break and the coefficients, with the persistence told, costs between 3.7 and 5.7 points of coverage: the sets cover 88.3% to 90.3%. Estimating only the persistence, with the break told, costs between 8.3 and 15.0 points: 80.0% to 85.7%. Both together cost 18.3 to 31.0 points, more than the two separately, because an estimated break simulated through a too-quiet error process compounds the two: the deficit at a break placed a few rows from the truth is small in a quiet world, and the threshold that results is small twice over.

So the transport problem the lead anticipated is real and has a specific address. It is mostly the persistence, and it is the persistence because the instrument that estimates it — the residuals of the very fit the set is about — is biased by that fit.

A persistence matched by simulation #

The persistence’s bias has a structure the sample can be asked about. The sample reported a residual correlation of about 0.62; a simulation can report what residual correlation a split fit would show at any trial persistence, on the sample’s own regressors with its own coefficients and break, and the persistence whose simulated residual correlation matches the sample’s is an estimate corrected for the fit. That is indirect inference on one number, and it costs a bisection of thirty simulated fits at each of nine trial values per sample.

It moves the persistence from 0.61–0.63 to 0.74–0.77, about two thirds of the way to the truth. Why it stops short is not settled here. The matching is done at the estimated break and coefficients — the break is off by a root-mean-square of nine to twenty-three rows across the break sizes — and the halfway run says those cost five points of coverage on their own, so a persistence matched at the right break would be worth more than one matched at the wrong one. The set built from the matched persistence covers 81.0% to 83.3% at every break size, against 64.0% to 75.7% for the plug-in — half the shortfall recovered, and the set still a promise of 95% that delivers about five in six.

What the sets cost in rows #

The width is where the construction’s idea pays even when its coverage does not. Simulated at the truth, the set holds 67.2 rows at the smallest break and 22.6 at the largest; the fixed 21-unit threshold, which covers 94% to 98%, holds 68.5 and 32.6. At a large break, a threshold that knows the break’s size gives a set a third narrower than one that has to be honest at every size. The matched-persistence set holds 55.7 falling to 21.9 — about as narrow as the truth’s at the large breaks, which is part of why it does not cover.

A practitioner choosing between these has a choice with a number on each side. The fixed 21-unit set is honest at every break size measured here and wide. The matched-persistence set is narrower and covers four samples in five. Nothing feasible here is both honest and narrow; the set simulated at the truth is, and it is not available.

There is a middle course the measurements support, and it is cheap. The matched-persistence threshold is a number per sample, and the fixed 21 units is a number for every sample; the larger of the two is a threshold that keeps the matched set’s narrowness only where the matched set is wider than the fixed one would allow and otherwise falls back on the fixed set’s honesty. At the large breaks, where the matched threshold is near eleven units, that is the fixed set, and at the small ones it is nearly the fixed set too, since the matched threshold there is fourteen. So the middle course is the fixed set, and the reason is the same one: the threshold a sample can estimate for itself is below the threshold its errors need at every break size measured. A rule that wants a narrower set than 21 units has to estimate the persistence better than a split regression’s residuals can, and the matching recovers two thirds of the way.

How the numbers were made #

Three hundred samples at each break size, the regressors and errors drawn from the field’s predictive regression with first-order errors at 0.8 and the shift added to every row from 60 after the sample is drawn, so the errors are identical to the null’s. The break search is the field’s at the fixed, unwhitened window: every candidate row in the middle sixty per cent of the sample, the regression’s five coefficients refitted on each side, the profile log-likelihood $-\tfrac{n}{2}\log$ of the combined residual sum of squares. Each threshold is the 95th percentile of 120 simulated deficits on the sample’s own regressors; the matched persistence is a nine-step bisection over trial persistences, each scored by the mean residual correlation of thirty simulated split fits sharing one set of innovations, so that the bisection is on a smooth function. Coverage is whether the set contains row 60, and with three hundred samples a coverage near 95% carries a standard error of about a point and a quarter and one near 70% about two and a half points.

Checked on the measurement: simulated at the truth the set covers within four points of 95% at every break size, and simulated at the estimate it covers under 80%; the split fit’s residual persistence is under 0.66 at every break and the matched one more than a tenth higher and still under the truth; the estimated persistence costs more coverage than the estimated break at every size; and at the largest break the truth’s set holds under three quarters of the fixed threshold’s rows.

Still open: a set that simulates its own uncertainty about the break #

The two-stage construction simulates at one break, the estimated one, and the halfway runs show that being told the break is worth five points. A set that simulated over the break’s own uncertainty — drawing the break row for each simulated response from the profile’s own spread around $\hat b$, the way a Bayesian predictive would — would charge for not knowing where the break is, and its threshold would rise where the profile is flat. Paired with the matched persistence it is the construction that could close the rest of the gap, and its cost is a second loop inside the first. Whether it reaches 95%, and at what width, has not been measured.

What is claimed here, and what is not #

This essay takes a parametric bootstrap of the break profile’s deficit, calibrated at the estimate and at the truth. The claims are that simulated at the true break, shift and persistence the set covers between 93.7% and 95.0% at five break sizes and narrows from 67.2 to 22.6 rows; that simulated at the sample’s estimates it covers between 64.0% and 75.7%, with a median threshold less than half the truth’s; that the split fit’s residual persistence is 0.61 to 0.63 against a true 0.8; that the estimated persistence alone costs 8.3 to 15.0 points of coverage and the estimated break and coefficients alone 3.7 to 5.7; and that a persistence matched by simulation, at 0.74 to 0.77, gives sets covering 81.0% to 83.3%.

What stays out, named as a decision: the joint search with the window, since every set here is built at the fixed window; other laws, under which the residual persistence’s bias would differ; and the break’s own uncertainty in the simulation, deferred above.

The boundary against a break that is there is that it measured the thresholds a set needs and this essay builds a set that estimates them. Against the break a shared scale hid, which found the break’s location sharper once each regime had its own scale, this essay holds the model fixed and asks how well its uncertainty can be read off one sample.