Correcting the persistence

The forecast field measures a defect and leaves it where it found it. A fitted first-order autoregression forecasts by multiplying the last observation by φ̂ʰ, and that forecast returns to the mean too quickly: at eight steps ahead on forty observations the decay factor is 32.8% short of what the model would give if its parameter were known. The reason given there is that least squares estimates persistence low and the forecast raises the estimate to a power.

That reason is correct and it is also a repairable defect. The bias has a closed form, subtracting it is one line, and the interesting question is not whether the line works but what it costs — which is the question every bias correction has to answer and most are not made to.

Where the bias comes from, and why it is downward at every φ #

An autoregression is a regression of the series on itself one step back, and the thing that makes it different from every other regression on this site is that the regressor is made of the response. The sample mean subtracted from both sides is computed from the same observations that appear on both sides. That entanglement is the whole source of the problem.

The mechanism, in one sentence: a series that happens to wander upwards in a particular sample has a sample mean above the truth, so its deviations from that sample mean cross zero more often than the true deviations do, and a series that crosses its own mean more often looks less persistent than it is. The estimate is pulled towards zero, and it is pulled towards zero whichever way the series wandered, because the sample mean chases the wander in both directions.

Kendall’s expression for the leading term is

E[φ̂] − φ ≈ −(1 + 3φ)/n

and three properties of it are worth reading off before any of it is counted. It is negative for every φ in the stationary range, so persistence is understated whatever the truth is. It is largest where φ is largest, which is where forecasts are most often wanted. And it goes as 1/n rather than 1/√n, so it shrinks faster than the standard error does — but at fifty observations and φ = 0.9 it is still about two-thirds of a standard error, and no confidence interval anywhere mentions it.

At φ = 0.5 the counted bias is −0.0494 against a predicted −0.0500. At φ = 0.8 it is −0.0697 against −0.0680. The formula is doing real work: it was derived for a different purpose, in a different century, and it lands within a few thousandths of a Monte Carlo count that was never shown it.

Where the formula stops being right, and in which direction #

At φ = 0.95 the counted bias is −0.0882 and the formula says −0.0770. The expression is a leading-order approximation in 1/n, the neglected terms grow as φ approaches one, and they all point the same way: the real bias is larger than the formula admits, in the corner of the parameter space where the bias matters most.

That is worth naming as a limitation rather than absorbing into a tolerance. A correction built on the formula will under-correct exactly where a forecaster most needs it corrected, and the size of the under-correction — a fifth of the bias at φ = 0.95 — is measurable and is measured here rather than assumed away.

The bias is a fact about the estimator, not about the sample #

One reading of a downward bias is that the sample happened to be uninformative and a larger one would fix it. That reading is available for a standard error and not for this. Every series in the count above is stationary, correctly specified, and generated by exactly the model being fitted; there is no misspecification anywhere and nothing to diagnose. The estimator is simply not centred on the thing it estimates, and the offset is a deterministic function of the truth and the sample size.

The correction, and a second route to it #

The correction is the formula evaluated at the estimate rather than at the truth, since the truth is what is missing:

φ̃ = φ̂ + (1 + 3φ̂)/n

Evaluating at the estimate introduces a second-order error and does not introduce a circularity — the added quantity is a known function of a known number. At φ = 0.8 and n = 50 the bias goes from −0.0697 to −0.0059, which is 92% of it removed.

The site’s habit is a closed form beside every simulation, and here the simulation is the estimator itself. Simulate from the fitted model, refit, and see how far the refits fall below the value they were simulated from; that difference estimates the bias without knowing any formula. Across three hundred series at φ = 0.8 and n = 50 the bootstrap’s correction averages 0.0684 and the closed form’s 0.0642 — two routes that share no arithmetic, agreeing to within seven percent of themselves, one of which costs a multiplication and the other a hundred and twenty refits.

The disagreement between them is not noise, and its direction is the useful part: the bootstrap corrects by slightly more, because it is estimating the actual bias of the actual estimator rather than the leading term of an expansion, and the actual bias is larger. It is the same shortfall the formula showed at φ = 0.95, arriving through a completely different door.

What the correction costs #

A bias correction adds a quantity computed from the data to an estimate computed from the data. Whatever that quantity is, it has its own sampling variation, and the corrected estimate is more variable than the one it repairs. That is not a defect of this correction — it is arithmetic, and it is the reason bias corrections are a trade rather than an improvement.

At n = 50 the standard deviation rises from 0.1033 to 0.1095 at φ = 0.8, and from 0.1274 to 0.1350 at φ = 0.5 — six percent in both cases, which is what adding 3/n times a variable quantity to a variable quantity does.

Root mean squared error is the sum of the two effects and it is the one that decides. At φ = 0.8 it falls from 0.1246 to 0.1096. At φ = 0.95, from 0.1231 to 0.0925 — a quarter of it, because there the bias was the dominant term. At φ = 0.5, from 0.1366 to 0.1350, which is nothing: the bias was small, the variance cost is the same as ever, and the trade is roughly neutral.

Whether to correct is therefore not a question with one answer. It is worth doing where the persistence is high, worth almost nothing where it is low, and the boundary between those is a measurement rather than a rule. What can be said unconditionally is narrower and more useful: the correction never costs much, and where it helps it helps by a quarter.

The estimate that leaves the parameter space #

There is a cost that no bias-variance accounting contains, and it is the one an implementation hits first. The correction adds a positive quantity to an estimate that may already be near one. At φ = 0.95 and n = 50, 27.0% of corrected estimates come out above 0.995 — outside the stationary range, where the model they belong to has no unconditional variance and the forecast interval built from them is not defined.

Something has to be done, and what every implementation does is cap the estimate. Capping puts a piece of the bias straight back, and it does so on exactly the samples where the correction was doing the most work: the ones where φ̂ was highest. At n = 25 and φ = 0.95 the truncated share is 34.1%, so a third of the corrected estimates are not the corrected estimate at all.

This is reported here rather than hidden because it changes what the correction is. Below about φ = 0.9 it is a clean improvement in squared error at a small cost in spread. Near the edge of the stationary range it is a different estimator — a capped one — whose properties are the properties of the cap as much as of the formula, and whose reported bias includes a term nobody wrote down.

The exponent, read off four sample sizes #

−(1 + 3φ)/n makes two separate claims and only one of them has been tested above. The first is the constant: at a given φ and n the bias should be that number, and the counts agree to a few thousandths everywhere except the top of the range. The second is the exponent, and it is the claim that decides whether any of this transfers to a sample size nobody simulated. It can be read off the counts already made, with no arithmetic beyond division.

At φ = 0.8 the counted bias is −0.0697 on fifty observations, −0.0345 on a hundred and −0.0168 on two hundred. The successive ratios are 2.02 and 2.05: doubling the sample halves the bias, twice over, to within three percent of exactly halving it. That is what an exponent of one looks like when it is measured rather than asserted. At φ = 0.95 the same reading between twenty-five and fifty observations gives −0.1811 against −0.0882, a ratio of 2.05 — and that is the corner where the constant was at its worst, understating the bias by a fifth.

The two failures are therefore separable, and only one of them has been found. A formula whose constant is wrong by a fifth and whose exponent is right can still be trusted to say how the error shrinks with more data; a formula whose exponent is wrong cannot be trusted at any sample size outside the ones it was checked at, whatever its constant does. Nothing above fitted anything to these counts, which is what makes the exponent a check rather than a restatement.

What the offset is worth, in units of standard error #

The bias goes as 1/n and the spread goes as 1/√n, so the ratio between them goes as 1/√n and shrinks more slowly than either. The counts give that ratio directly. At φ = 0.8 and n = 50 the bias is −0.0697 against a standard deviation of 0.1033, which is 0.67 of a standard error. At n = 200 it is −0.0168 against 0.0455, which is 0.37. Quadrupling the sample moved the offset from two-thirds of a standard error to a third — a factor of 1.8 where the square-root law predicts 2 — and did not come close to removing it.

Two-thirds of a standard error is the number worth carrying out of this, because it is the one that says what an interval built on φ̂ is actually doing. An interval placed symmetrically about a point that sits two-thirds of a standard error below the truth is not a 95% interval for that truth. Its coverage fails asymmetrically, missing high far more often than low, and nothing in its own construction contains a term for the offset that causes it. The interval that forgets it estimated is about a different omission in the same band — variability the plug-in leaves out, rather than an offset it inherits — and the two compound rather than cancel, because both pull the covered region towards the mean.

The practical consequence is narrow, and it can be stated without settling whether the correction is worth taking. The offset is a function of n and of the estimate, and both are known. Whatever is done to the point estimate, that quantity can always be computed and reported, and an analysis that prints φ̂ on its own has withheld a number that cost one multiplication. The trade in squared error is a judgement; naming the size of the bias is not.

Why this is not the winner’s curse #

Two biases on this site pull an estimate away from the truth and it is worth separating them, since both involve a fitted quantity being wrong in a predictable direction.

The winner’s curse is a selection bias: the estimate is unbiased before anything is chosen, and becomes biased because it was chosen for being large. Nothing about the estimator is wrong; what is wrong is conditioning on having won. The repair is to estimate from data the selection did not use, and the cost is that data.

This is an estimation bias: no selection has happened, every series is used, and the estimator is biased on its own. The repair is arithmetic on the estimate itself and the cost is variance. The two require different repairs and neither repair touches the other’s problem, which is why an essay about one is not an essay about the other.

There is a third relation worth naming because it runs the other way. The shrinkage the hierarchical field is about pulls an estimate towards a population mean on purpose, accepting bias to buy variance. This essay is the same trade taken in the opposite direction: accepting variance to buy back bias. That both are worth doing in their own settings is the clearest evidence that neither bias nor variance is the quantity to optimise on its own.

Three quantities, three answers #

It is worth setting the accounting out in one place, because a correction that is described as working or not working is being described in a currency that has not been named.

Bias. The correction removes most of it everywhere: 92% at φ = 0.8 and n = 50, 81% at φ = 0.95. Scored on bias alone, it works, and scored on bias alone every bias correction ever written works, which is why nothing here stops at that line.

Spread. The correction costs about six percent of the standard deviation, at every persistence and every sample size measured. That number is stable because the added term is 3φ̂/n and its variability is 3/n times the variability of φ̂ — a fixed proportion, not a function of where in the range the truth sits.

Squared error. The two effects meet, and the result runs from a quarter saved at high persistence to nothing at all at low. This is the only one of the three that a forecaster can spend, and it is the one that makes the correction a judgement instead of a rule.

The next essay adds a fourth quantity, which is the one the whole exercise was for — the accuracy of the forecast the estimate is fed into — and the answer there does not follow from any of these three.

What is claimed, and what is not #

The claim is the bias of least squares in a first-order autoregression, and what removing it costs: the closed form, its failure near the edge of the range, the bootstrap route to the same correction, the variance it buys the bias reduction with, and the truncation nobody accounts for. The forecast field named bias-corrected estimation of a persistence parameter as unclaimed when it was written.

What stays out: median-unbiased estimation, which conditions on a different summary of the sample and is a separate construction; bias correction in models with more than one lag, where the correction becomes a matrix expression and the truncation problem becomes a question about the roots of a polynomial; and unit-root inference, where φ = 1 is the hypothesis rather than a boundary and the whole distribution theory changes — that is the spurious-regression field’s subject and nothing here touches it.

The checks, and the refusal #

Four claims are gated in this field’s library. The counted bias must match −(1 + 3φ)/n to within a quarter of itself. The correction must remove at least two-thirds of it. The corrected estimator must be more variable than the uncorrected one, which is the cost being asserted rather than assumed. And the bootstrap correction must agree with the closed form to within thirty percent, which is the two-routes requirement applied to an estimator rather than to a number.

The refusal in this half of the field is in the next essay, where the corrected forecast is scored on the quantity it was supposed to improve.