A fixed point with no likelihood under it

A regression on dependent errors is fitted by generalised least squares through an estimate of the errors’ covariance, and the estimate has to be read off something. The errors are not observed, so it is read off residuals — and the fit that takes the memory out showed that residuals report less dependence than the errors had, by an amount that is arithmetic rather than noise. The regression’s own fit absorbs part of the errors’ slow wander, and what is left looks less persistent than it was.

A dependence fitted with the line repaired that for the one dependence with a likelihood: a first-order autoregression’s persistence estimated jointly with the regression, by maximising the likelihood over both. Iterating is not maximising then compared the joint maximum with the rule a practitioner actually reaches for — re-read the persistence from the generalised residuals, refit, repeat — and found the fixed point of that iteration converging in seven steps to a value that is not the maximum, because the likelihood has one term the residuals cannot see. It named the one thing the iteration can do that the maximiser cannot: it generalises without a likelihood.

That is the case this essay measures. A tapered band of sample autocovariances — the field’s covariance estimate for a dependence of unknown shape — is an estimate substituted into a model rather than a model, and there is nothing to maximise. The iteration still runs: read the band off the residuals, whiten and refit, read the band again off the new fit’s residuals in the original metric, and repeat until the coefficients stop moving. Whether it converges, what it converges to, and whether the answer is worth having are questions with no likelihood to answer them.

The iteration, and the three things recorded #

The setting is the field’s: a regression of a response on four persistent predictors at 120 rows, the full four-predictor model, and errors from one of three laws — a first-order autoregression at 0.8, a five-period moving average and long memory at a fractional difference of 4/9 — every one with a lag-one correlation of 0.8 in its errors. The band is Bartlett’s at four, eight or twenty lags. On each draw the band is read once off the least-squares residuals and the regression refitted through it, which is the rule every earlier essay used; then the loop continues until no coefficient moves by more than $10^{-8}$, or sixty fits. Three things are recorded on 300 draws per law and window: whether and how fast the loop converged, the lag-one correlation of the residuals it ended on, and the prediction error of the final fit on an independent replicate, paired against the first fit’s.

One draw shows the shape. The least-squares residuals have a lag-one correlation of 0.693. The first band fit’s residuals have 0.713 at four lags, and the next 0.714, after which nothing moves in the third decimal; at twenty lags the path is 0.693, 0.710, 0.714, 0.715 and settles in twelve fits. Each step moves less than the one before. The fixed point is where reading the band off the residuals gives back the band that produced them.

It converges, every time #

On all 2,700 runs — three laws, three windows, three hundred draws — the loop converges. The median is seven fits at four lags under every law, eight or nine at eight lags, ten or eleven at twenty; the largest on any draw is 23. A wider band takes a few more fits, because it has more autocovariances to settle, and long memory takes about as many as the others.

The count is worth setting beside the first-order case. Iterating is not maximising found the iterated first-order rule converging in seven steps; the band at four lags converges in seven at the median under every law here, and the band at twenty in ten or eleven. The two iterations are the same loop with a different object read off the residuals each time — one number in the first case, four or eight or twenty autocovariances in this one — and the number of fits grows slowly with what is read, not with how it is read.

That is not guaranteed by anything. A residual-based update has no objective it is increasing, so there is no general argument that it cannot cycle or wander, and a band has fewer parameters’ worth of structure to lean on than the first-order case does. The measurement is that at these laws, sizes and windows it behaves like a contraction, and that is a statement about these settings rather than a theorem.

How much of the memory comes back #

The least-squares residuals report a lag-one correlation of 0.714 under the first-order law, 0.738 under the moving average and 0.467 under long memory, against 0.8 in the errors in every case — the attenuation the field has measured before, at its most severe under long memory, where four persistent regressors soak up most of a slow wander. The fixed point gives some of it back. Under the first-order law it ends at 0.750 to 0.757 across the three windows, about half the 0.086 the fit took out; under the moving average at 0.769 to 0.774, a little over half of 0.062. Under long memory it ends at 0.495 to 0.505 — a tenth of the 0.333 that was lost.

A reading consistent with these numbers, and not itself measured here, is that the iteration stops short for the same reason it starts at all. Each refit through a better band leaves residuals with more of the dependence in them, which gives a better band; but the fit is still a fit, and it still absorbs a share of whatever wander is in the errors, so the loop settles where the two balance rather than at the errors’ own dependence. That share is small for short memory and large for long, which is the pattern above. What the iteration cannot do is charge for the absorption directly, the way a likelihood’s determinant does — the term iterating is not maximising found the first-order iteration missing, and which a band has no counterpart for.

What it is worth #

The restored memory has a consequence, and it is the opposite of the first-order case. There, the fixed point and the maximum differed by a mechanism with a size and made no difference to the decision. Here the fixed point is better than the one-step band at every law and window, and by more than the paired noise. Under the first-order law prediction error falls by 2.84, 3.43 and 4.16 thousandths at four, eight and twenty lags; under the moving average by 2.04, 2.69 and 4.32; under long memory by 0.83, 1.13 and 1.33. Every one is more than two and a half paired standard errors below zero, most of them five to eight. Under the short-memory laws, on an error of about 1.1, that is a fifth to two fifths of a per cent; under long memory, on an error of 1.61, under a tenth of one — small either way, and paid for by nothing more than running the same fit ten times.

The direction is the one the attenuation predicts: a band read off residuals that understate the dependence whitens too little, and a fit through it leans too much on the rows that sit close together in time. The gain grows with the window under the two short-memory laws, which is where the band has most to estimate and the first reading of it was furthest off. Under long memory it is smallest, because the iteration restores least of what was lost there.

Against the likelihood’s maximum #

The fairer comparison is with the rule that does have a likelihood. Under the first-order autoregression the joint maximum of its likelihood has prediction error 1.10012, and the best band fixed point, at twenty lags, 1.10364: when the parametric model is right, fitting it properly beats estimating its covariance nonparametrically, by about three thousandths, which is what a correct model buys.

Under the moving average the first-order likelihood is wrong — no first-order autoregression is a five-period moving average — and its maximum has error 1.06717. The band’s fixed point beats it at eight lags, 1.06468, and at twenty, 1.05983; at four lags, too narrow to hold the moving average’s dependence, it does not. Under long memory the maximum is 1.61074 and the fixed point beats it narrowly at eight and twenty lags, 1.61035 and 1.60863. So the iterated band is the better rule exactly where the parametric model is wrong and the band is wide enough to hold the dependence, and the worse one where the model is right.

The window, once the band is iterated #

The figures carry a practical result that is easy to miss, about the window rather than the iteration. Read once, the band’s prediction error barely depends on the window: under the moving average it is 1.07280, 1.06737 and 1.06415 at four, eight and twenty lags, a spread of under a per cent, which is the flat surface the window that has to be chosen and the field after it kept finding. Iterated, the spread widens and its direction is clear: 1.07076, 1.06468 and 1.05983. The iteration’s gain is largest at the widest band, because a wide band has the most autocovariances to read and the first reading of each is the most attenuated, so iterating rewards the window that was hardest to estimate once.

So the two choices interact. A practitioner who reads a band once can choose its width almost freely and lose little; one who iterates it should choose it wide, because the iteration repairs exactly what a wide band gets wrong on its first reading. At twenty lags under the moving average the iterated band is the best rule in this essay, ahead of the first-order maximum by seven thousandths. Under the first-order law the wide iterated band is still the best band, three thousandths behind the correct parametric model.

That is also the honest version of the claim a covariance with no parameter made for the band: an estimate substituted into a model rather than a model has no likelihood, and so none of the parametric apparatus — a maximum, a profile, a standard error from curvature — is available to it. What is available is a fixed point, and the measurements here say what it is worth: about half the attenuated memory back under short memory, and a lower prediction error at every setting, without needing any parametric model to be right.

What this settles about the field’s choice #

The field’s earlier question, asked in iterating is not maximising, was which of the two first-order rules to use, and its answer was that the choice does not matter to any decision. This essay changes the question to which family of rules: a first-order likelihood, fitted jointly and correctly, or a band, fitted iteratively because it has to be. The answer has the shape the field has found everywhere it compared a model with a nonparametric estimate: the model wins when it is true, the estimate when it is not, and the margins are small either way — three thousandths here, seven there.

What the iteration adds is that the band need not be the one-step band. Read once, a band rule leaves two to four thousandths of prediction error on the table under every law measured, because the residuals it reads understate the dependence. Iterated, it recovers that much, converges on every draw, and needs no theory to run. Of the rules that do not assume a parametric shape, it is the one to use.

How the numbers were made #

Three hundred draws for each law, the response and predictors from the field’s predictive regression and the errors from each law’s exact covariance by a dense Cholesky factor. On each draw, at each window, the least-squares residuals give the first Bartlett band; each subsequent band is read off the previous generalised fit’s residuals in the original metric — not the whitened ones, which are independent by construction and would report no dependence at all — and the loop stops when the largest change in any coefficient is below $10^{-8}$. Prediction error is the field’s replicate error for the fitted coefficients. The first-order maximum is the field’s joint maximiser of the profile likelihood over the persistence, on the same draws. Every comparison is paired on the draw, so the standard errors quoted are those of a difference between two rules on the same series, which is why gains of a few thousandths on errors near one and a half are measurable at three hundred draws: the draw-to-draw variation in the error itself, which is far larger than any of the gains, is common to both rules and cancels out of the pair.

Checked on the measurement: the iteration converges on every draw at every law and window; it raises the residual lag-one correlation at every setting and leaves it below the errors’ own; and the prediction error at its fixed point is below the one-step band’s at every setting by more than two and a half paired standard errors.

Still open: whether the fixed point is unique #

Every run here starts from least squares, and every run converges. That does not say the fixed point is the only one: a residual-based update can have several, and the one reached can depend on the start. The measurement that would say is cheap — start the same loop from the residuals of an over-whitened fit, a band at thirty lags, or from the first-order maximum’s residuals, and see whether it arrives at the same coefficients. If it does on every draw, the fixed point is a property of the sample; if not, the band’s iterated estimate has a choice in it that nothing here prices. A second open question is a target other than prediction. The field’s coverage results say a covariance estimate enters an interval at first order, and an interval for a coefficient after the iterated band is the measurement that would say whether the half of the memory it restores is worth more there than the quarter of a per cent it is worth here.

What is claimed here, and what is not #

This essay takes the iterated Bartlett band, at four, eight and twenty lags, under three laws at 120 rows. The claims are that it converges on every one of 2,700 runs, at a median of seven to eleven fits and never more than 23; that it raises the residuals’ lag-one correlation from 0.714 to 0.750–0.757 under the first-order law, from 0.738 to 0.769–0.774 under the moving average and from 0.467 to 0.495–0.505 under long memory, against 0.8 in the errors; that it lowers prediction error by 2.04 to 4.32 thousandths under the short-memory laws and 0.83 to 1.33 under long memory, more than two and a half paired standard errors at every setting; and that its fixed point beats the first-order likelihood’s maximum under the moving average and long memory at eight and twenty lags and loses to it under the first-order law.

What stays out, named as a decision: other windows than Bartlett’s; a target other than prediction, deferred above; and the fixed point’s uniqueness, also deferred.

The boundary against iterating is not maximising is that it compared an iteration with a maximum on the one model that has both, and this essay runs the iteration on an estimate that has only the first. Against a dependence fitted with the line, whose joint fit is the parametric answer to the same attenuation, this essay is the nonparametric one, and the comparison between them is the last figure.