Iterating is not maximising

A practitioner told that the two-step estimate of a serial correlation is attenuated has an obvious next move, and it is not to write down a likelihood. It is to iterate: whiten at the estimate, refit, read the correlation off the new residuals, whiten again, and keep going until nothing moves.

It does converge. Under a first-order autoregression at 0.8 it takes 6.9 steps on average and never more than fourteen, and it gets most of the way: from 0.7237 to 0.7673 against a maximum-likelihood answer of 0.7762.

The last tenth of that distance is the subject of this essay, because the reason it is not covered is a term rather than a tolerance.

One curve, three answers #

The Gaussian likelihood of a regression with first-order errors, concentrated over the coefficients and the error variance, is a function of one number:

$$\ell(\rho) = -\tfrac{n}{2}\log \hat\sigma^2(\rho) + \tfrac12 \log(1 - \rho^2).$$

The first term is a sum of squares on the whitened sample. The second is the Jacobian of the whitening — the first row is scaled by √(1 − ρ²) and every other row is a difference, whose Jacobian is one — and it is there because a likelihood written in transformed coordinates that does not carry the determinant of the transform is not a likelihood.

On the sample drawn here the two-step rule reads 0.7616 off the least-squares residuals and gives up 0.304 of log-likelihood. Iterating moves it to 0.8080 and gives up 0.0023. The maximum is at 0.8044.

Notice what the middle number does. The fixed point is not between the two-step answer and the maximum: it is past it. The iteration overshoots on this sample, and the likelihood it reaches is nevertheless below the maximum, because the maximum is a maximum.

The rule that is not one rule #

“The two-step rule” names a family rather than a procedure, and two of its members differ in a way that matters here.

The whitening used throughout this collection is the Prais–Winsten transform, which scales the first row by √(1 − ρ²) and differences the rest. The older Cochrane–Orcutt form simply drops the first row. On a hundred and twenty rows that is a row of data thrown away, and worse, it is a different row for candidates that need different numbers of lags — which is the shared-sample trap the criterion field refuses, because two criteria computed on two different subsets of the rows are not comparable.

It is also the reason the Jacobian term exists at all. The scaled row is the only row whose transformation has a determinant other than one; a rule that drops it has no determinant to carry, and the likelihood it maximises is a likelihood for n − 1 rows rather than for n. So the difference this essay measures is not available to Cochrane–Orcutt in any form: it is a difference about a row that rule does not have.

Where the iteration goes #

The first step is nearly all of the movement — 0.7616 to 0.8049 — and the remaining seven steps move the estimate by 0.003. That shape is what a contraction looks like, and it is why the iterated rule is a rule rather than a search: the map has a fixed point, the map is a contraction near it, and eight iterations of a whitening and a least-squares fit cost a fraction of what a golden-section search on a profile costs.

What the fixed point solves is

$$\hat\rho = \frac{\sum_t e_t(\hat\rho)\,e_{t-1}(\hat\rho)}{\sum_t e_t(\hat\rho)^2},$$

where the residuals are the generalised least-squares residuals in the original metric rather than the whitened ones. That distinction is worth a sentence: the whitened residuals are independent by construction, so a rule reading the correlation off those would report zero at every step and converge instantly to nothing.

The equation above is the first-order condition of the sum of squares in ρ, and nothing else. It has no Jacobian in it, because the Jacobian does not depend on the coefficients and so never appears in a residual update.

The omitted term, evaluated #

The whole difference between the two rules is one term in the objective, so it is worth computing that term rather than describing it.

At ρ = 0.8 the Jacobian contributes $\tfrac12\log(1-0.64)$, which is −0.511, to the log-likelihood, and across the range the three answers occupy — 0.7237 to 0.8080 — it moves from −0.371 to −0.526. A swing of 0.155, which is sixty-seven times the 0.0023 the iteration gives up.

More to the point is its derivative, because a maximiser is looking for a stationary point rather than a level:

$$\frac{d}{d\rho}\;\tfrac12\log(1-\rho^2) = -\frac{\rho}{1-\rho^2} = -2.22 \quad\text{at } \rho = 0.8 .$$

The iteration finds where the sum-of-squares term alone is stationary. The likelihood is stationary where that term’s slope is +2.22 instead. So the iteration is solving the right equation with a term of size 2.22 missing from it, and since the missing term is negative and grows in magnitude with ρ, the fixed point sits above the maximum.

That is the direction, derived rather than observed, and it is why the overshoot on the worked sample is not a curiosity: it is what a positive-slope omission does.

What eighty-three per cent looks like on both readings #

The averages and the single sample say different things about whether the iteration overshoots, and both are worth having.

On average the iteration goes from 0.7237 to 0.7673 against a maximum-likelihood 0.7762, closing $(0.7673-0.7237)/(0.7762-0.7237)$, which is 83% of the distance, and stopping short.

On the worked sample it goes from 0.7616 to 0.8080 against a maximum at 0.8044 — past it, by 0.0036.

So the overshoot is a property of individual samples rather than of the rule, and the rule’s average behaviour is to undershoot by about a sixth of the gap. Both follow from the same missing term: it displaces the fixed point upward in ρ by an amount that depends on the curvature of that sample’s own likelihood, and the curvature varies enough from draw to draw for the displacement to land on either side of a maximum that is itself moving.

In the currency a criterion charges in, the two-step rule’s 0.304 is about six tenths of a parameter of likelihood thrown away, and the iteration recovers all but five thousandths of one.

A difference with a direction #

One sample proves nothing about a direction, and the direction is the claim.

Two readings of the same comparison say different things and both are worth having.

The likelihood at the fixed point is below the maximum on 100% of draws, by 0.0557 on average. That is a definition rather than a finding — a maximum is a maximum — and it is here as a check that the two rules are being compared on the same curve.

The coefficient is the finding. The fixed point lands below the maximum on 36% of draws and above it on the rest, and its mean sits 0.0089 below the maximum at 7.3 paired standard errors. A spread that straddles zero with a mean that does not is exactly what a systematic term looks like when it is smaller than the noise on a single draw, and the term is the ½log(1 − ρ²): it rises as ρ falls, so it pulls the maximiser towards zero relative to the sum of squares — which, on average and against a fixed point that overshoots, puts the maximiser above.

The same three columns under the other laws give paired gaps of 0.0077, 0.0063 and 0.0092, at 6.9, 7.4 and 7.4 standard errors. It is the same term doing the same thing in four different worlds, which is what makes it a mechanism rather than a feature of one.

What the whole ladder looks like #

Put the three estimates next to the two fixed points every comparison in the estimated-covariance field is read against, and the sizes fall into place.

The law is 0.800. The two-step rule is short by 0.0763, the iterated rule by 0.0327, and the joint rule by 0.0238. So iterating recovers 0.0436 of the 0.0525 gap between the two-step answer and the maximum — 83% of it — and the likelihood’s own term is worth the remaining seventeen.

The spreads barely move: 0.0678, 0.0632, 0.0628. Nothing here is a variance–bias trade. The iteration and the maximisation are both removing the same bias, and they remove different amounts of it.

And it changes nothing #

The regret of the three rules on the same fifteen-candidate table, over three hundred draws, is 0.01771, 0.01554 and 0.01549 — against 0.01551 for a whitening told the whole covariance. The paired difference between the fixed point and the maximum is 0.00006 at 0.1 standard errors.

So the two things this essay separates are, for the purpose they exist for, the same thing. Iterating recovers 83% of the coefficient’s bias and 98% of the decision’s cost; maximising recovers the rest of the first and none of the second.

That is not an argument for either rule. It is an argument about what a comparison is for. A textbook comparison of these two estimators would report the coefficient, find a difference at seven standard errors, and stop; and a reader would take away that the maximum likelihood estimator is meaningfully better. Measured on the thing the estimator feeds, it is not — and the only way to know that is to run the criterion.

Under long memory the gap is real: 0.00027 at 3.0 standard errors, out of a total available gain of 0.02525. It is a ninetieth of what the whitening is worth. Under the break it is 0.00059 at 0.7, and under the moving average 0.00044 at 1.4 the other way.

The reason the decision is so insensitive is worth stating, because it is not that criteria are blunt. The fifteen candidates are all whitened by the same transform, whichever ρ̂ is used, so a change in ρ̂ moves every criterion in the table together and mostly cancels out of the differences the ranking depends on. What does not cancel is the change in the effective number of independent rows each candidate is scored on, and that is second-order in ρ̂ near the truth. So a criterion that is very sensitive to whether the sample is whitened is nearly insensitive to at what value — which is the same observation the trace that is the optimism makes from the penalty’s side.

Why the overshoot #

The fixed point going past the maximum on nearly two thirds of draws is the part that is easiest to misdescribe, so it is worth one paragraph of mechanism.

The iteration’s update is a ratio of a lag-one cross-product to a sum of squares, both computed on residuals that were produced by a whitening at the previous estimate. A whitening at ρ removes ρ times the previous row from each row, and if ρ is too small the whitened series still carries positive dependence — so the next update raises it. The map is therefore increasing and has a fixed point where the generalised residuals carry no lag-one dependence at all, which is a stronger condition than the likelihood asks for: the likelihood is content with residuals whose remaining dependence is exactly offset by the Jacobian’s pull.

So the fixed point is where the whitening is complete and the maximum is where the whitening is worthwhile, and complete is further out.

The overshoot is not universal — a third of draws land below — and that is the shape of a term that is smaller than the sampling noise on any one sample. It is the same shape the winner’s curse takes at low power and a bias in a persistence parameter takes at short horizons: a distribution that straddles the truth with a mean that does not, where every individual draw is compatible with there being nothing and two hundred of them are not.

What a fixed point cannot say #

There is one thing the iterated rule can do that the maximiser cannot, and one thing it cannot.

It generalises without a likelihood. The update above is a formula in residuals, so the same iteration runs for a whitening built from a window or from a fitted autoregression — constructions that have no likelihood to maximise because a tapered covariance estimate is an estimate substituted into a model rather than a model. That is the practical case for iterating and it is a real one: the rule that is available everywhere is worth something over the rule that is available in one place.

What it cannot do is say how far it is from anything. A fixed point is a solution of an equation and carries no scale; a likelihood carries one, and the 0.0023 the fixed point gives up on the sample drawn above is a number only because there is a maximum to measure it against. Every claim in this essay about the size of the difference is a claim the iterated rule could not have made about itself.

The cost of each, in what it actually costs #

The last comparison worth making is the one a practitioner would make first, and this collection almost never does: how long each rule takes.

The iterated rule runs a whitening and a least-squares fit per step, and takes 6.9 steps on average under the geometric law, 6.5 under the moving average, 8.1 under long memory and 7.5 under the break — never more than twenty-one on any draw in any world. The maximiser runs a grid of forty-one profile evaluations to bracket the maximum and then a golden-section refinement inside the bracket, each evaluation being the same whitening and fit. So the maximiser is roughly seven times the work of the iteration, for a difference in the selected model of 0.00006.

That ratio is the honest summary of the field’s first half, and it is the sort of number the field on what a reference distribution costs exists to make ordinary: a rule is not better because it is more principled, it is better if the thing it does better matters, and whether it matters is a measurement.

What is claimed here, and what is not #

This essay takes the difference between a fixed point and a maximum. The claims are that the iterated two-step rule converges in about seven steps under every law here; that it recovers 83% of the two-step rule’s bias in the correlation; that the remaining seventeen belongs to the Jacobian term, which no residual-based update contains; that the fixed point’s likelihood is below the maximum on every draw and its coefficient is above the maximiser’s on about two thirds of them, with a mean 0.0089 below at 7.3 paired standard errors; and that the whole difference is worth 0.00006 of selection regret at 0.1 standard errors.

There is a fifth claim that is quieter and is the reason the essay exists. A difference at seven standard errors in a parameter can be worth nothing, and the only way to find that out is to measure the thing the parameter is for. That is the same discipline the essay on what a balanced trial is worth applies to an imbalance and the essay on where the gain is not the decision applies to an allocation rule, and it keeps producing the same shape of answer.

What stays out, and is named as a decision: which rule to use. Nothing here recommends one. The iteration is cheaper, the maximisation is a maximisation, and on this design they select the same models. What the essay claims is that the difference between them has a mechanism and a size, and that the size is not the one the coefficient advertises.

The boundary against the essay that fits the dependence with the line is that it prices what a joint fit is worth against a two-step rule, and this one takes apart what “joint” was doing — and finds that most of it was available to a rule that never wrote a likelihood down.

The checks, and the refusals that make them mean something #

Three claims are gated. The concentrated profile is required to equal the Gaussian log-likelihood evaluated at its own coefficient vector and variance, with the quadratic form written out on the transformed sample — because the concentration is an algebraic step and the Jacobian is exactly the piece an algebraic step drops. The iteration is required to converge, on every draw, within the iteration cap. And the fixed point’s likelihood is required to be below the maximum on effectively every draw, which is what says the two rules are on the same curve rather than optimising two different things.

The refusal is the one the field’s other half carries: a criterion minimised over a tuning list and then compared across candidates is refused, which is the same shape one level up — a rule that improves its own objective by searching, and a comparison that then reads the improved objective as though nobody had searched.