Choosing the order

Every number in the two essays before this one was computed with the order known. A first-order model was fitted to a first-order series, and the only thing being estimated was the value of the coefficients. Nobody is in that position. A series arrives with no label on it, and the first decision — before any estimate, any interval, any forecast — is how many lagged terms the model is going to have.

The standard answer is an information criterion: fit every order in a range, score each one by its fit penalised for its size, and take the smallest score. Two of them are in universal use and differ in a single number.

AIC = n·ln σ̂² + 2k and BIC = n·ln σ̂² + k·ln n

with k the number of coefficients. The first term is the fit — the residual variance, logged, scaled by the number of observations — and it falls as the model grows because a larger model can never fit worse. The second is the price of a coefficient, and the entire disagreement between the two criteria is whether that price is 2 or ln n.

The rate that does not depend on n #

The received account of the difference is that BIC is consistent and AIC is not: as the series grows, BIC’s probability of selecting the true order goes to one, and AIC’s does not. That is correct and it is a statement about a limit. What is more useful is the arithmetic underneath it, because it produces a number.

Take two nested models, the true order p and one lag more. The criterion prefers the larger model exactly when the improvement in fit outweighs the extra penalty:

n·ln(σ̂²ₚ / σ̂²ₚ₊₁) > penalty difference

The left-hand side is a likelihood-ratio statistic for a coefficient that is truly zero, so under the null it is χ² on one degree of freedom. The right-hand side is 2 for AIC and ln n for BIC. So the probability of adding one lag that is not needed is

• AIC: P(χ²₁ > 2) = 15.73%, which contains no n at all, and
• BIC: P(χ²₁ > ln n), which is 2.13% at two hundred observations and 0.66% at sixteen hundred.

That is the whole of consistency and inconsistency, in closed form, in two lines. AIC’s threshold is a fixed number of standard errors and never moves; BIC’s grows without limit, slowly, like the square root of the log.

The counted rates are larger than both, because the closed form is about one extra lag and the simulation selects over a whole range of them. Over orders up to eight, AIC lands above the truth 29.7% of the time at two hundred observations and 24.6% at sixteen hundred — the same rate, within the simulation’s own noise, across an eightfold change in sample size. BIC’s goes 2.3% to 0.4% over the same range.

Two routes, then, and they agree about the thing that matters: whether the rate depends on n. The closed form says AIC’s does not and the count confirms it does not. Neither route can confirm itself.

The half of it that nobody quotes #

Consistency is a statement about the limit and the limit is not where anybody works. At fifty observations — a perfectly ordinary length for a quarterly series with a decade of history — the two criteria are as follows.

AIC finds the true order 55.1% of the time. BIC finds it 54.1% of the time.

They are the same. And they are wrong in opposite directions: AIC lands above the truth 25.7% of the time and BIC lands below it 41.9% of the time. The criterion whose defence is that it does not overfit is, at this sample size, wrong more often than the one that does, because it has replaced one failure with a larger amount of the other.

The reason is the same arithmetic. BIC needs a likelihood-ratio statistic above ln 50 = 3.91 to include a lag, and a real coefficient of moderate size on fifty observations does not reliably produce one. The penalty that makes BIC consistent is the penalty that makes it blind at small n, and there is no setting of it that avoids both — a criterion is a threshold and a threshold trades the two errors against each other.

This is the site’s standing gotcha arriving in a new place. An assertion conditioned on a parameter is usually a fact about the default, and here the parameter is the sample size: the property AIC and BIC are compared on is a property of a limit, and at the sizes where the choice is being made it has not switched on. A reader told only that BIC is consistent will pick it for a fifty-point series and be wrong more often than if they had not.

It is the same shape as the winner’s curse, and worth naming as such. There, an estimate taken from the arm that was chosen for being ahead is ahead by more than it should be, and the defect is invisible in the estimate itself. Here, an order chosen for scoring best is chosen by a rule whose error rate is a property of a limit nobody is near, and the defect is invisible in the order itself. Both are selections made on the data and then reported as though the selection had not happened.

Why the counted rate is twice the single-lag one and not four times #

The closed form prices one extra lag at 15.73% and the counted over-selection over a range of orders is 29.7%, and the gap between those two numbers has an explanation that is worth having.

If the successive additions were independent decisions, the chance of over-selecting somewhere in six extra orders would be 1 − 0.8427⁶ = 64%. It is not, because the criterion takes a minimum over the whole range rather than a sequence of yes-or-no votes.

Write the criterion’s value at order p + k relative to its value at p. Each step adds a χ²₁ to the fit term and 2 to the penalty, so the sequence of differences is a random walk with increments χ²₁ − 2, whose mean step is −1. The criterion over-selects exactly when that walk ever goes positive, which is the probability that a negative-drift random walk has a positive maximum — and for these increments that is about 0.29.

So 29.7% is not the single-lag rate inflated by a search; it is the maximum of a random walk, and the reason it is roughly double 15.73% rather than four times it is that the walk’s drift pulls it back after each step.

That also gives AIC’s ceiling in closed form. Its under-selection vanishes as the series grows, so its accuracy tends to one minus the over-selection rate — about 71%, which is what the counted curve is flattening towards when it is described as settling just below eighty per cent. AIC does not asymptote at a number somebody has to measure; it asymptotes at the probability that a random walk with steps χ²₁ − 2 ever turns positive.

Where the two cross, more precisely #

The crossing is placed “somewhere in the seventies” and the three measured sizes locate it lower.

At fifty observations AIC leads by 1.0 point; at a hundred BIC leads by 11.9. Interpolating in log n puts the crossing at about n = 55.

The caveat is the size of the fifty-observation gap. One point on seven hundred series is well inside the counting error of about 1.9 points, so the two are tied there rather than measurably ordered, and anything from about forty-five to seventy is consistent with the three readings.

What is not in doubt is the shape of the advice either side of it. Below about fifty observations the two criteria are indistinguishable and both are wrong about half the time; by a hundred BIC is ahead by twelve points and by two hundred by twenty-five. So the practical rule is not that BIC is better — it is that BIC is better everywhere the choice is decidable at all, and at the sizes where it is not better it is not worse either.

Where the crossing is #

Between fifty and sixteen hundred the two criteria change places, and the size at which they do is worth locating because it is the only practical thing this comparison produces.

At fifty they tie at 55.1% and 54.1%. At a hundred BIC is ahead, 81.0% against 69.1%. At two hundred it is 95.0% against 70.3%, and from there the gap only widens: AIC’s accuracy flattens out just below eighty per cent while BIC’s climbs to 99.6%.

So the crossing is somewhere in the seventies, and on either side of it the advice is different. It also depends on the truth: the second coefficient in the series used here is −0.3, which is a real effect of moderate size. Make it smaller and BIC needs longer to see it, and the crossing moves right; make it larger and BIC finds it immediately and the crossing moves left. The sample size at which the consistent criterion becomes the better one is a function of the coefficient nobody has, which is this site’s most-repeated finding arriving in a place where it is rarely stated.

Overfitting is not obviously the expensive error #

There is an objection to everything above, and it is a good one. AIC is not trying to find the true order. It is an estimate of out-of-sample prediction error, and if the aim is a forecast rather than a description of the process, then including a lag whose coefficient is nearly zero costs one parameter’s worth of variance and no bias at all — which is cheap. Leaving out a lag whose coefficient is real costs bias, which does not shrink with anything.

So the two errors are not symmetric in what they cost a forecast, and the criterion that makes the cheaper error more often may well be the better one to forecast with, even though it is right about the order less often.

That is an argument about a different quantity, and the honest response is to measure the different quantity rather than to argue.

The measurement, taken there rather than here, is that the criterion which is right about the order more often also produces the better interval — so the objection is a real one and it does not survive the count. But it survives long enough to be worth stating, because it is the reason AIC is defended and because “which criterion is better” turns out to have different answers for different questions, only one of which has been asked here.

The trap underneath, and it is worse than either criterion #

There is a step before any of this that decides the answer and is almost never stated.

Fitting an autoregression of order p requires p lagged values, so the first p observations cannot be used as responses. An order-1 model on a hundred observations has ninety-nine responses; an order-8 model has ninety-two. The natural implementation — a loop over p, each fitted from its own first usable observation — gives every order a different sample.

An information criterion compares log-likelihoods. Log-likelihoods computed on different numbers of observations are not comparable, and the failure is not a bias that could be argued about. It is a loss of invariance.

Multiply the whole series by a constant c. Every residual variance is multiplied by c², so every row’s first term gains 2·nₚ·ln c. Where every row shares a sample that is one constant added to every row and the argmin cannot move. Where the rows have different nₚ it is a different constant on each row, and the order selected becomes a fact about the units the series was recorded in.

Measured over eight hundred series from a second-order process: the shared-sample table selects a mean order of 2.66 and selects the identical order for every series at scales spanning a factor of ten thousand. The ragged table selects a mean order of 0.90 at one scale, 3.06 at another and 8.00 — the largest offered, on every single series — at a third. The two tables disagree on 35.3% of series at the natural scale alone.

Neither of the ragged answers is about the series. A model selection that changes when a measurement is reported in millimetres rather than metres has stopped being a statistical procedure, and every number in it is finite, ordered and entirely plausible on the way past.

This is why the field’s library has an orderTable function rather than a loop over fitAR, and why the starting index is an argument rather than a default. The refusal beside it is the loop itself — kept, run, and required to disagree — because a check that the samples must be shared is worth nothing until it has been shown a table that does not share one.

What is being selected, and what is not #

Three boundaries, because “model selection” is a phrase that covers far more than what is measured here.

The order is a single integer here. Real selection ranges over differencing, seasonal terms and moving-average components as well, and the search is over a grid rather than a line. Nothing about the two criteria changes; what changes is the number of ways to be wrong, and the overfitting rates above are lower bounds for a wider search.

Neither criterion is estimating the truth. AIC is derived as an estimate of out-of-sample prediction error and BIC as an approximation to a posterior model probability. They answer different questions, and the fact that both are used to pick one number and compared on how often they get it right is a compression the literature made and this essay has followed. Under that compression AIC is being judged on something it never claimed.

A criterion is not the only way to choose. Order can be read off a correlogram by eye, chosen by sequential testing, or picked by cross-validation on held-out observations, and each of those has its own failure modes. Sequential testing in particular has one this site has already measured at length: a sequence of tests at the nominal level does not hold the nominal level, and what looking repeatedly costs is exactly the arithmetic that would apply to a stepwise search over orders. None of those alternatives is measured here.

And selecting is not the same as selecting and then using. Every rate above is about the order. What the choice does to the interval computed afterwards is the next essay, and it is where the two criteria’s comparison finally separates them — in a direction neither the closed form nor the counts above would predict.

The check, and the two things it is arranged to reject #

Two claims are gated in this field’s library.

The rate is constant for one and not the other, asserted as a comparison between a hundred observations and sixteen hundred, with the closed forms beside it: AIC’s counted over-selection has to stay within six points across the range, BIC’s has to fall below 3%, and the closed form for BIC has to be smaller at the larger n while AIC’s is required to be the same number at both. That last is the one that would catch a chiSquare regression, because it asserts an equality that the arithmetic makes exact.

There is one more thing that check is arranged to catch and it is worth saying because it is a common way for a simulation like this to lie. Both criteria are computed from the same fitted models — one orderTable call returns every order with both scores attached, so AIC and BIC are never looking at different fits of the same order. Had they been computed in separate passes, a difference between them could have come from anything, and the comparison would have measured the implementation rather than the penalty. Every rate in this essay is a comparison within seeds and within fits: the two criteria see identical residual variances and disagree only about what to charge for a coefficient.

And the table’s rows share a sample, asserted as an invariance rather than as a bias. The shared table must select the identical order for all eight hundred series at three scales spanning ten thousand, and the ragged one must select a mean order at least three apart between the extreme scales. If the ragged table were to become scale-invariant the check throws explicitly, because a refusal that has stopped refusing is a refusal that proves nothing — which is the third time in a row this site has had to state that rule, and the reason it is stated again here.