What differencing costs

The standard repair for a regression between wandering series is to regress the changes instead of the levels. It works, it is one line of code, and it is applied by habit to series that did not need it — so this essay measures both directions.

The repair works #

Two independent random walks, regressed on each other in levels, produce a significant slope 76.7% of the time. Regressed in first differences — each series replaced by the change from one step to the next — the rate is 4.9%, which is the nominal 5% within simulation error.

The reason is immediate once stated. A random walk is a running total of independent increments, so its increments are independent, and a regression of one series’ increments on another’s is a regression of noise on noise. The assumption that ordinary least squares needs is restored, exactly, rather than approximately.

That is a rare thing on this site: a repair that returns a broken rate to its nominal value at every sample size, with no residual error and no tuning parameter. It is also why the advice to difference is so firmly established.

It is worth noticing what makes it exact where the other corrections in this field are approximate. The effective-sample-size repair in the first essay estimates the dependence from the data and inherits the error in that estimate, recovering coverage to 89.1% rather than 95%. Differencing estimates nothing. It is an algebraic operation that removes a known structure exactly, and that is why it either works completely or damages the data completely, with no middle setting to tune.

The repair is not free #

The same operation applied where a relationship is real removes most of the evidence for it.

Take a case where the levels are genuinely tied together: y is x plus a stationary wobble, so the two series move together permanently and the connection between them is not an artefact. Regressed in levels, the R² is 0.91. Regressed in differences, it is 0.33.

Nothing about the relationship changed. What changed is which question is being asked. The levels regression asks whether the two series are in the same place; the differences regression asks whether they move at the same time, which they only partly do, because one step of x is a small part of what determines the gap between them.

So the two halves are both real and they are in tension. Differencing removes a false relationship completely and hides a substantial share of a true one, and neither effect is small.

What the fall in R² does to the verdict, which is less than it looks #

An R² falling from 0.91 to 0.33 sounds like a relationship being destroyed. Converted into the statistic anybody would actually read, it is not.

The t on a slope is $\sqrt{R^2(n-2)/(1-R^2)}$, so at a hundred observations the levels regression carries

$$\sqrt{\frac{0.91 \times 98}{0.09}} = 31.5,$$

and the differenced regression, on ninety-nine changes, carries $\sqrt{0.33 \times 97 / 0.67}$, which is 6.9.

A t of thirty-one becomes a t of seven. Both are overwhelming. Nothing about the decision changes, and a reader shown only the two verdicts would see no cost at all.

Where the cost lives is in how much data the two need. Reaching t = 2 takes about ten observations in differences and about three in levels. So at a hundred rows the differenced analysis has spent an enormous surplus and still has plenty; at fifteen or twenty rows it is the difference between a finding and nothing.

That is the honest scaling of “the repair is not free”. It is free at a sample size where the relationship is strong and the rows are many, and it costs roughly a factor of three in the sample size required — which is the number to carry into a setting where rows are expensive.

Two repairs, and what share of its own gap each closes #

The comparison with the effective-sample-size repair is worth making in the units both are measured in, because it is the sharpest illustration of what “algebraic” buys.

The estimating repair. An interval covering 47.0% is taken to 89.1% against a nominal 95%. It closes 42.1 of the 48 points available — 88% — and the residue is the error in the estimate of the dependence it had to make.

The algebraic repair. A rejection rate of 76.7% is taken to 4.9% against a nominal 5%. It closes 71.8 points of the 71.7 available — all of it, to within simulation error.

So one repair leaves six points of the thing it was fixing and the other leaves none, and the difference between them is not effort or sophistication. It is that the first estimates a nuisance and inherits its error, and the second performs an operation that removes a structure exactly and inherits nothing.

Which is also why only the second has a price. A repair that estimates something can be applied where it was not needed and will simply estimate a small number; a repair that transforms the data applies itself whether or not the structure it removes was there.

Cointegration is the case where the levels are right #

The situation above — two non-stationary series whose difference is stationary — has a name, and it is the reason “always difference” is not the conclusion.

Two series are cointegrated when each wanders without a mean but some combination of them does not. Consumption and income, two prices of the same commodity in different markets, a stock and its future: in each case both series wander freely and the gap between them does not, because something holds them together.

When that is true, the levels regression is estimating a real quantity — the long-run relationship — and differencing throws it away, keeping only the short-run co-movement. The R² of 0.33 against 0.91 is exactly that loss, measured.

The practical rule is therefore not “difference” but a decision:

If the series are cointegrated, model the levels (with an error-correction term, which uses both: the long-run relationship and the short-run changes).

If they are not, difference, because the levels regression is measuring nothing.

And deciding which requires evidence, since the two cases produce data that looks alike over a few hundred observations.

What over-differencing does #

The third case is a series that was stationary to begin with and gets differenced anyway, by pipeline or by habit. The failure is quiet and measurable in three ways.

It doubles the variance. For a stationary AR(1), Var(Δx) = 2γ₀(1 − φ), so at φ = 0 the differenced series has exactly twice the variance of the original. The measured ratio at fifty observations is 2.040 against a finite-sample prediction of 2.041, where the asymptotic form says 2 — the difference is the same sample-variance bias described in the first essay of this field, and putting it in makes the two routes agree to four digits rather than to one.

It installs a correlation that was not there. The differences of independent observations have a lag-one autocorrelation of exactly −0.5 in the process, and measure −0.479 on average over four thousand fifty-point series — the small gap being the usual downward bias of a sample autocorrelation, which is worth knowing about since it applies to every correlogram in this field. A plot that was clean now has a large negative spike at lag one, and an analyst reading it will conclude the data has structure. The structure was created by the analysis.

And it discards the level entirely. The mean of a differenced series is (last − first)/(n − 1), identically — an algebraic fact, checked here on four thousand realisations to machine precision. Whatever the other forty-eight observations of a fifty-point series were, they cancel.

That last one is not a loss of precision, it is a change of estimand, and it is why over-differencing does not show up as a wider interval. The differenced series is answering a different question perfectly well.

How to tell whether it was needed #

The diagnostic is the correlogram before and after, and the signature of over-differencing is specific enough to be actionable.

A series that needed differencing has autocorrelations that decay slowly — still substantial at lag ten or fifteen — and its differenced version has a correlogram that dies away quickly.

A series that did not has autocorrelations that were already small or decaying fast, and its differenced version has a large negative value at lag one and near-zero after. That negative spike is the fingerprint, it is close to −0.5 when the original was independent, and it does not occur naturally in the kinds of process this field is about.

The practical instruction: look at the correlogram of the differenced series, not only of the original. Differencing is applied to fix a diagnostic, and whether it fixed it or overshot is visible in exactly the same plot.

Differencing twice, and seasonal differencing #

Two extensions appear constantly in practice and both amplify everything above.

Second differences — differencing the differences — are occasionally right, for a series whose rate of change wanders rather than its level. They are much more often the result of a rule applied twice. Each application doubles the variance of an already-stationary series, so a doubly over-differenced series has four times the variance of the original and a correlogram with negative values at two lags.

Seasonal differencing subtracts the value one period earlier — twelve months, seven days, four quarters — and is the standard treatment for a repeating annual or weekly pattern. It carries the same risk in a form that is easier to miss, because a seasonal pattern that is stable rather than wandering should be modelled with seasonal terms rather than differenced away, and differencing it installs a negative correlation at the seasonal lag instead of at lag one.

The combination — one ordinary difference and one seasonal — is the default in a great deal of forecasting practice, and it is two decisions taken at once. Where both were needed it is right; where neither was it produces a series with negative correlations at lag one and at the seasonal lag, twice the variance in each direction, and no level information at all.

The diagnostic is unchanged and so is the instruction: look at the correlogram of what comes out, not only at the correlogram of what went in.

The pipeline problem #

Over-differencing is rarely a decision. It is usually a default: a forecasting library that differences until a test passes, a template that starts with diff(), a colleague’s script reused on a new series.

Two things make it durable.

The output stays plausible. A differenced series has a mean near zero, a correlogram with one negative spike, and a perfectly ordinary appearance. Nothing about it announces that an unnecessary operation was applied.

And the test that is automated tends to be one-sided. Automated differencing usually applies a unit-root test and differences when it fails to reject a unit root — a test with limited power, so it frequently fails to reject on stationary series with strong dependence, and the default action on an inconclusive answer is to difference.

The site’s own gate feeds the machinery a differenced independent series and requires the check to find the negative lag-one correlation, for the same reason every other refusal here exists: a recommendation to inspect a diagnostic is worth nothing unless the diagnostic has been shown to catch the case it exists for.

The two questions it answers, and only one of them was asked #

The clearest way to hold the levels-or-differences decision is to notice that they answer different questions and that only one is usually the question of interest.

Are these two series in the same place? That is the levels regression: a statement about positions, meaningful when both series have positions to be in — either because they are stationary, or because they are tied together.

Do these two series move together? That is the differences regression: a statement about changes, always meaningful, and much less informative when the interesting relationship is a long-run one.

The 0.91 and the 0.33 measured above are answers to those two different questions on the same data, and neither is wrong. What would be wrong is reporting one while discussing the other — a differenced analysis described as showing that two quantities are related, or a levels analysis described as showing that they move together.

The failure that motivates this whole field is a special case: for two independent random walks neither question has an affirmative answer, and the levels regression answers the first one wrongly 76.7% of the time.

The other repairs, briefly #

Differencing is the crudest of several treatments and the others are worth naming, because “difference or nothing” is a false choice.

A correlated error model. Fit the regression in levels with an error structure that allows the residuals to be autocorrelated, and the standard errors account for it directly. This is available in standard software, keeps the levels, and requires estimating the dependence from the same data.

Robust standard errors. Newey–West and its relatives compute a standard error that is valid under autocorrelation of unknown form, without changing the estimate. The estimate stays the ordinary least squares one; only its advertised precision changes, which is the honest minimum.

A block bootstrap. Resample contiguous blocks rather than individual observations, so the dependence within a block is preserved. This is the bootstrap adapted for dependent data, and it inherits both the strengths and the failure modes recorded there.

And an explicit model of the process. An ARIMA or state-space model treats the dependence as the object of study rather than as a nuisance, which is the right move whenever the dynamics are themselves of interest.

All four share the property that they require the dependence to be estimated from the data, so all four degrade in short series, which is the same limit found everywhere in this field.

A repair with a cost is the normal case #

It is worth putting this essay’s shape beside the rest of the site, because the pattern is the same one several other fields end on and it is the reason none of them ends with a recommendation.

Correcting for multiple comparisons removes false positives and costs power. Stopping rules with proper boundaries hold the error rate and require more observations for the same power. A conservative interval covers what it claims and is wider than one that does not. Differencing removes a false relationship and hides most of a true one.

In every case the repair is correct, the cost is real, and the decision needs a quantity the data does not by itself supply — how many of the hypotheses are true, how the trial will be monitored, whether the series are cointegrated. What this site can do is put a number on each side of the trade, which is what turns “difference the series” from advice into a decision.

The corresponding failure is treating any of these repairs as free and applying it by default. A pipeline that differences everything is making the same class of mistake as one that Bonferroni- corrects everything or one that always uses the widest interval available: buying a guarantee that was not needed, with power that was.

What this establishes #

Differencing is a genuine repair and a genuine cost, and both are measured rather than argued.

It takes the false-positive rate between two unrelated random walks from 76.7% to 4.9%, exactly and at every sample size, because the increments of a walk really are independent.

It takes the visible strength of a real long-run relationship from 0.91 to 0.33, because the levels and the changes are different questions and only one of them is about where the series are.

And applied where it was not needed it doubles the variance, puts a −0.5 correlation into data that had none, and reduces the mean to a function of two observations out of fifty.

The asymmetry between the two directions is worth one last sentence, because it decides what to do under uncertainty. Failing to difference a series that needed it produces a finding that is not there — 76.7% of unrelated pairs called significant. Differencing a series that did not need it produces a weaker analysis of a real question: twice the variance, a manufactured correlation at lag one, and a level that has been discarded.

Neither is acceptable and they are not equally bad. One invents results and the other loses them, which is an argument for differencing when the evidence is genuinely ambiguous — provided the differenced series’ own correlogram is then read, since that is where the overshoot shows.

The decision it rests on — are these series individually non-stationary, and are they tied together — is not one this essay can settle from the data, and the diagnostic that informs it is the subject of the last essay in this field, along with the uncomfortable measurement of how often that diagnostic misses.