Two walks and a finding

The previous essay was about an interval that is too narrow. This one is about a relationship that is not there, which is a different kind of failure: the first exaggerates something real, the second manufactures something absent.

The experiment #

Generate two random walks of a hundred steps each. Each is a running total of its own independent normal draws; the two share no input, no seed and no mechanism.

Regress one on the other and record the p-value for the slope. Repeat three thousand times.

The slope comes out significant at the 5% level 76.7% of the time. The median R² across the three thousand pairs is 0.172.

The calibration that makes those numbers mean something is the same experiment with independent noise instead of walks, where the rejection rate is 4.7% — the nominal 5%, within simulation error. The regression machinery is not broken. It is being applied to series for which its assumption about the errors is false.

Why the t statistic is not a t #

The regression’s standard error for the slope is computed on the assumption that the residuals are independent. Between two random walks they are anything but: the residual at one step is nearly the residual at the previous step, because both series moved only a little.

So the standard error is computed from a divisor that assumes a hundred independent pieces of information, when the effective number is a handful. The estimate is divided by something far too small, the t statistic comes out far too large, and it is compared against a distribution it does not follow.

The picture is the clearest statement of the failure available. The t distribution that the p-value is read from is the narrow curve; the statistic’s real distribution is the wide spread of bars behind it. Reading a value from the second against the first is what produces a 76.7% rejection rate at a nominal 5%.

What a 76.7% rejection rate says about the standard error #

The rejection rate can be turned back into the quantity that produced it, which puts a size on “the standard error is far too small”.

A two-sided test at 5% rejects when the reported statistic exceeds 1.96. If the reported statistic is the properly scaled one multiplied by k, the test rejects whenever the properly scaled one exceeds 1.96/k, so a rate of 0.767 gives 1.96/k = 0.296 and

$$k = 6.6 .$$

The standard error is about six and a half times too small, which is a variance understated by a factor of forty-four.

Since a standard error scales as $1/\sqrt{n}$, that is the arithmetic behind the phrase the effective number is a handful: a hundred steps of a random walk carry about 2.3 observations’ worth of information about a slope, by this reckoning. The calculation treats the properly scaled statistic as standard normal, which is not exactly right in a spurious regression — the limiting distribution is a functional of Brownian motion rather than a normal — so 2.3 is an order of magnitude rather than a measurement. The order of magnitude is the point.

And what the median R² says, independently #

The second reported number gives the same conclusion by a route that does not go through a standard error at all.

On independent data with a hundred observations and one regressor, R² under a true null has mean $1/(n-1) = 0.010$ and a median near 0.0045. Between two random walks the median is 0.172.

Nearly forty times the median an honest null produces, on data with nothing whatever connecting it.

The two readings agree, which is worth checking rather than assuming. An R² of 0.172 on a hundred observations gives $t^2 = 0.172 \times 98 / 0.828 = 20.4$, so the median pair carries a t of about 4.5 against a critical value of 1.98. A median statistic more than twice the threshold is exactly what a rejection rate in the high seventies looks like, and it arrives from the fit rather than from the test.

More data makes it worse #

The property that distinguishes this failure from every other on this site: it does not improve with sample size, it deteriorates.

At 25 steps the rejection rate is 53.7%. At 100 it is 76.7%. At 400 it is 88.0%.

Every other error measured on this site — a Wald interval’s undercoverage, a normal approximation’s tail, a small-sample t — is a finite-sample problem that shrinks as data accumulates. This one grows, and it grows because the quantity being estimated does not exist. Two random walks have no true correlation to converge on; each drifts, and over a longer window they drift further, so the apparent relationship between them strengthens.

That is worth stating as a rule, because it inverts the usual instinct. Where the failure is a misspecified model rather than a small sample, more data increases confidence in the wrong answer. Collecting more is the standard response to an uncertain result and it is precisely the wrong response here.

What a random walk is, and why it breaks the machinery #

A random walk has no mean. Not an unknown mean — no mean at all: its expected value at step t is wherever it started, and its variance grows in proportion to t without limit.

That breaks the assumption underneath ordinary regression in a specific way. Least squares treats the x values as a fixed set of numbers spread around their average, and its standard error is built from the spread of the residuals around a line. When both series are non-stationary, the residuals are themselves a random walk — the difference of two walks is a walk — so the “errors” wander instead of scattering, and every quantity computed from their spread is wrong.

The vocabulary for this is the unit root: the process $x_t = \phi x_{t-1} + \varepsilon$ has one when φ = 1, and the whole character of the series changes at that point rather than degrading smoothly.

The change is genuinely discontinuous. At φ = 0.99 the series is stationary, has a mean, returns to it eventually, and the arithmetic of the previous essay applies with an inflation factor of 199. At φ = 1 there is no mean to return to and no inflation factor to compute. In a series of two hundred observations the two are almost indistinguishable by eye or by test.

Strong dependence without a unit root is bad enough #

The unit root is the extreme case and it is not required for the failure. Two independent stationary series with a lag-one correlation of 0.8, regressed on each other, reject the true null 34.0% of the time at a hundred observations.

That number deserves attention because it is the case most analysts believe is safe. The series is stationary, it has a mean, standard tests will not flag a unit root, and every textbook condition for ordinary regression appears satisfied except the one about independent errors — and the rejection rate is nearly seven times the nominal level.

So the practical boundary is not “avoid regressing non-stationary series”. It is that autocorrelated residuals invalidate the standard error, and the strength of the dependence decides how badly, with the unit root as the limiting case where it fails completely.

Why it is so persuasive #

Every ingredient of a convincing result is present and each of them is doing what it should.

The R² is respectable. A median of 0.17 across pairs with no relationship, and 30% of pairs exceed 0.35. That is a scatter plot with a visible slope.

The p-value is small. Not marginal: at four hundred steps a large share of pairs produce p below 0.001, which reads as overwhelming.

The picture looks right. Two series that both wander upwards over the window produce a scatter that any reader would describe as a relationship.

And a story is always available. Any two real series that both trend have a mechanism somebody can propose connecting them, and the mechanism will be plausible because it was constructed after seeing the data. That is the forking-paths problem in its most comfortable habitat: the analysis was not chosen from many, but the explanation was.

The four together are why the classic examples in this area — a stock index against a rainfall series, one country’s suicide rate against another country’s cheese consumption — are funny rather than obviously wrong. They have the whole apparatus of a finding.

Two kinds of trend, which need opposite treatments #

The single most consequential distinction in this area is between two things that look identical on a chart, and it is worth drawing carefully because the standard repairs are wrong for one of them.

A deterministic trend is a series that equals a straight line plus stationary noise. It has a definite level at every time, it returns to the line after a disturbance, and the right treatment is to put the line in the model — regress on time, and analyse what is left.

A stochastic trend is a random walk. There is no line to return to, and each disturbance is permanent: a shock at step ten shifts every subsequent value forever. The right treatment is to difference, and regressing on time does not help at all.

Applying the wrong repair to either fails in a knowable direction. Differencing a trend-stationary series over-differences it — the subject of the next essay — and installs a negative correlation at lag one. Detrending a random walk leaves a residual that is still a random walk, so the spurious regression problem survives the repair entirely.

The two are hard to tell apart in short series and are distinguished by unit-root tests, which have low power in exactly the cases where the distinction is difficult. That is an honest limitation of the field rather than a gap in the tooling: two hundred observations often do not contain enough information to decide, and an analysis that depends on which of the two it is should say that it does.

An old result, rediscovered regularly #

This is not a recent finding and its history explains why it keeps reappearing.

Yule described “nonsense correlations” between time series in 1926, with the observation that two series each of which wanders will correlate with each other, and that the usual significance test says nothing useful about them. Granger and Newbold measured it by simulation in 1974 and gave the modern name, spurious regression, along with the practical advice that a regression between time series with a high R² and a low Durbin–Watson statistic should be treated as spurious until shown otherwise.

The rule of thumb they proposed — suspect a regression whose R² exceeds its Durbin–Watson statistic — is crude and remains a good first filter, precisely because the two quantities move in opposite directions under the failure: the R² inflates while the residual autocorrelation drives the Durbin–Watson towards zero.

What makes the result keep needing rediscovery is that it lives in a corner of the syllabus. It is taught thoroughly in econometrics, where non-stationary data is unavoidable, and barely at all in the general statistics or machine-learning curricula — where the same data types appear as “sensor readings”, “engagement over time” or “monthly totals”, and the same regression is run.

What it is not #

Three distinctions, because this failure is often merged with others that have different fixes.

It is not the “correlation is not causation” point. That one says a real association may have a different cause than the obvious one. Here there is no association: the two series are independent by construction, and the relationship is created by the standard error rather than misattributed.

It is not a small-sample problem. It is worse at four hundred than at twenty-five.

And it is not fixed by adding controls. Regressing on a third series, or including a time trend, changes the numbers and does not address the mechanism, because the residuals are still autocorrelated. A trend term helps when the series are trend-stationary and does nothing when they have unit roots, and the two look alike.

What to do, in outline #

The repairs are the subject of the next essay, and the shape of them is worth stating here because it decides what the next essay measures.

Difference the series and regress the changes. Two independent walks have independent increments, so the false-positive rate returns to nominal — measured at 4.9%. This works and is not free.

Or model the dependence explicitly, with a correlated error structure, and let the standard error account for it. This keeps the levels and requires estimating the dependence from the same data.

Or establish that the levels are genuinely tied together, which is what cointegration means: two non-stationary series whose difference is stationary really do move together, and the relationship is estimable. That is a claim requiring evidence rather than an assumption to be made.

All three require deciding first whether the series are stationary, which is the check the last essay of this field is about.

Where this is met outside economics #

The failure needs two ordered series and a regression, and that combination is more common than the field’s vocabulary suggests.

A/B tests analysed by day. Two variants’ daily conversion rates, regressed on each other or on a covariate, with each day’s value correlated with the day before because the underlying population and traffic mix drift. The row count is the number of days; the effective size is much smaller.

Sensor and telemetry correlations. Two instruments logging every second, on processes that drift over hours. Any pair of such channels will correlate strongly over a long enough window, and dashboards that surface “correlated metrics” automatically are running exactly the experiment measured above, thousands of times, and reporting the results that clear a threshold.

Aggregate social indicators. Two national statistics both rising over decades — enrolment, consumption, an index of anything. Both are near-random walks over a few dozen annual observations, and the regression between them carries every visible sign of a finding.

And model residuals in ordinary regression. Fitting a model to data collected over time and checking the residuals against a fitted value hides the ordering entirely. Nothing in a residual-versus-fitted plot shows autocorrelation, which is why the diagnostic in the last essay of this field is a plot against time rather than against anything the model produced.

The common feature is that the ordering is present in the data and absent from the analysis. Where a dataset has a time column that no step of the modelling refers to, the assumption of independent errors has been made silently, and the measurements here say what it is worth.

What this establishes #

Two independent series, regressed on each other, produce a significant slope 76.7% of the time at a nominal 5%, with a median R² of 0.17 — and 88.0% of the time when the series are four times as long.

The mechanism is the same one as the interval that covers 47%: a standard error computed as though the observations were independent, when they repeat each other. The consequence is different in kind. An interval that is too narrow overstates a real quantity; a regression between unrelated series reports a quantity that does not exist, with every visible sign of a result.

The difference matters for what a reader should do about it. A too-narrow interval can be widened after the fact. A finding produced this way cannot be corrected downward, because there is nothing underneath it to correct to.

There is one consolation and it is a practical one. The failure needs both series to be dependent, so it is bounded by whichever of them is closer to independent: regressing a wandering series on a genuinely independent one does not produce the same inflation, because the residuals inherit the dependence of only one side. That is why a controlled experiment with randomised, independent treatment assignment is safe even when the outcome is measured over time — the randomisation essay’s reference distribution does not care how the outcome wanders.

It is also the clearest case on this site of a measurement that only exists because the truth was generated rather than collected. No real dataset can demonstrate a spurious regression, because with real data nobody knows whether the relationship is there — which is precisely the state the seventy-seven per cent describes. The two series here were built to have nothing in common, so the rejection rate is a count of errors rather than an interpretation of results, and that is the only way the number could have been obtained.