R² is not a measure of fit

Take thirty observations of pure noise. Regress them on twenty predictors, every one of which is also pure noise, unrelated to the response and to each other. The true relationship is nothing at all.

The R² is 0.69.

Why it can only go up #

The mechanism is not subtle and it is worth stating before the measurement, because it makes the result inevitable rather than surprising.

R² is one minus the residual sum of squares over the total sum of squares. Fitting is minimisation: least squares finds the coefficients that make the residual sum of squares as small as possible.

Add a predictor. The old model is still available — set the new coefficient to zero — so the new minimum cannot be larger than the old one. The residual sum of squares can only fall or stay the same, so R² can only rise or stay the same.

That is true of every predictor, related or not. Nothing in the argument mentions whether the new variable has anything to do with the response. A column of random numbers gets a coefficient fitted to it, that coefficient will not be exactly zero, and the fit improves.

So R² is not measuring whether a model is right. It is measuring how much freedom the model was given, and freedom always helps.

How fast, in closed form #

The expected value has a clean form. With k predictors bearing no relationship to a response, on n observations:

$$E[R^2] = \frac{k}{n-1}$$

At n = 30 and k = 20 that is 20/29 = 0.690, and the measurement lands on it. At n = 200 the same twenty predictors give 20/199 = 0.101.

The same number of useless predictors, an R² of 0.69 in one case and 0.10 in the other. What determines the inflation is not how many variables were added but how many were added relative to how much data there is, which is the ratio that governs overfitting generally.

The closed form also gives the rule of thumb: each useless predictor buys about 1/(n − 1) of R². With thirty observations that is 3.4 percentage points per variable, free, for nothing.

What k/(n − 1) is worth at the sizes people work at #

The closed form is easiest to feel with its own spread attached, and the spread is available too: under a true null, R² follows a Beta(k/2, (n−1−k)/2), whose standard deviation is $\sqrt{2k(n-1-k)}\,/\,\left[(n-1)\sqrt{n+1}\right]$.

Twenty observations and five predictors, none of them related to anything. The expected R² is 5/19 = 0.263, with a standard deviation of 0.136. So a fit reporting R² = 0.26 is reporting the average of pure noise, and R² = 0.44 is one standard deviation above it.

A hundred observations and five predictors. The expected R² is 5/99 = 0.051 with a standard deviation of 0.031, so 0.11 is two standard deviations above nothing.

The contrast between those two rows is the whole practical content. At a hundred rows, an R² of a quarter is a real finding; at twenty rows it is the null’s own mean. The same number means opposite things at two sample sizes a reader will meet in the same afternoon, and nothing about the number says which one it came from.

That is also the cleanest way to state what adjusted R² buys. Under the null the expectation of $1 - R^2$ is exactly $(n-1-k)/(n-1)$, so the adjustment multiplies it back to one and the adjusted statistic has expectation exactly zero — at every n and every k, not approximately. What it does not change is the spread: at twenty rows and five predictors the adjusted statistic still has a standard deviation of 0.18 around that zero.

So adjusting fixes the centre and leaves the width. A model with an adjusted R² of 0.3 on twenty rows is less than two standard deviations from a model of pure noise, and the adjustment has made the number honest about its mean without making it informative about a single dataset.

The implementation detail that nearly hid it #

Worth recording, because getting this figure right required noticing that two plausible computations differ.

The obvious way to add k predictors one at a time is to regress the response on the first, take the residuals, regress those on the second, and so on. That is not multiple regression. Each new predictor is correlated with the ones already used, so it removes less than a full dimension, and the sequential route gives 0.50 where genuine multiple regression gives 0.69 at n = 30, k = 20.

The closed form describes the second. The first is a different quantity that happens to look like the same computation.

The fix is to orthogonalise each new predictor against the ones already in the model before projecting — Gram–Schmidt — at which point the measurement agrees with k/(n − 1) to three decimals, and the agreement is the check that the code computes the thing it is named for.

This is the two-routes discipline catching a real error rather than confirming a correct one. Had the closed form not been asserted beside the simulation, the figure would have shipped a plausible curve of the wrong quantity.

What adjusted R² does and does not fix #

The standard response is to use adjusted R², which penalises the number of predictors:

$$1 - (1 - R^2)\frac{n-1}{n-k-1}$$

It is calibrated exactly against the null case above: under pure noise the adjusted R² has expectation approximately zero, so the automatic inflation is removed. That is a real improvement and it is why the adjusted version should be preferred wherever the plain one is reported.

Three things it does not fix.

It still rises when a predictor helps slightly more than chance. The penalty is set so that a useless predictor is neutral in expectation, which means a predictor with any real association at all improves it. Selecting a model by maximising adjusted R² will still choose too many variables.

It is not a measure of fit either. A model can have a high adjusted R² and be the wrong shape entirely — the parabola in the quartet is nearly deterministic and the linear fit reports 0.67, and adjusting for one predictor barely moves that.

And it does not survive selection. If twenty candidate predictors are screened and the best five are kept, the adjusted R² of the final model is computed as though five variables had been chosen in advance. The correction accounts for the variables in the model, not the ones that were looked at — which is exactly the forking-paths problem arriving in a new setting.

The third is the one that matters most in practice and the one no adjustment to the formula can address, because the information required is not in the data.

What R² is actually good for #

The essay has been negative, so it is worth being precise about the legitimate uses, of which there are some.

Comparing models with the same number of parameters on the same data. The inflation is identical for both, so the comparison is fair. This is the case R² handles correctly and it is a common one.

Communicating how much variation a model accounts for, when the model is already known to be right. If the shape has been checked by other means, R² is a reasonable summary of how much scatter remains.

And as a rough scale-setter. An R² of 0.02 and an R² of 0.9 are describing very different situations, and no amount of the above changes that.

The uses to avoid are the ones where it functions as evidence: choosing between models with different numbers of predictors, arguing that a model is adequate, or reporting it as though a large value validated the specification. In each of those the number is being asked whether the model is right, and it does not contain that information.

The general shape of the failure #

R² belongs to a family, and recognising the family is more useful than remembering the individual case.

A criterion that improves whenever a model is given more freedom will select the freest model. In-sample error does this, likelihood does this, and R² does this. The response in every case is the same: penalise complexity, either explicitly with a term like the adjusted formula or AIC, or implicitly by measuring performance on data the model has not seen.

Out-of-sample measurement is the more reliable of the two, because it needs no assumption about the right size of the penalty. A model with twenty noise predictors on thirty observations has an R² of 0.69 in sample and predicts new observations no better than the mean — the gap between those two numbers is the whole story, and it appears without anyone having to derive a correction.

The reason it is not always used is that it costs data, and at n = 30 there is none to spare. Which is the uncomfortable summary: the situation where the inflation is worst is the situation where the best remedy is least affordable.

What to report instead #

Short, and consistent with the rest of this field.

Report adjusted R², not R², if either. The plain version has no defensible use that the adjusted one does not serve better.

Report the number of predictors considered, not only the number retained. This is the piece no formula recovers and the piece that determines how much the final figure is inflated.

Report an out-of-sample measure where the data allows one, and say when it does not.

And do not use any of them as evidence the model is the right shape. That question is answered by diagnostics aimed at specific failures — the residual plots of the previous essay, the leverage of the one before — and no summary statistic substitutes for them.

Why the name does the damage #

A note on vocabulary, because the misuse is downstream of the label rather than of the arithmetic.

R² is routinely glossed as “the proportion of variance explained”. Every word in that phrase is doing work it cannot support.

“Explained” implies a causal or structural relationship, and the quantity is a geometric projection. The twenty noise predictors above explain 69% of the variance in exactly the sense the phrase is used, and they explain nothing whatever in the sense a reader takes.

“Proportion” implies a share of a fixed total, which suggests that the unexplained part is irreducible noise. For the parabola in the quartet the unexplained third is not noise; it is the model being wrong.

“Variance” is the only accurate word, and it carries an assumption: that variance is what should be accounted for. For a skewed outcome or a count, variance is a poor summary of what varies, and a model can account for most of it while being useless for the cases anyone cares about.

The consequence is that a number computed as a ratio of sums of squares is read as a verdict on scientific adequacy, and the reading is not unreasonable given the words attached to it.

A more honest gloss would be: the fraction by which this model reduced the squared error, in the data it was fitted to. Both qualifications matter. It is squared error, so it is dominated by large residuals. And it is in-sample, so it is measured on the data the coefficients were chosen to accommodate.

What the numbers look like at the sizes people work at #

The closed form is easy to state and hard to feel, so a short table of the automatic R² from pure noise — what a model gets for free, before any real relationship is involved:

The top-left corner is unremarkable and the top-right is a catastrophe, and the difference is a ratio that anybody can compute before fitting anything.

The row worth staring at is the first. Thirty observations is a normal sample size in a great deal of applied work, and ten predictors is a modest model. A third of the variance, from nothing, and a reported R² of 0.45 in that setting means the model has found about a tenth of the variance beyond what noise supplies.

The rule that follows is the one every textbook states and few analyses observe: the number of predictors must be small relative to the sample size, and “small” means something like a tenth rather than a half.

The connection to the rest of the site #

This is the same failure as twenty analyses of nothing, wearing different clothes, and seeing that is worth more than either case alone.

There, twenty honest analyses of pure noise found something significant 57% of the time, because the reported quantity was the minimum of twenty p-values and its null distribution is not the one the threshold was chosen for.

Here, twenty useless predictors produce an R² of 0.69, because the reported quantity is the best fit achievable with twenty free parameters and its null distribution is not zero.

In both cases the arithmetic is correct at every step, nobody has done anything dishonest, and the reported number is compared with a reference computed for a procedure that was not followed. In both cases the fix has the same shape: work out what the statistic does under the null for the procedure actually used, and compare against that.

For R² the null distribution under the procedure actually used is available in closed form when the predictors are chosen in advance — it is the k/(n − 1) above. When the predictors are selected from a larger pool, it is not available in closed form and is considerably worse, for the same reason the minimum of twenty p-values is worse than one.

That case has no tidy correction and it is the one most analyses are in.

The most useful single habit #

If one thing survives from this essay, it is a comparison rather than a rule.

Before reporting an R², compute k/(n − 1) for the model. That is what the fit would have achieved from noise alone. Report the two side by side, or at minimum look at them side by side.

A model with an R² of 0.69 built from twenty predictors on thirty points has achieved exactly what nothing achieves. A model with an R² of 0.45 on two hundred points with five predictors has achieved 0.42 beyond chance, and that is a different result entirely — despite the smaller headline number.

The comparison takes one division and it reorders conclusions.

Where a high R² is genuinely bad news #

An inversion worth stating, because it runs against every intuition attached to the statistic.

In several common situations a very high R² is a warning rather than a success.

A predictor that contains the outcome. The commonest cause of an R² above 0.95 in applied work is that something on the right-hand side is a proxy for the thing on the left — a total that includes the component being predicted, a measurement taken after the outcome, a variable derived from it. The model is reporting an identity.

A time trend on both sides. Two series that both increase over time correlate strongly whatever their relationship, and a regression of one on the other reports a large R² and a small p-value. Nothing has been learned about either.

And a high-leverage design. The reversed slope from the previous essay comes with a respectable R², because a single distant point pulls the line toward itself and accounts for a large share of the total spread. The feature that makes the fit fragile is the same feature that makes the statistic look good.

So the number is not monotone in trustworthiness in either direction. Low can mean noisy data or a wrong model; high can mean a good fit, a tautology, a shared trend, or one dominant observation. A statistic compatible with that many situations is not evidence for any of them.

The summary #

R² rises whenever a model is given more freedom, by 1/(n − 1) per useless predictor, and the closed form matches the count to three decimals. Adjusted R² removes that specific inflation and does not make the statistic a measure of fit, does not survive variable selection, and does not detect a wrong shape.

What answers the question R² is asked is a check pointed at a particular way the model could be wrong, or a measurement on data the model has not seen. Both cost something. Neither is a number that can be read off the standard output, which is why the number that can be read off the standard output continues to be used for the job.