Four datasets, one summary

Four datasets. Every one has a fitted slope of 0.50, an R² of 0.67, and the same intercept to two decimals. Only the first is a linear relationship with noise.

What is being claimed, and how it is checked #

Anscombe’s quartet is usually presented as a picture with the agreement asserted in the caption. On this site the agreement is a check: the four slopes are computed and required to fall within 0.01 of one another, and the four R² values likewise. The measured slopes are 0.5001, 0.5000, 0.4983 and 0.4999; the R² values are 0.6665, 0.6662, 0.6692 and 0.6667.

That matters because the whole force of the example rests on the agreement being exact rather than approximate. If the summaries merely resembled each other, a reader could reasonably conclude that a sharper eye on the numbers would have caught the difference. They do not merely resemble each other, and no amount of attention to the summary would separate them.

What each one actually is #

The first is what the summary describes: a linear relationship with roughly constant scatter. Everything the fit reports is true of it.

The second is a clean parabola with almost no noise. The relationship is deterministic and perfectly predictable, and the linear summary describes it as a moderately noisy straight line. The R² of 0.67 is reporting the failure of the model, not the presence of noise — there is essentially no noise.

The third is an exact straight line with one point displaced. The true relationship is deterministic with a single anomaly, and the fitted slope is pulled off it by the outlier. Removing one point would give an R² of 1.

The fourth is eleven points at a single x value and one point elsewhere. There is no information about the slope in the eleven — they are all at the same x — so the entire slope is determined by one observation, which is the leverage case in its purest form.

Four completely different situations, four different correct analyses, one summary.

How much tighter than the gate the agreement is #

The check requires the four slopes to fall within 0.01 of each other, and it is worth reading what the data actually delivers against that tolerance.

The slopes span 0.4983 to 0.5001, a range of 0.0018. The R² values span 0.6662 to 0.6692, a range of 0.0030. So the gate allows a hundredth and the quartet uses a fifth and a third of it.

More useful is the range set against the noise in any one of the four. With eleven points and an R² of 0.667, a slope’s own standard error is

$$\frac{b}{t}, \qquad t = \sqrt{\frac{R^2(n-2)}{1-R^2}} = 4.25,$$

so $\text{se}(b) = 0.5/4.25$, which is 0.118.

The four slopes differ from one another by 0.0018 against a standard error of 0.118 — about one and a half per cent of a single standard error.

That is the number that makes the example inescapable. The four datasets do not merely produce similar slopes; they produce slopes sixty-five times closer together than the uncertainty attached to any one of them. No refinement of the summary, no extra decimal place, no attention to the standard errors would separate them, because the separation is far inside what the summaries themselves can resolve.

Why no summary would work #

The natural response is that better summary statistics would separate them, and it is worth explaining why that response does not survive.

A summary is a function of the data that returns fewer numbers than went in. Any such function maps many datasets to the same output — that is what “fewer numbers” means. So for any summary, however clever, there exist datasets it cannot distinguish, and the only question is whether those datasets are ones that arise.

Anscombe’s construction shows they are. And the construction generalises: given any fixed set of summary statistics, datasets can be built to match on all of them and differ in shape, and modern versions do this by optimisation, producing datasets with identical means, variances and correlations that draw pictures of animals.

So the argument is not that the standard summary is badly chosen. It is that the exercise has no good answer, because the information being asked for is not in the numbers being reported.

What the residual plot adds #

The residual plots separate the four immediately, and it is worth being precise about why, because “look at the data” is advice rather than a method.

A residual plot shows what the model failed to explain, arranged against something the model used. If the model is right, that leftover is noise and has no relationship to anything — so any structure in the plot is a statement that the model is missing something specific.

The second dataset’s residuals change sign exactly twice across the range: negative at the ends, positive in the middle. That is a curve, and it is the signature of fitting a straight line to one. The site’s gate asserts this rather than describing it: the sign changes are counted, and the check requires the curved dataset to have at most two.

The third dataset’s residuals are near zero everywhere except at one point, which is the signature of an exact relationship with an anomaly. The fourth’s are concentrated at a single x, which is the signature of a design with no leverage anywhere else.

Each shape corresponds to a different failure and suggests a different response — fit a curve, investigate the anomaly, redesign the study — and none of them is visible in the summary.

The reason this is not a solved problem #

Anscombe published in 1973 and the example is in every textbook, so the natural question is why the failure it describes is still routine.

Three reasons, and only the first is about ignorance.

A summary travels and a plot does not. A slope and a p-value can go into a table, an abstract, a meta-analysis and a database. A residual plot goes in an appendix if anywhere, and it cannot be aggregated across studies. The parts of an analysis that get reused are exactly the parts that carry least information about whether the model was right.

The number of plots scales badly. One predictor means one residual plot. Twenty predictors means twenty of them plus interactions, and nobody looks at four hundred plots. The advice that works for the quartet does not scale to the models people actually fit, which is the honest reason it is skipped.

And judging a plot by eye is uncalibrated. Even for someone who looks, the question “is this structure real?” has no answer without knowing how much structure noise produces — which is the problem the next essay is about, and it is the one that makes looking harder than it appears.

The third is the interesting one, because it means the standard advice is incomplete rather than merely ignored. “Plot the residuals” is not actionable without a reference for what a correct model’s residuals look like at that sample size, and that reference is almost never supplied.

What the quartet does not show #

Two limits worth stating, since the example is often asked to carry more than it can.

It does not show that regression is unreliable. Three of the four datasets are pathological by construction, assembled to defeat the summary. Real data is not usually adversarial, and the first dataset — the one the summary describes correctly — is the common case.

And it does not show that plots are sufficient. Looking at these four separates them because they are two-dimensional and eleven points long. A model with twenty predictors has no picture that shows its shape, and the diagnostics available there are themselves numerical summaries with the same fundamental limitation.

The claim the quartet actually supports is narrower and still strong: agreement between summaries is not evidence of similarity between datasets, so a summary alone cannot justify a model. What justifies a model is a check aimed at a specific way it could be wrong, and the residual plot is one such check rather than a general safeguard.

The habit that follows #

The quartet argues for a small change in what an analysis produces, and it is the same change the rest of this field keeps asking for.

Report the diagnostic alongside the estimate, not in place of it and not in an appendix. The residual plot, the maximum leverage, the largest Cook’s distance: three items, all cheap, and between them they distinguish all four of these datasets.

Say what the diagnostic was checking for. “Residuals were inspected” is unfalsifiable. “Residuals showed no sign change across the range of the predictor” is a claim, and a reader can ask to see it.

And treat a summary as a claim about a shape rather than a description of one. A slope of 0.50 is a statement that the relationship is linear with that gradient. The statement can be checked, the check is a plot, and the quartet is the demonstration that the statement is not self-verifying.

Why R² is the most misleading number in the set #

The three summary statistics fail differently, and R² fails worst, because it is the one read as a verdict on the model.

An R² of 0.67 is normally taken to mean that the model captures about two thirds of what is going on and that the remaining third is noise. Look at what it means for each of these four.

For the first dataset the reading is correct: about a third of the variation really is noise.

For the second the reading is wrong in an interesting direction. The parabola is nearly deterministic — there is almost nothing unexplained in the data, only something unexplained by a straight line. The R² of 0.67 is measuring the model’s inadequacy and presenting it as the world’s variability.

For the third the same, localised: the relationship is exact and one point is displaced, so the missing third of the variance is one observation rather than pervasive scatter.

For the fourth the number is close to meaningless, because with eleven points at one x value the notion of variance explained by a slope barely applies.

The same 0.67, four different meanings, only one of them the one it will be given. That is the general case rather than a quirk of these datasets, and it has its own essay because the failure runs deeper than not distinguishing shapes.

The construction, and what it takes to build one #

Worth knowing how these datasets were made, because it says something about how much room the summary leaves.

Anscombe built them by hand in 1973, which is why the numbers are awkward and the agreement is to two decimals rather than exact. The modern version treats it as optimisation: fix the target summary statistics, start from a dataset that has them, and move points in small steps that preserve the statistics while pulling the cloud toward a target shape. Run long enough and the data draws a dinosaur while its mean, variance and correlation stay put to several decimals.

That construction settles the question of how much freedom the summary leaves, and the answer is: essentially all of it. Fixing a handful of numbers constrains a dataset of eleven points hardly at all, and the set of datasets consistent with any given summary contains almost anything a person might want to draw.

The instructive part is that this is not a statement about statistics being weak. It is arithmetic. Eleven points are twenty-two numbers; a summary is four or five. Seventeen dimensions of freedom remain, and seventeen dimensions is a great deal of room.

What survives when there is no picture #

The uncomfortable case, and the one that matters for real work: the model has too many predictors to plot, so what replaces looking?

Three things, and they are less satisfying than a scatter plot.

Numerical diagnostics that target specific failures. Leverage and Cook’s distance for influence, a test for curvature by adding a squared term and seeing whether it helps, a check on the residuals against each predictor in turn. Each is aimed at one way the model could be wrong, and the coverage is only as good as the list.

Out-of-sample prediction. A model that is wrong in shape usually predicts badly on data it has not seen, and this catches failures no in-sample diagnostic will. It is the most reliable general check available and it costs data.

And plots of low-dimensional projections, chosen deliberately rather than exhaustively — residuals against the fitted values, and against any predictor whose functional form is in doubt. Not a solution, and it recovers most of the value for a fraction of the four hundred plots.

The honest summary is that the quartet’s lesson does not fully generalise. In two dimensions, looking solves the problem completely. In twenty, nothing solves it completely, and the practical response is a short list of targeted checks plus out-of-sample validation — which is a weaker guarantee than the quartet implies is available.

The one-sentence version #

A summary is a claim that the data has a certain shape. It is not evidence for that claim, because summaries of very different shapes coincide — and the four datasets here coincide to two decimal places on every statistic a regression normally reports.

What makes a model believable is a check pointed at a specific way it could be wrong. The quartet is worth its place in every textbook because it shows, in one picture, that the alternative is not an option.

What this argues about reporting formats #

A structural observation to close on, since the failure has outlived fifty years of the example being taught.

The formats that carry results between people — the abstract, the results table, the meta-analytic database, the model card — are all summaries. They exist because summaries travel and pictures do not, and that is a real constraint rather than a bad habit.

The consequence is that the information distinguishing these four datasets is systematically lost at every step where a result moves from the person who fitted the model to anyone else. The analyst may have looked at the residuals; the reader of the abstract cannot, and the meta-analysis that pools the slope has no way to know which of the four situations produced it.

That suggests the useful demand is not “look at the residuals” — the analyst probably did — but carry the diagnostic into the summary. A slope reported with its maximum leverage and largest Cook’s distance is three numbers instead of one, all of them travel, and they separate the fourth dataset from the first immediately.

It is a small change and it addresses the part of the problem that repetition of Anscombe’s picture has demonstrably failed to fix.

Where the quartet sits in this field #

The three essays around it divide the problem cleanly, and it is worth saying how.

The leverage essay takes the fourth dataset’s failure and gives it a number: leverage is computable from the design before any outcome is measured, so the fragility is predictable rather than discovered.

The R² essay takes the second dataset’s failure and generalises it: the statistic read as goodness of fit rises when pure noise is added as a predictor, so it is not measuring what its name suggests.

And the residual line-up takes the response this essay recommends — plot the residuals — and asks the question that makes it usable: how much structure a correct model produces, so that a reader knows what they are comparing against.

Together they say the same thing three times with different machinery. The summary is a claim, the claim needs a check, and the check needs a calibration. Anscombe supplies the first step and is usually taught as though it supplied all three.

That is the reading the quartet rewards on a second encounter, and it is why it belongs at the front of a field rather than in a box in a first chapter.

The example rewards being met twice, and the second meeting is the useful one.