Two numbers for the fit's geometry

The summary that was meant to work ran distance correlation, the one dependence measure with a theorem behind it, on Anscombe’s four datasets and found that no summary of the dependence between x and y could flag the third — a line with one outlier — or the fourth, whose slope is set by a single point. Its closing argument was narrower than “no summary would work”. A summary of the fit, rather than of the relationship, is a different object: the largest leverage, the largest Cook’s distance, each a single number from the whole dataset that would flag two of the four. Whether one such number can flag both defects, and what it costs in false alarms on honest data, was left to be counted.

It takes two numbers, and they are not two of a kind.

The four datasets again #

Four datasets, one summary introduced them as a warning, and the warning’s force is that every number a regression usually reports is the same for all four. The third and fourth are the cases this essay is about. The third is a line with one point off it, which a person sees at once and a correlation coefficient averages away. The fourth is ten points at one value of x and one point at another, which a person also sees at once and which no summary of the dependence can distinguish from a line through a spread of x values — because the dependence is a line through two x values, one of them represented by a single observation.

The two defects are different in kind, and that is visible in the pictures before any number is computed. The third dataset’s problem is in its y values: one of them disagrees with the others. The fourth dataset’s problem is in its x values: the design gives one observation the whole job of fixing the slope. A summary that could flag both would have to read both the design and the data, and the measurements below say that the available ones read one or the other.

Why a maximum escapes the dilution #

The argument against summaries in the essay on distance correlation was arithmetic: a summary is a function of the whole dataset, and a defect that lives in one observation is diluted by a factor of $n$ when it is averaged in. Correlation, rank correlation and distance correlation are all averages over pairs of points, so one bad point among eleven moves them by a tenth of its effect.

A maximum over observations is not an average. The largest leverage in a dataset is exactly the leverage of its most extreme point; the largest Cook’s distance is exactly the influence of its most influential one. A defect in one observation is carried whole into the summary. That is the loophole the essay on distance correlation pointed at, and it is real — with the cost that a maximum over $n$ things is itself a search over $n$ things, and has the false-alarm rate of a search.

The design’s number and the data’s number #

The three candidates split cleanly into two kinds.

The largest leverage is $\max_i h_{ii}$ with $h_{ii} = 1/n + (x_i - \bar x)^2/S_{xx}$. It depends on the x values only. On Anscombe’s first three datasets, which share one design, it is 0.318 — the leverage of the end points of eleven evenly spaced values — and below the conventional threshold of $2p/n = 0.364$. On the fourth it is 1.000: one point stands alone at $x = 19$, the fit must pass through it, and it determines its own fitted value entirely. The largest leverage flags the fourth dataset and nothing else, and it does so before any $y$ is looked at.

The largest Cook’s distance and the largest deleted residual depend on $y$. Cook’s distance measures how far all the fitted values move when one point is dropped; the studentised deleted residual measures how far a point sits from the line fitted to the others, in units of the others’ spread. On the third dataset the largest Cook’s distance is 1.393 and the largest deleted residual is 1,192 — the remaining ten points lie almost exactly on a line, so the outlier is more than a thousand of their standard deviations away. On the first and second datasets the largest deleted residuals are 2.08 and 2.24, unremarkable.

On the fourth dataset both $y$-based quantities are undefined. A point of leverage one has a residual of exactly zero, and removing it leaves ten points all at $x = 8$, through which no line can be fitted. The quantities that compare a point with the others have nothing to compare it with.

So no single number here reaches both defects on its own terms. The leverage reaches the fourth because it reads the design; the deleted residual reaches the third because it reads the data; and each is silent, or undefined, on the other’s defect. The pattern is general rather than a property of these four: any quantity computed only from the x values is blind to a bad y, and any quantity that compares a point with a line fitted to the others is blind to a point the line cannot be fitted without. A reader who wants one number can have it by declaring “undefined” an alarm — which is honest, since an unverifiable point is exactly what the fourth dataset contains — but that is a rule about two numbers written as one.

Two diagnostics, as pictures #

The per-point versions of the two numbers are the diagnostics the essay on distance correlation pointed to, and the pictures show why they reach different things. Leverage is a function of position along the x axis alone: it rises with the square of the distance from the mean of x, and a point far out along x has high leverage whether its y lies on the line or off it. Influence is leverage multiplied by disagreement: a high-leverage point on the line has no influence, since removing it would not move the fit, and a low-leverage point off the line has little, since the other points hold the line in place.

The fourth dataset is the extreme of the first kind: leverage one, a point that is the line at its end. The line that one point drew measured what that does to a slope’s reliability; here it shows why influence cannot report it. Influence asks what happens if the point is removed, and with the point removed there is no line. Residuals are not the errors met the milder version, where a high-leverage point’s residual is shrunk towards zero by the fit it controls; at leverage one the shrinkage is complete.

What each rule costs on honest data #

A rule that flags a defect has to be priced by how often it flags honest data too. Here the honest data are straight lines with normal errors on Anscombe’s first design, eleven points, with the error spread of his first dataset.

The rule of thumb most often printed — flag any point with Cook’s distance above $4/n$, here 0.364 — flags two thirds of honest datasets. On Anscombe’s quartet it flags the first, straight, dataset along with the third, which is the rule working as it always does rather than failing on an unusual case. It is a threshold written for a single point and applied to the largest of eleven, and the largest of eleven Cook’s distances from an honest line is usually above it.

The threshold of one flags 5.83% of honest datasets, close to a 5% test, and catches a point moved by five standard deviations only a quarter of the time. Cook’s distance weighs a residual by its point’s leverage, so a bad point in the middle of the design, where the leverage is low, barely moves it.

The deleted residual against a Bonferroni threshold — the largest of eleven compared with the $t$ quantile at $0.05/22$ — is built as a 5% test for the largest of eleven, and holds it: 4.78% of honest datasets. It catches a point moved by five standard deviations 71.08% of the time and one moved by eight almost always. Of the three $y$-based numbers it is the only one that is both a calibrated test and a useful one.

The leverage has no false-alarm rate in this sense. On a given design it is either above its threshold or not, for every dataset: on Anscombe’s first design, never; on his fourth, always. It is a statement about the design, and a design with a point of very high leverage is one whose line depends on that point whatever the data turn out to be.

The search inside a maximum #

The deleted-residual rule works because it prices the search a maximum performs, and the price changes with the size of the dataset in a way the popular threshold does not track.

The largest of $n$ deleted residuals is compared with the $t$ quantile at $0.05/2n$ — the Bonferroni correction for $n$ looks, the same correction twenty analyses of nothing needed for twenty analyses of one dataset. At eleven points the threshold is 3.90; at thirty, 3.49; at a hundred, 3.60; at a thousand, 4.07. It falls at first as the $t$ distribution’s degrees of freedom grow, then rises as the number of looks does, and at every size it holds the rate at which an honest dataset is flagged near 5%. The $4/n$ threshold for Cook’s distance falls in proportion to $n$, from 0.364 at eleven points to 0.004 at a thousand, which is the rate a single point’s Cook’s distance falls at and not the rate the largest of $n$ does.

The correlation between neighbouring residuals makes Bonferroni slightly conservative, as the count of how many analyses there really were found for correlated tests; here the residuals of eleven points share only two fitted parameters and the correlation is weak, which is why the measured rate, 4.78%, sits just under the nominal 5%.

What no point rule sees #

The second of Anscombe’s datasets is a smooth curve, and none of the three numbers flagged it. That is not a threshold problem.

With the curvature of Anscombe’s second dataset and honest noise added, the deleted residual rule flags 5.08% of datasets — its false-alarm rate. At twice the bend it flags 2.20%, less than on a straight line. A bend moves every residual together, raising the spread of the others as it raises each point’s residual, so no single point stands out against the rest; the stronger the bend, the more thoroughly each point is masked by all the others. Cook’s distance over one does better, 22.09% at the second dataset’s curvature, because the bend puts the end points, which have the most leverage, furthest from the line.

Cook’s distance over $4/n$ flags 90.94% of the bent datasets. It is not seeing the bend; it flags 66.45% of straight ones.

A bend is a shape, and a shape is a pattern across points rather than a property of any one of them. The instrument for it is a lack-of-fit test, a residual plot, or twenty residual plots set side by side — which is where the conclusion that a plot remains the first diagnostic comes back, now with a count attached.

What the quartet asks of a report #

The quartet was built to show that the numbers a regression usually reports are the same for four different situations. The measurements here say what the report would need to add for its numbers to tell them apart.

The largest leverage, which costs nothing — it needs only the x values and could be printed with the design. It separates the fourth dataset from the others with certainty, and it says, for any dataset, whether its line rests on one point.

The largest studentised deleted residual with its Bonferroni p-value. It separates the third dataset at a level no honest dataset approaches, and it is a calibrated 5% test on the others.

A test or a plot of the shape. Neither number sees the second dataset, and no maximum over single points can, because the defect belongs to no single point.

That is three items for four datasets — and the t statistic wearing different clothes is a reminder that the usual report’s two headline numbers, $R^2$ and the slope’s p-value, are one number twice. A report that exchanged one of them for the largest leverage and the other for the largest deleted residual would carry more information in the same space.

Why the popular threshold survives #

The $4/n$ threshold for Cook’s distance is printed in textbooks and applied by software, and on these measurements it flags two thirds of perfectly good datasets. It survives because it is almost never used as a test. It is used as a prompt to look — flag the points above it, then decide — and a prompt that fires on most datasets is at least a prompt that fires on the bad ones.

Used that way it is harmless. Read as a finding it is not: a paper that reports “no influential points (all Cook’s distances below 4/n)” has reported something true of only a third of honest datasets of this size, and a paper that removes the points above it has removed data from two datasets in three with nothing wrong in them. The threshold is a fact about one point’s Cook’s distance under the null; the largest of $n$ Cook’s distances is a different statistic, and it needs a different threshold, which is what the Bonferroni deleted-residual rule supplies.

What the design’s number and the data’s number each reach #

The largest leverage is 0.318 on Anscombe’s first three datasets and 1.000 on the fourth, which it alone of the three numbers flags; it depends on the design only.

The largest studentised deleted residual is 1,192 on the third dataset and against its Bonferroni threshold flags 4.78% of honest datasets of the same design and 71.08% of those with one point moved by five standard deviations.

Cook’s distance over 4/n flags 66.45% of honest datasets; over 1, 5.83%, and neither of the $y$-based rules sees a bend: the deleted residual rule fires on 5.08% of datasets with the second dataset’s curvature and 2.20% with twice it.

Every rate is a count over twenty thousand seeded honest datasets, or eight thousand at each shift and each bend, with errors of standard deviation 1.24 on Anscombe’s first design; the quartet’s own values are computed from his published numbers.

Not claimed: that eleven points are typical, or that the moved point’s position is. The moved point is placed at random among the eleven, so the power is an average over positions, and a bad point at the end of the design is caught more often than one in the middle. Not claimed either that the Bonferroni rule is the best test for one outlier; it is the simplest calibrated one, and two points can hide each other from it.

Still open: the design a report should have to state #

The largest leverage is the one number here that needs no data, and the one that most changes how the rest should be read: a line whose largest leverage is near one rests on a single observation, and no residual can check it. It could be stated at the design stage, before any outcome is measured, as easily as the sample size.

What has not been measured is how often published regressions carry a leverage that high — how many reported slopes are, in effect, the line that one point drew — and whether a threshold on the largest leverage, calibrated like the Bonferroni rule to a stated false-alarm rate over designs rather than over data, could be defined sensibly at all, given that the leverage has no sampling distribution once the design is fixed.