Twenty points and one more, at leverage 0.74

Without the distant point the slope is 0.495; with it the slope is -0.389. Its leverage is 0.737 and its Cook's distance is 24.1, against a conventional threshold of 1.

Leverage depends only on x and averages 0.095; Cook's distance combines leverage with the residual and says how far the whole fit moves if the point is dropped. One point sits at leverage 0.74 and Cook's distance 24.1.

Every one of these fits reports the same slope to two decimals and the same R². Only the first is a linear relationship with noise: the second is a curve, the third is a line with one outlier, and the fourth has its slope set by a single point.

Every panel is a correctly specified linear model with normal errors. The apparent curvature, funnelling and outliers are all produced by noise, and the largest single residual across the twenty is 2.13 standard deviations of the error. This is the reference nobody has when judging a real residual plot.

The response is pure noise and so is every predictor, so the true relationship is nothing at all. R² rises from 0.000 to 0.648 anyway, following k/(n − 1) — which is what a criterion that rewards higher R² is actually rewarding.

Twenty clean points and two rows near x = 9. The slope is −0.511 with every row, −0.376 with one far row deleted, and 0.495 with both deleted. Deleting one of them barely moves the line, because the other is still there.

Leverage needs only x; Cook's distance needs the fit without the row. The far rows sit at leverage 0.410 and 0.444, against 2p/n = 0.182, and at Cook's distance 0.966 and 0.772. The largest clean row reads 0.208.

One far row stays eight below the true line at x = 9 while a second, at x = 9.3, is lowered from the line to twelve below it. On the line, the second hides nothing and the first reads Cook's distance 6.57. Level with it, the two read 0.966 and 0.772. Lowered further, the second takes the flag and the first falls to 0.021.

With one far row its Cook's distance is 24.1. With two the largest single-row value is 0.966, with eight it is 0.096, under both 1 and 4/n. Deleting the whole cluster reads 53.3 at two rows and 301.5 at eight.

Deleting the two far rows together gives Cook's distance 53.3. The next pair reads 0.973 and holds one far row and one clean one; the largest pair of two clean rows reads 0.564. Only the first crosses 1.

Least squares gives slope −0.389, Huber 0.171 with the extra row at weight 0.115, and least trimmed squares 0.420, fitted to the 12 rows it keeps. The twenty clean rows alone give 0.495. Open circles are the rows the trimmed fit leaves out.

The extra row carries weight 0.115. The lowest weight among the twenty clean rows is 0.612, at x = 0.2; 1 clean row is below 1. The weight is a function of the residual on the fitted line and of nothing else.

The extra row sits on a line of slope −0.7111 through the true line's centre. At x = 9 least squares reads −0.389 and Huber 0.171; by x = 40 both read about −0.70, the wrong line's slope. The trimmed fit reads 0.420 throughout. Open circles mark distances where Huber, started from the trimmed fit, stops at a different line.

The median slope first falls below zero at 1 bad row for least squares, 2 for Huber and 8 for least trimmed squares, found exactly by visiting every subset of 11 rows. At half the rows bad, the trimmed fit is reversed on 100% of datasets.

On 100 datasets of 20 rows with 6 bad, the exact fit is known by visiting all 167,960 subsets of 11. From one start the search misses it on 68% of datasets; from 200 starts on 0%. The exact fit itself is reversed on 15% of them.

On clean normal data, Huber's efficiency reads 0.95, 0.95, 0.96 at n = 20, 80, 320, against a limit of 0.95. Least trimmed squares reads 0.239, 0.144, 0.086, falling towards its limit of 0.0713. One reweighting step reads 0.43, 0.68, 0.79. Bars are two batch standard errors.

Of 200 datasets at each count of far rows, the share on which the exact trimmed fit is reversed: 2.0% at 4, 4.0% at 5, 9.0% at 6, 27.0% at 7, 55.5% at 8, 79.5% at 9. The MM-estimator carried on from it repairs 1 of 4, 0 of 8, 0 of 18, 0 of 54, 0 of 111, 0 of 159 of those reversals, and spoils 0 of 196, 0 of 192, 0 of 182, 0 of 146, 0 of 89, 0 of 41 of the datasets where the trimmed fit held. The bad rows' average bisquare weight is 0.015, 0.039, 0.088, 0.263, 0.539, 0.771.

Every panel has the same slope of 1, the same intercept of 0 and the same residual standard deviation of 1.00. Only the range of x differs. R-squared runs from 0.021 to 0.849, and the estimated residual spread is 0.9932 in all five.

Each bar is the largest value divided by the smallest across five designs of the same relationship. R-squared spans a factor of 18.8 and the slope's standard error 16.0; the fitted slope spans 1.0006 and the residual spread 1.0000, both of which are 1 to within the counting.

Observations outside a window of the given half-width are discarded and the correlation refitted. At a window of ±0.25 the reported correlation is 0.107; at ±3 it is 0.595. The line is the closed-form correction and the points are counted over fitted samples.

The left study's points sit 0.50 from the line and the right study's 2.00 — a factor of 4.0. Both report an R-squared of 0.85 and, at the same sample size, the same standard error for the slope. The design was chosen to make it so, and it can always be chosen.

Each bar is the largest value minus the smallest across the four datasets. Pearson spans 0.0018, which is the construction working. Distance correlation — the measure that is zero if and only if the variables are independent — spans 0.101, and Spearman spans 0.491.

Every residual has standard deviation sigma times the square root of one minus its leverage. On this design the leverages run from 0.045 to 0.663, so the residual spreads differ by a factor of 1.68 — and the model is exactly right. The high-leverage point's residual averages 0.46 of the fitted spread where a typical point's averages 0.79.