Twenty points and one more, at leverage 0.74

Without the distant point the slope is 0.495; with it the slope is -0.389. Its leverage is 0.737 and its Cook's distance is 24.1, against a conventional threshold of 1.

Twenty rows, 8 of them near x = 9 on a line with a reversed slope. The exact trimmed fit keeps the eleven rows whose own line has the smallest residual sum: 5 far rows and 6 clean ones, slope -0.629, residual sum 0.587. The best eleven rows whose line slopes the other way are all clean, slope 0.398, residual sum 0.657 — 1.12 times the winner's.

Median ratio of the best opposite-sign half's residual sum to the winning half's, over two hundred datasets at each count. Where the winner held: 16.32 at 1, 10.07 at 2, 8.16 at 3, 5.20 at 4, 3.56 at 5, 2.74 at 6, 1.96 at 7, 1.51 at 8, 1.45 at 9. Where it was reversed, at the counts with at least four such datasets: 1.17 at 4, 1.42 at 5, 1.60 at 6, 1.35 at 7, 1.63 at 8, 2.15 at 9. The flag at 1.70 is set so that five per cent of clean datasets are flagged.

Of two hundred datasets at each count, the datasets on which the exact trimmed fit — and so the MM-estimator carried on from it — is reversed, and how many of them the flag at a ratio of 1.70 catches; and the datasets on which it held, and how many of those the flag also raises. 3 far rows: 1 of 2 reversed flagged, 11 of 198 held flagged; 4 far rows: 4 of 4 reversed flagged, 11 of 196 held flagged; 5 far rows: 6 of 8 reversed flagged, 26 of 192 held flagged; 6 far rows: 10 of 18 reversed flagged, 47 of 182 held flagged; 7 far rows: 35 of 54 reversed flagged, 62 of 146 held flagged; 8 far rows: 61 of 111 reversed flagged, 54 of 89 held flagged; 9 far rows: 48 of 159 reversed flagged, 32 of 41 held flagged.

Each curve moves the threshold on the ratio from 1 to 4. At the threshold that raises 5% of clean datasets, 1.70, it catches 64.8% of the reversed fits at seven far rows, 55.0% at eight and 30.2% at nine. To catch 80% at nine it has to raise about 18% of clean datasets.

Datasets of two hundred on which the exact trimmed fit is reversed, keeping 11, 13 or 15 of twenty rows. 11 rows: 3, 1, 2, 2, 4, 8, 18, 54, 111, 159. 13 rows: 1, 0, 1, 0, 1, 7, 13, 56, 200, 200. 15 rows: 1, 0, 0, 0, 0, 4, 200, 200, 200, 200 — at 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 far rows.

One design of a hundred lognormal covariate values. The fitted value at a covariate value of 7.10 is a weighted sum of the outcomes, with weights fixed by the design. Under a line and its square it rests on 10.7 effective points, the largest carrying 19.1% of its variance. Under a natural spline with knots at 0.17, 0.57, 1.18, 3.95 it rests on 3.8, the largest carrying 41.1%.

Median over 400 designs of a hundred lognormal covariate values. A straight line: 88.2, 85.1, 13.7, 8.7, 7.5, 7.3. A line and its square: 86.5, 44.3, 27.5, 20.8, 10.2, 3.3. A natural spline, four knots: 51.3, 36.1, 28.4, 17.7, 4.8, 3.4 — at the 50th, 75th, 90th, 95th, 97.5th, 99th percentiles. At the 97.5th the spline's value rests on 4.8 points with the largest carrying 35.2% of its variance; the square's on 10.2.

Root mean squared error of the fitted value about the curve log x, with noise of standard deviation 0.5, over 400 designs of a hundred — exact given each design, as squared bias plus variance. A straight line: 0.278, 0.531, 0.501, 0.477, 0.819, 1.789. A line and its square: 0.285, 0.242, 0.274, 0.497, 0.807, 2.344. A natural spline, four knots: 0.104, 0.088, 0.127, 0.176, 0.257, 0.468. At the 97.5th percentile the square's value rests on 10.2 points and errs by 0.807; the spline's rests on 4.8 and errs by 0.257.

Median effective points behind the natural spline's fitted value at the 97.5th percentile of a lognormal covariate, 7.10, over 400 designs at each size, with the first three knots at the 5th, 35th and 65th percentiles and the last at the 80th, 90th, 95th, 99th. Last knot at the 80th: 5.1, 7.9, 12.8, 20.0. Last knot at the 90th: 5.0, 7.7, 12.7, 21.4. Last knot at the 95th: 5.0, 7.9, 13.7, 26.3. Last knot at the 99th: 7.3, 15.1, 40.9, 103.2 — at 100, 200, 400, 800 points.

Median effective number of points behind each coefficient over a thousand designs of a hundred, with the population share in brackets: treatment × a two-level subgroup, 94.6 (100.0); a normal covariate's own coefficient, 34.5 (33.3); treatment × a normal covariate, 34.9 (33.3); product of two normals, uncorrelated, 16.6 (11.1); product of two normals, correlated 0.9, 12.2 (6.7); square of a normal covariate, 12.2 (6.7); treatment × a lognormal covariate, 10.6 (0.88); square of a lognormal covariate, 6.9 (0.00).

Median effective share: a covariate's own coefficient 0.381, 0.357, 0.345, 0.342, 0.337, 0.334; the product of two uncorrelated normals 0.269, 0.203, 0.166, 0.143, 0.131, 0.120; the square of a normal 0.241, 0.162, 0.122, 0.100, 0.089, 0.074, at 25, 50, 100, 200, 400, 1600 points. The population shares are one third, one ninth and one fifteenth.

One design of a hundred normal points shifted to mean μ. The square's coefficient rests on 4.88 points at every μ, centred or not. The linear coefficient rests on 25.8 centred at every μ, and uncentred on 23.9, 9.7, 7.5, 6.8, 6.5, 6.2, 6.0 at μ = 0, 0.5, 1, 1.5, 2, 2.5, 3, where its variance inflation, one plus twice the square of μ, is 1, 1.5, 3, 5.5, 9, 13.5, 19.

Over a thousand designs of a hundred on two normal covariates, the point carrying the most of a covariate's own coefficient is inside both covariates' central 95% in 0.0%, 2.8%, 21.6%, 55.8%, 80.1% of designs at correlations 0, 0.25, 0.5, 0.75, 0.9; the point carrying the most of their product's coefficient, in 14.2%, 8.5%, 2.8%, 0.5%, 0.1%.

Without the distant point the slope is 0.495; with it the slope is -0.389. Its leverage is 0.737 and its Cook's distance is 24.1, against a conventional threshold of 1.

Leverage depends only on x and averages 0.095; Cook's distance combines leverage with the residual and says how far the whole fit moves if the point is dropped. One point sits at leverage 0.74 and Cook's distance 24.1.

Every one of these fits reports the same slope to two decimals and the same R². Only the first is a linear relationship with noise: the second is a curve, the third is a line with one outlier, and the fourth has its slope set by a single point.

Every panel is a correctly specified linear model with normal errors. The apparent curvature, funnelling and outliers are all produced by noise, and the largest single residual across the twenty is 2.13 standard deviations of the error. This is the reference nobody has when judging a real residual plot.

The response is pure noise and so is every predictor, so the true relationship is nothing at all. R² rises from 0.000 to 0.648 anyway, following k/(n − 1) — which is what a criterion that rewards higher R² is actually rewarding.

Twenty clean points and two rows near x = 9. The slope is −0.511 with every row, −0.376 with one far row deleted, and 0.495 with both deleted. Deleting one of them barely moves the line, because the other is still there.

Leverage needs only x; Cook's distance needs the fit without the row. The far rows sit at leverage 0.410 and 0.444, against 2p/n = 0.182, and at Cook's distance 0.966 and 0.772. The largest clean row reads 0.208.

One far row stays eight below the true line at x = 9 while a second, at x = 9.3, is lowered from the line to twelve below it. On the line, the second hides nothing and the first reads Cook's distance 6.57. Level with it, the two read 0.966 and 0.772. Lowered further, the second takes the flag and the first falls to 0.021.

With one far row its Cook's distance is 24.1. With two the largest single-row value is 0.966, with eight it is 0.096, under both 1 and 4/n. Deleting the whole cluster reads 53.3 at two rows and 301.5 at eight.

Deleting the two far rows together gives Cook's distance 53.3. The next pair reads 0.973 and holds one far row and one clean one; the largest pair of two clean rows reads 0.564. Only the first crosses 1.

Least squares gives slope −0.389, Huber 0.171 with the extra row at weight 0.115, and least trimmed squares 0.420, fitted to the 12 rows it keeps. The twenty clean rows alone give 0.495. Open circles are the rows the trimmed fit leaves out.

The extra row carries weight 0.115. The lowest weight among the twenty clean rows is 0.612, at x = 0.2; 1 clean row is below 1. The weight is a function of the residual on the fitted line and of nothing else.

The extra row sits on a line of slope −0.7111 through the true line's centre. At x = 9 least squares reads −0.389 and Huber 0.171; by x = 40 both read about −0.70, the wrong line's slope. The trimmed fit reads 0.420 throughout. Open circles mark distances where Huber, started from the trimmed fit, stops at a different line.

The median slope first falls below zero at 1 bad row for least squares, 2 for Huber and 8 for least trimmed squares, found exactly by visiting every subset of 11 rows. At half the rows bad, the trimmed fit is reversed on 100% of datasets.

On 100 datasets of 20 rows with 6 bad, the exact fit is known by visiting all 167,960 subsets of 11. From one start the search misses it on 68% of datasets; from 200 starts on 0%. The exact fit itself is reversed on 15% of them.

On clean normal data, Huber's efficiency reads 0.95, 0.95, 0.96 at n = 20, 80, 320, against a limit of 0.95. Least trimmed squares reads 0.239, 0.144, 0.086, falling towards its limit of 0.0713. One reweighting step reads 0.43, 0.68, 0.79. Bars are two batch standard errors.

Of 200 datasets at each count of far rows, the share on which the exact trimmed fit is reversed: 2.0% at 4, 4.0% at 5, 9.0% at 6, 27.0% at 7, 55.5% at 8, 79.5% at 9. The MM-estimator carried on from it repairs 1 of 4, 0 of 8, 0 of 18, 0 of 54, 0 of 111, 0 of 159 of those reversals, and spoils 0 of 196, 0 of 192, 0 of 182, 0 of 146, 0 of 89, 0 of 41 of the datasets where the trimmed fit held. The bad rows' average bisquare weight is 0.015, 0.039, 0.088, 0.263, 0.539, 0.771.

Every panel has the same slope of 1, the same intercept of 0 and the same residual standard deviation of 1.00. Only the range of x differs. R-squared runs from 0.021 to 0.849, and the estimated residual spread is 0.9932 in all five.

Each bar is the largest value divided by the smallest across five designs of the same relationship. R-squared spans a factor of 18.8 and the slope's standard error 16.0; the fitted slope spans 1.0006 and the residual spread 1.0000, both of which are 1 to within the counting.

Observations outside a window of the given half-width are discarded and the correlation refitted. At a window of ±0.25 the reported correlation is 0.107; at ±3 it is 0.595. The line is the closed-form correction and the points are counted over fitted samples.

The left study's points sit 0.50 from the line and the right study's 2.00 — a factor of 4.0. Both report an R-squared of 0.85 and, at the same sample size, the same standard error for the slope. The design was chosen to make it so, and it can always be chosen.

Each bar is the largest value minus the smallest across the four datasets. Pearson spans 0.0018, which is the construction working. Distance correlation — the measure that is zero if and only if the variables are independent — spans 0.101, and Spearman spans 0.491.

Every residual has standard deviation sigma times the square root of one minus its leverage. On this design the leverages run from 0.045 to 0.663, so the residual spreads differ by a factor of 1.68 — and the model is exactly right. The high-leverage point's residual averages 0.46 of the fitted spread where a typical point's averages 0.79.

The same twelve errors in every panel. With one run at the top (upper panels) a bad observation and a bent line produce identical data, point for point. With three runs at the top (lower panels) the bad observation leaves one run far from its two neighbours and the bent line moves all three together. The dashed line is the true line; the grey line is the least-squares fit.

At a discrepancy of four σ the replicated design names a bad run as one 84.1% of the time and a bend as one 91.6%; it calls a bend a bad run 5% of the time by construction and a bad run a bend 3.8%. The spread design flags a four-σ discrepancy 91.9% of the time whatever its cause, and can name the cause no better than a coin.

Moving six runs to the ends lowers the slope's standard error from 0.0836 to 0.0709 and the largest leverage from 0.295 to 0.235, raises the power against a bend of 0.05(x − 5.5)² from 44.7% to 66.5%, and makes a bad run nameable. It observes 8 distinct x values instead of 12, so a departure confined to x = 1, 2, 9 or 10 is invisible to it.

Largest leverage: 0.318, 0.318, 0.318, 1.000 against a threshold of 0.364. Largest Cook's distance: 0.489, 0.808, 1.393, undefined, the point fits itself against 1 and against 0.364. Largest studentised deleted residual: 2.1, 2.2, 1192.2, undefined against a Bonferroni threshold of 3.90.

On honest data Cook's distance over 4/n flags 67.2% of datasets, over 1 flags 5.9%, and the largest deleted residual against a Bonferroni threshold 5.3%. With one observation moved by five error standard deviations they flag 87.8%, 26.6% and 71.2%.

At a bend of 0.127 — the curvature of Anscombe's second dataset — Cook's distance over 4/n flags 90.9% of datasets, over 1 22.1%, and the deleted residual 5.1%, which is its false-alarm rate. At twice the bend the deleted residual fires on 2.2%: a bend raises every residual together and masks each point against the rest.

The effective number of points behind a slope is n divided by the design's kurtosis. Over designs of 100, its median share is two points, half at each: 1.000; uniform: 0.551; normal: 0.349; exponential: 0.160; Student's t, 3 df: 0.148; lognormal, σ = 1: 0.076 — 34.9 of 100 points for a normal covariate and 7.6 for a lognormal one.

Each point's share of the covariate's sum of squared deviations from its mean, which is its leverage less one over the number of points. The largest single share is 24.6%, so that point's leverage is 0.256; the design's effective number of points is 10.8 of 100.

Median largest leverage: normal 0.083, exponential 0.178, Student's t on three degrees of freedom 0.190, lognormal 0.300. A largest leverage above one half occurs in 0.0%, 0.5%, 6.7% and 15.7% of designs.

On a straight line with unit errors, at thirty points: normal 4.4%, exponential 18.1%, Student's t on three degrees of freedom 18.0%, lognormal 28.4%. At a hundred: 0.1%, 4.9%, 8.3% and 16.3%.

A hundred points on two normal covariates with correlation 0.9. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the first coefficient's information, 14.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 26.1 effective points, with a variance inflation of 6.27.

Median effective number of points behind the first coefficient over a thousand designs of a hundred at correlation 0: normal with 1, 34.9 (variance inflation 1.00); normal with 2, 35.0 (variance inflation 1.01); normal with 5, 35.4 (variance inflation 1.04); normal with 10, 35.0 (variance inflation 1.09); lognormal with 1, 7.7 (variance inflation 1.00); lognormal with 2, 7.7 (variance inflation 1.00); lognormal with 5, 8.1 (variance inflation 1.03); lognormal with 10, 9.1 (variance inflation 1.08).

The 95% level of the largest leverage over normal designs. At a hundred points: 1 covariate, 0.123; 2 covariates, 0.153; 3 covariates, 0.177; 5 covariates, 0.214; 7 covariates, 0.248; 10 covariates, 0.294. At fifty points the ten-covariate level is 0.511; at two hundred, 0.162. Designs with correlation 0.9 give the same levels exactly. The conventional rule 2(p + 1)/n sits at 0.04 to 0.22 for a hundred points.

Share of a thousand lognormal designs of a hundred whose largest leverage exceeds one half, at correlation 0: 1 covariate, 16.4%; 2 covariates, 31.0%; 3 covariates, 43.9%; 5 covariates, 64.2%; 7 covariates, 78.2%; 10 covariates, 90.8%. No normal design of a hundred reaches one half at any of these dimensions.

Over a thousand normal designs of a hundred, the share in which the point carrying most of the first coefficient's information is not among the five per cent most extreme values of any single covariate. With 2 covariates: 0.1% at 0, 2.6% at 0.25, 22.9% at 0.5, 53.2% at 0.75, 77.3% at 0.9. With 5 covariates: 1.3% at 0, 11.0% at 0.25, 35.9% at 0.5, 63.3% at 0.75, 79.1% at 0.9. With 10 covariates: 2.9% at 0, 13.7% at 0.25, 39.6% at 0.5, 64.0% at 0.75, 80.7% at 0.9.