Normal Distribution
nothing is called 95% until it has been counted
Concept

Outliers — where it appears

Observations far from the rest, which may be errors, may be the interesting part, and are in either case what a squared-error criterion is most sensitive to. Removing them changes the estimate and removing them because they changed the estimate changes the procedure.

Named by 7 essays across 3 fields — each of them below, with the objects they name alongside it.

Twenty points and one more, at leverage 0.74. Without the distant point the slope is 0.495; with it the slope is -0.389. Its leverage is 0.737 and its Cook's distance is 24.1, against a conventional threshold of 1.

The line that one point drew

A single observation among twenty-one reverses the sign of a fitted relationship. Its leverage is known from its x value before the outcome is looked at, so this is a property of the design rather than a surprise in the data.

regression · Leverage One group 6 population widths from the rest. Squared error for each group under partial pooling, with each group's own mean beside it. Seven of the eight are estimated better by pooling. The eighth, which was never from the population, is estimated 6.1 times worse — 13.9 against 2.3.

When borrowing goes wrong

Partial pooling wins on the total and can lose badly on one group. Placed six population widths out, the group that was never from the population is estimated six times worse than by its own mean — and nothing in the output says so.

hierarchical · Pooling What each correction is worth, exactly. Each variance estimate's expectation under a constant error variance, divided by the variance the slope actually has, at 20 rows on an even design, by two routes: the closed form E[eᵢ²] = σ²(1 − hᵢᵢ) carried through each correction's own weight, and the mean of 20000 counted estimates. The maximum leverage here is 0.1857 and the design's fourth-moment share Σu⁴/(Σu²)² is 0.0897, which is the only thing the closed form reads. HC0 comes out at 0.8603 — short by construction, since its factor is exactly 1 − 1/n − Σu⁴/(Σu²)². HC1 reaches 0.9559, HC2 is exactly 1.0000 at every design and every sample size, and HC3 overshoots to 1.1647. On an even design the four are within a fifth of each other and the choice barely matters.

Three corrections and a leverage

On an even design of twenty rows the four robust corrections read 0.8603, 0.9559, 1.0000 and 1.1647 of the truth and the choice barely matters. Add one point at x = 8 and they read 0.3191, 0.3419, 1.0000 and 5.1127.

sandwich · Misspecification Two far rows, and the line with one of them deleted. Twenty clean points and two rows near x = 9. The slope is −0.511 with every row, −0.376 with one far row deleted, and 0.495 with both deleted. Deleting one of them barely moves the line, because the other is still there.

Two points that hide each other

One far observation among twenty-one has a Cook's distance of 24.1. Put a second beside it and the two read 0.966 and 0.772, neither crossing 1, while together they reverse the slope and deleting both moves the fit by 53.3.

regression · Leverage One row at x = 9 on a wrong line, fitted three ways. Least squares gives slope −0.389, Huber 0.171 with the extra row at weight 0.115, and least trimmed squares 0.420, fitted to the 12 rows it keeps. The twenty clean rows alone give 0.495. Open circles are the rows the trimmed fit leaves out.

A robust loss and a far x

One far row drags least squares to a slope of −0.389. Huber's loss, the standard robust line, reaches only 0.171, and carried further out the same row gets its full weight back. Least trimmed squares reads 0.420 at every distance, and at the normal model keeps 7.13% of least squares' efficiency to do it.

regression · Leverage One row at x = 9 on a wrong line, fitted three ways. Least squares gives slope −0.389, Huber 0.171 with the extra row at weight 0.115, and least trimmed squares 0.420, fitted to the 12 rows it keeps. The MM-estimator carried on from the trimmed fit gives 0.479, with the extra row at weight 0.000 and the scale fixed at 0.319. The twenty clean rows alone give 0.495. Open circles are the rows the trimmed fit leaves out.

The start an efficient robust line inherits

The MM-estimator carries a trimmed fit on through a redescending loss, and it does what it promises on one far row: slope 0.479 at every distance, the row at weight exactly zero, and 87.2% of least squares' efficiency at twenty rows. What it cannot do is choose. At eight far rows of twenty the exact trimmed fit picks the wrong half on 111 datasets; the efficient step repairs none of them, spoils none of the other 89, and ends nearer the wrong line than the start did.

regression · Leverage Two halves of one dataset with 8 far rows of twenty: the trimmed fit's winner and the best half with the other sign. Twenty rows, 8 of them near x = 9 on a line with a reversed slope. The exact trimmed fit keeps the eleven rows whose own line has the smallest residual sum: 5 far rows and 6 clean ones, slope -0.629, residual sum 0.587. The best eleven rows whose line slopes the other way are all clean, slope 0.398, residual sum 0.657 — 1.12 times the winner's.

The other half a trimmed fit rejected

The exact trimmed fit visits every half of the data, so it can report more than its winner: the best half whose line slopes the other way. Whenever the winner is reversed by a far cluster, that other half is the clean line. Printed beside the winner when its residual sum is within 1.70 times — a rule that raises five per cent of clean datasets — it flags 65% of the reversed fits at seven far rows of twenty, 55% at eight and 30% at nine, and no rule built on fit can do better at nine, where the far cluster has become the tighter half by every measure the data hold.

regression · Leverage

Named alongside it

The objects these essays reach for when they reach for this one.

LeverageClosed formLeast squaresMaskingBreakdown pointLeast trimmed squaresLocal minimumM estimatorRobust regressionCook's distanceEfficiencyHat matrix

All concepts