A robust loss and a far x

The line that one point drew listed four things to do about an influential observation, and the third was to use a method less sensitive to it: robust regression, it said, down-weights points with large residuals and will resist this one. The sentence is a prediction, it can be tested on the essay’s own data, and it survives only in part.

Fit the same twenty-one rows three ways. Least squares, as before, gives a slope of −0.389 where the twenty clean rows alone give 0.495. Huber’s M-estimator — quadratic loss for small residuals, linear beyond 1.345 scales, with the scale re-estimated from the median absolute residual at every step — gives 0.171. It resists: the slope is positive again. But it recovers only 63% of the distance back to the clean answer. And least trimmed squares, the line that best fits the twelve rows that fit a line best, found exactly by checking all 293,930 ways of choosing those twelve, gives 0.420, with the far row among the nine it leaves out.

The gap between the second and third numbers is the subject of this essay, and it is not a matter of tuning. Huber’s loss limits how hard a large residual can pull on the line. A far observation does not need a large residual to decide the line; it needs a large x, and the loss has no term that reads x at all. The trimmed fit limits something else — how much of the data the line has to pass near — and that turns out to be the limit that matters against leverage, at a price in precision that is equally measurable.

What a bounded loss bounds #

Every M-estimator of a straight line solves the same pair of equations,

$$\sum_i \psi\!\left(\frac{r_i}{s}\right) = 0, \qquad \sum_i \psi\!\left(\frac{r_i}{s}\right) x_i = 0,$$

where $r_i$ is the i-th residual, $s$ a scale and $\psi$ the derivative of the loss. Least squares takes $\psi(u) = u$, so a residual’s pull grows without limit. Huber’s loss takes $\psi_c(u) = \max(-c, \min(c, u))$: identical to least squares within c scales of the line, and constant beyond. That is the whole of the robustness, and it is a genuine guarantee — a residual, however large, can contribute at most $c$ to either sum.

Read the second equation again. A row contributes $\psi(r_i/s)\,x_i$ to it, and the loss caps only the first factor. A row at x = 9, when the rest of the data sits between 0.2 and 3.4, has its capped residual multiplied by a lever several times longer than anyone else’s. The loss bounds how hard the row can push; it does nothing about where the row is pushing from.

In practice the equations are solved by iteratively reweighted least squares: each row gets the weight $\psi(u)/u = \min(1, c/|u|)$ at the current line, least squares is rerun with those weights, and the process repeats until nothing moves. The weights at convergence show exactly what the loss decided.

The far row ends at weight 0.115. On Huber’s own converged line it sits 6.46 below, which is 11.7 scales, so the loss has indeed capped its residual. And the capped residual still reaches the slope equation through a lever of length 9 − 1.8, where an average clean row’s lever is about a twelfth of that; the row still holds leverage 0.737, and weight times lever is enough to keep the line tilted to 0.171.

One clean row is down-weighted as well: the observation at x = 0.2, at weight 0.612 — a residual of the size a correct model routinely produces, discounted because a wrong row elsewhere moved the line. That is the same geometry two far rows forced onto a clean one. The far row pulls the right-hand end of the line down, the left-hand end rises, and the leftmost honest observation acquires a residual large enough for the loss to start discounting it. A robust method applied to data with one far error spends part of its protection against a row that is right.

The row the loss was built for #

Move the same bad observation inside the data — to x = 3, eight below the true line — and the loss does exactly what it promises.

Least squares is dragged from 0.495 to 0.030; a row with ordinary leverage and a huge residual still moves a squared-error fit a long way. Huber’s estimate is 0.443, with the row at weight 0.058, and the trimmed fit again reads 0.420. The row’s leverage here is 0.112, barely above the average for twenty-one rows, so its capped residual is multiplied by an ordinary lever and has an ordinary effect, which is to say very little.

Lowering the row further shows what “bounded” means in its exact form.

Least squares falls by 0.0565 for every unit the row is lowered, without limit — −1.777 at forty below. Huber’s slope is 0.462 with the row on the line and 0.443 from two below onwards, and from two below to forty below it does not change in the twelfth decimal place. That is not approximately flat. Once the row’s residual is beyond $c\,s$, its $\psi$ is the constant $-c$, so lowering it further changes nothing in either equation; and the median absolute residual does not notice either, because a row that is already in the upper half of the residuals can move anywhere within that half without moving the median. The influence of a large residual under Huber’s loss is not small. It is fixed, and a fixed influence at an ordinary x is a small one.

Carried out along a wrong line #

Now take the far row of the leverage essay and move it along a line: the line through the true line’s centre, at x = 1.8, and through the point the essay placed, (9, −4.4). That line has slope −0.7111. At x = 9 the row is exactly the leverage essay’s observation; further out it is the same kind of error, one row that agrees with a different relationship, placed at longer and longer range — the situation in which what a straight line estimates is decided by where the design puts its rows rather than by the relationship the rest of the data follow.

Least squares follows the row from the start: 0.349 at x = 3.4, −0.389 at 9, −0.698 at 40, where it is within 0.02 of the wrong line’s slope. Huber’s estimate resists for a while, declining gradually to 0.1255 at x = 10. Between 10.5 and 13 it collapses, and from x = 13 outward the far row has weight exactly 1 — Huber’s estimate is then least squares with a handful of clean rows discounted, reading −0.565 at 13 where least squares reads −0.556, and −0.700 at 40.

The mechanism is not that the row became too extreme for the loss. It is that the row became far enough out to pull the line onto itself. At x = 13 its leverage is 0.870; the line passes close to it, so its residual on Huber’s line is only −1.70. Meanwhile the tilted line has pushed the clean rows’ residuals up, and the median absolute residual — the scale — has grown from 0.552 to 1.29. The row’s residual is now inside $c\,s$, the loss treats it as an ordinary observation, and the full weight comes back at the moment the row is doing the most damage.

Because a row can be carried out along any wrong line, the slope Huber’s estimate reports can be dragged to any value by a single observation. That is the formal statement that an M-estimator in regression has a breakdown point of one row in n against leverage, and the figure is that statement drawn.

The open circles record something less comfortable. At x = 10.5, 12 and 12.5 the same data give Huber’s iteration two different answers depending on where it starts: 0.0398 or 0.1025 at 10.5, and −0.3226 or −0.1561 at 12. With the scale held fixed, Huber’s loss is convex and its minimum is unique; re-estimating the scale at every step, which is what makes the method usable when the error variance is unknown, gives up that uniqueness. In that band the robust line is a function of the data and of the starting point, which a published robust fit rarely reports.

The trimmed fit reads 0.420 at every distance in the figure. The far row is never among the twelve rows it uses, so where the row sits is irrelevant to it.

Breakdown, counted #

One row, placed deliberately, shows a mechanism. The rate at which each estimator gives way needs many datasets and a count of bad rows, and here both kinds of bad row are counted. Twenty rows; a number m of them replaced by bad ones; 200 datasets at each count; and the trimmed fit computed exactly on every one, by visiting all 167,960 subsets of eleven rows, so that anything that goes wrong in its column is the estimator’s doing and not a search’s.

Against far rows, least squares is reversed on every one of the 200 datasets by a single bad row, at a median slope of −0.416. Huber’s estimate survives one far row in the median, at 0.112, but is already reversed on 16.0% of datasets, and at two far rows it is reversed on every dataset, at a median of −0.539. The trimmed fit holds its median through six bad rows of twenty — 0.396 at six, reversed on 9.0% — then weakens to 0.324 at seven with 27.0% reversed, and its median is reversed at eight.

Against rows that are low at an ordinary x, the ranking of the first two changes and the third does not. Least squares is reversed on 83.5% of datasets by one such row and on all of them by two. Huber’s estimate is the method working as designed: 0.365 with one bad row, 0.216 with three, reversed on 39.0% at four, and reversed on 99.5% at five. The trimmed fit holds through nine bad rows — median 0.391, reversed on 2.5% — and at ten is reversed on every dataset.

Ten is not an accident. With $h = \lfloor (n + p + 1)/2 \rfloor = 11$ rows used out of twenty, the trimmed fit’s breakdown point is $(n - h + 1)/n$, which is ten rows; at ten, the bad rows can supply a whole eleven-row subset of their own together with one clean row, and against both kinds of contamination every one of the 200 datasets reverses exactly there. So the two estimators’ breakdown points, stated in textbooks as one row in n and one half, are visible directly in the counts: Huber’s estimate breaks at two far rows of twenty and would break at one placed further out, and the trimmed fit breaks at half.

What the trimmed fit assumes #

The far-row count shows something the breakdown point does not promise against. At eight far rows of twenty the exact trimmed fit is already reversed on 55.5% of datasets, and at nine on 79.5%, below the half at which it is supposed to break.

The enumeration says why, because it records which rows the optimum chose. On the datasets where the trimmed fit is reversed at eight far rows, the eleven rows it uses contain on average 6.77 of the eight bad ones; on the datasets where it holds, they contain none. A tight cluster of rows at one far x, together with four or five clean rows, can be fitted by a straight line more closely than the best eleven of the clean rows can, because the clean rows carry noise along their whole range while the cluster sits at one place. Least trimmed squares chooses whichever half of the data looks most like a line. It does not know which half is right, and nothing in its objective could tell it — the same limit that stops a single summary from telling four datasets apart, since the objective is one number computed from whichever rows it keeps.

That is a precise sense in which a high breakdown point is a guarantee about bias being bounded rather than about the answer being correct. The low rows, spread across an ordinary range of x, never make a competing line of their own through the data, which is why the trimmed fit holds through nine of them; the far cluster does make one, and at eight rows it often wins. It is also exactly the redundancy that lets a pair of far rows hide from deletion, arriving from the other direction: rows that agree with each other are what fools a diagnostic, and rows that agree with each other are what a trimmed fit can mistake for the truth.

A search that can miss #

Enumeration is affordable at twenty rows and nowhere near affordable in practice, so what is called least trimmed squares in software is a search. From a random pair of rows it draws a line, refits least squares on the h rows closest to that line, and repeats — a concentration step, which can never increase the trimmed objective and so always stops at a local minimum. Many starts are tried and the best local minimum kept. Whether that is the global minimum is not something the search can say about itself.

With the exact answer known by enumeration, the search’s record can be counted. From a single start it misses the minimum on 68% of datasets; from two starts on 63%; from five on 37%; from ten on 14%; from fifty on 2%; and from two hundred on none of them. The misses are not harmless: from one start the reported slope is reversed on 48% of datasets, against 15% for the exact optimum on the same data. And no search, on any dataset at any number of starts, returned an objective below the enumerated minimum — which is the check that the enumeration is exact, since a search able to beat it would prove it wrong.

The same comparison runs inside the efficiency measurement below: on 200 clean datasets of twenty rows, a search with a hundred starts reached the exact optimum on 98.5%. At eighty rows, let alone thousands, no enumeration is possible and the check cannot be made. A trimmed fit computed from random starts is a sample, in the sense every seeded computation is a sample, and the number of starts is part of its definition.

The price when nothing is wrong #

Resistance is paid for in precision on data where there was nothing to resist, and here the price has two routes to it.

The closed forms come from each estimator’s influence function. For Huber’s loss the efficiency at the normal model is $(2\Phi(c) - 1)^2 / \mathrm{E}[\psi_c(Z)^2]$, and at c = 1.345 it is 0.9500 — the value is chosen to make it so. For least trimmed squares at half coverage the numerator and the denominator of the asymptotic variance turn out to be one integral, so the efficiency is that integral itself, $2\Phi(q) - 1 - 2q\,\phi(q)$ with $q = \Phi^{-1}(3/4)$, which is 0.0713: the 7.1% that Rousseeuw and Leroy’s account of the estimator reports.

Measured as a paired ratio — least squares’ mean squared error over each estimator’s, on the same clean datasets — Huber’s estimator reads 0.954 ± 0.008 at twenty rows over 2,000 datasets, 0.953 ± 0.016 at eighty over 1,000 and 0.960 ± 0.022 at 320 over 400: at its limit already at twenty rows, within its own error throughout. The trimmed fit reads 0.239 ± 0.007, 0.144 ± 0.012 and 0.086 ± 0.011, falling towards 0.0713 as the sample grows. It is more efficient at small n than its limit, not less, and the limit is the honest number to plan with. Read as a multiplier on the experiment, a trimmed fit at 320 rows needs roughly eleven times as many observations as least squares for the same precision.

The standard repair is a single reweighting step: estimate the error scale from the trimmed sum, corrected to be consistent at the normal, keep every row within 2.5 scales of the trimmed line, and fit least squares to those. It reads 0.432 ± 0.013, 0.683 ± 0.035 and 0.794 ± 0.041, keeping 86.6%, 95.8% and 98.0% of the rows against the 98.8% a 2.5-scale cut keeps under the normal. That recovers most of the precision at large samples and less than half of it at twenty rows, and it is measured here only; no closed form for it is attempted.

None of this is about the robust standard error, which is a different object with a different price. A robust standard error leaves the estimate alone and changes the claim made about its spread. These estimators change the estimate.

What was measured, and on what #

Every result here is about one straight-line design — twenty rows on a grid from 0.2 to 3.4, normal errors with standard deviation 0.35 — and two stated kinds of contamination. The mechanisms are general; the counts are about this design.

Huber’s estimator is measured in one configuration: c = 1.345, the scale re-estimated from the median absolute residual at every step, and least squares as the start. A fixed scale would make the answer unique and would need a scale from somewhere else. A redescending loss such as the bisquare gives a far row weight zero rather than a capped weight, and it is not measured here; it is non-convex, and it needs a start that has already resisted the far row, which is the role a trimmed fit is usually given.

Each breakdown count rests on 200 datasets, so a reversal share carries a binomial standard error of up to 3.5 points; the claims made from them are about where the medians and the shares change, which is far larger than that. The breakdown sweep’s trimmed fit is exact. The efficiency sweep’s is a hundred-start search, checked against enumeration where enumeration is possible.

And the efficiencies are at normal errors only. With heavy-tailed errors least squares is no longer the reference to beat, and the ranking of the first two estimators changes; that is a different question from the one about leverage, and it is left alone. So is regression with several predictors, where a far observation need not be far along any single axis.

Where this goes next #

Neither estimator measured here gives both things. Huber’s keeps 95% of least squares’ precision and breaks at one far row. The trimmed fit survives until nearly half and keeps 7% of it, and a single reweighting step buys back most of the rest at large samples and less than half at small ones.

Estimators built to have both exist. The MM-estimator starts from a high-breakdown fit, fixes the error scale from it, and then solves a redescending M-estimation from that start, aiming at a breakdown near one half and an efficiency near 95% at once. What this essay’s counts make measurable is whether those two properties survive together in finite samples, and in particular in the region where the exact trimmed fit was already choosing the wrong half — seven to nine far rows of twenty. If the initial fit picks the far cluster, an efficient step started from it may entrench that choice rather than repair it, and whether it does is a count of the same kind as the ones above, which has not been made.