The start an efficient robust line inherits

A robust loss and a far x found that neither estimator it measured gave both things a robust line is wanted for. Huber’s loss kept 95% of least squares’ efficiency and broke at one far row; least trimmed squares survived until nearly half the rows were bad and kept 7.13% of the efficiency. It named the estimator built to have both. The MM-estimator starts from a high-breakdown fit, fixes the error scale from that fit’s residuals, and then solves a redescending M-estimation from that start — aiming, in its textbook description, at a breakdown point of one half and an efficiency of 95% at once.

Both halves of that description are statements about limits, and this essay measures them in finite samples on the same design: twenty clean rows on a grid from 0.2 to 3.4 with a true slope of 0.4, the same far rows near x = 9 and the same low rows at an ordinary x. The start is the trimmed fit found exactly, by visiting every subset, so that anything the MM-estimator does wrong is its own doing. The question the earlier essay left open is the sharp one: in the region where the exact trimmed fit was already choosing the wrong half of the data — seven to nine far rows of twenty — does an efficient step taken from that start repair the choice, or entrench it?

Three pieces, and the two numbers that tune them #

The estimator is built from Tukey’s bisquare. Inside c scales of the line a row’s weight is $(1 - (r/cs)^2)^2$, which is near one for a small residual and falls smoothly; outside c scales it is exactly zero. That is the difference from Huber’s loss, whose weight past its bend is c divided by the residual and never reaches zero. A bisquare row that is far enough from the line is not capped. It is ignored.

The scale comes first and is then held fixed. It is an M-scale of the start’s residuals: the s at which the average of the bisquare’s ρ at r/s equals one half. The tuning constant for that step is chosen so that a sample of normal errors gives the right scale, and computed by bisection on the normal integral it is 1.5476, the value that makes the scale break down only when half the rows are bad. The M-estimation that follows uses a different tuning, 4.685, and its efficiency at the normal model is a closed ratio of two integrals of the bisquare’s ψ, which by quadrature is 0.95000. A tuning of 3.44 gives 0.849.

So the estimator’s two promises come from two constants, and the order of the steps matters. The high-breakdown start decides where the iteration begins, the M-scale is computed once from it and never re-estimated — the re-estimated scale was what gave Huber’s iteration two fixed points in the band from x = 10.5 to 12.5 — and the redescending loss, which is not convex, is left to find the nearest fixed point to its start.

One far row at weight zero #

On the leverage essay’s own data, with the far row at x = 9 on a line of slope −0.711, least squares reads −0.389, Huber’s estimator 0.171 and the exact trimmed fit 0.420. The MM-estimator reads 0.479, where the twenty clean rows alone give 0.495. The far row’s weight is 0, exactly; the scale the step held fixed is 0.319, against a true error standard deviation of 0.35.

The 0.479 is not a failure to reach 0.495, and the reason is the price the estimator is designed to pay. The bisquare discounts every row in proportion to its residual, clean rows included, and the lowest weight among the twenty clean rows is 0.547. A fit that discounts ordinary residuals a little is a fit that gives up a little efficiency on clean data, which is the 5% the tuning of 4.685 was chosen to spend. The trimmed fit’s 0.420 is further from 0.495 because it uses only eleven of the twenty-one rows; the MM-estimator uses twenty of them at substantial weight.

Moved inside the data — the extra row at x = 3, eight below the line, where Huber’s estimate read 0.443 — the MM-estimator reads 0.479 again, with the row again at weight zero. Placed exactly on the true line, the row is kept at weight 0.962 and the slope reads 0.461. The bisquare does not know which rows are bad; it knows which are far from the line it started near.

The weights make the difference between the two losses visible. Huber’s estimator gave the far row weight 0.115 and multiplied that capped residual by a lever of length seven; the MM-estimator’s weight is zero, so the lever has nothing to multiply. The clean row at x = 0.2 is the lowest-weighted clean row under both, 0.612 under Huber and 0.547 under the bisquare, but for different reasons. Huber’s estimate discounted it because the far row had tilted the line and given it a residual of the size a correct model routinely produces; the MM line was never tilted, and the bisquare discounts that row, and the one at x = 2.73 to 0.670, because the bisquare discounts every sizeable residual a little. That is the efficiency the tuning spends, paid row by row.

The start decides which fixed point #

Carry the row out along the wrong line, as the earlier essay did. The MM-estimator started from the trimmed fit reads 0.479 at every distance from x = 3.4 to x = 40, with the row at weight zero throughout, because the trimmed fit never uses the row and the scale taken from its residuals never sees it.

Start the same estimator from least squares instead and it reads 0.494 as far out as x = 11, and −0.465 at 11.5. The difference is not in the loss but in the scale the start hands it. The residuals of the tilted least-squares line are large for every row, so their M-scale is 1.154 at x = 10 and 1.226 at 11.5, nearly four times the trimmed start’s 0.319, and at that scale no row is anywhere near 4.685 scales from the line: the far row sits 1.88 scales off it at x = 10 and 1.60 at 11.5. The first steps therefore discard nothing, and where the iteration ends depends only on which basin of the non-convex loss the tilted line lies in. Up to x = 11 it drifts back towards the clean rows and the far row’s weight falls to zero; from 11.5 the row’s leverage holds the line in the other basin, the row keeps weight 0.673, and the fit settles near the wrong line — −0.701 by x = 40. The two starts reach different lines from the same data, and nothing in the converged estimate says which start produced it. It is the same lesson the line that one point drew taught about least squares, moved one step along: whatever decides the answer has to be looked at, and here what decides it is the start.

That is the precise sense in which the MM-estimator’s breakdown point belongs to its start. The redescending loss is not convex, its fixed points are local, and a start that has already been pulled by a far row delivers the iteration to the pulled fixed point. The textbook breakdown of one half is the start’s, borrowed.

Breakdown, counted exactly #

The same two hundred datasets at each count of far rows, the same exact trimmed fit, and the MM-estimator carried on from it. With no bad rows it is never reversed, where the trimmed fit is reversed on 1.5% of datasets by noise alone. With three far rows its median is 0.383 and its middle 80% runs from 0.277 to 0.507, against the trimmed fit’s 0.178 to 0.557 — the same centre with three-fifths of the spread, which is efficiency showing up as a narrower bar. At six far rows it is reversed on 9.0% of datasets, at seven on 27.0%, at eight on 55.5% and at nine on 79.5%.

Those four shares are the trimmed fit’s shares exactly. And started from least squares, the MM-estimator is reversed on every one of the two hundred datasets from two far rows onwards, with a median of −0.540 — breaking where Huber’s estimator broke.

Against low rows at an ordinary x the same pattern holds and holds longer: the MM-estimator is reversed on no dataset until eight low rows, where it reverses on 0.5%, and at nine on 2.5%, again the trimmed fit’s share. From least squares it is reversed on 53.5% of datasets with three low rows and on all of them from four.

Repair or entrench #

The counts can be split by what the start did. At eight far rows the trimmed fit is reversed on 111 datasets. The MM-estimator repairs 0 of them. On the other 89, where the trimmed fit held, it spoils 0. At seven far rows the trimmed fit is reversed on 54 datasets and at nine on 159, and at every count from five upwards the efficient step repairs none of the reversals and spoils none of the holds. With four far rows it repairs 1 of the trimmed fit’s 4 reversals. With three or fewer, the only reversals the trimmed fit has are the handful that noise produces — 3 with no bad rows at all — and the MM-estimator repairs every one of those.

So the step does both things the earlier essay wondered about, in two different regimes. Where the trimmed fit’s wrong sign is an accident of noise on the right half of the data, the efficient step, using twenty rows instead of eleven, corrects it. Where the trimmed fit’s wrong sign is a choice of the wrong half, the step never corrects it, because the scale it fixed came from that half’s residuals: measured from the line through the far cluster, the cluster’s rows are well inside 4.685 scales and many clean rows are not.

And it does not merely keep the choice; it commits to it. At eight far rows the MM-estimator’s median slope is −0.615 against the trimmed fit’s −0.541, nearer the wrong line’s −0.711, and across all two hundred datasets the bad rows’ average bisquare weight is 0.539. At nine far rows that weight is 0.771 and the median −0.649.

The fixed scale is where the start’s choice is carried forward, and its size can be read off the counts. With no bad rows the M-scale of the trimmed fit’s residuals has a median of 0.302, a little under the true 0.35 because the trimmed fit’s own residuals are its smallest half. It grows with the bad rows the start has to absorb: 0.459 at five far rows and 0.638 at eight. A bisquare step discards only rows beyond 4.685 scales, so at eight far rows it discards nothing closer than three units to its line. On the datasets where the start chose the clean half, three units is far closer than the far cluster sits, and the cluster is dropped; on the datasets where it chose the cluster, three units is wider than most clean rows’ residuals from the wrong line, and the clean rows stay in at partial weight, pulling nothing back. Carried on from a start on the wrong half, the efficient step is efficient about the wrong line.

That is the finite-sample content of the two promises. The breakdown point of one half is real, and it is the start’s; below it the efficient step is a genuine improvement, and past the point where the start chooses the wrong half — which for a tight far cluster is well below half, for the reason redundant far rows fool any fit that looks for agreement — nothing about the MM step can tell. A single summary cannot say which half of the data is right, and the MM-estimator’s objective, computed from whichever rows it weights, is one.

The efficiency the step buys #

On clean data the step pays for itself. Measured as a paired ratio against least squares on the same datasets, with the trimmed fit found by a hundred-start search as it is run in practice, the MM-estimator at a tuning of 4.685 reads 0.872 ± 0.015 at twenty rows, 0.947 ± 0.016 at eighty and 0.958 ± 0.024 at 320, at its limit of 0.95 by eighty rows. At twenty rows that is more than twice the 0.432 a single reweighting step recovers and 3.7 times the trimmed fit’s 0.239.

At a tuning of 3.44 it reads 0.683 ± 0.019, 0.838 ± 0.025 and 0.856 ± 0.036, towards its limit of 0.849. The lower tuning discounts moderate residuals more, which buys nothing in the breakdown counts here: its reversal shares match the higher tuning’s at every count from one far row to ten, because the choice of half was made before either tuning was applied. What a lower tuning would buy is less bias from rows that sit a few scales off the line, which these designs do not include.

Read as a multiplier on the experiment, a trimmed fit at 320 rows needed roughly eleven times the data least squares needed for the same precision; the MM-estimator carried on from it needs about four per cent more. That is the whole of what the step is for, and on clean data and on data with a minority of bad rows it delivers it.

What a robust regression report should state #

The start, and whether it was exact. An MM-estimate from a trimmed fit and an MM-estimate from least squares are different estimators with breakdown points of one half and one row in twenty, and the converged line carries no mark of which it was. A trimmed start found by search is a sample of the search’s starts and the MM step inherits that too.

The scale and both tunings. The scale fixed from the start, 1.5476 for its M-scale and the efficiency tuning for the step; a scale re-estimated during the iteration is a different estimator with different fixed points.

The rows the fit gave weight zero, and where they sit. A bisquare fit that ignores a cluster of rows far out on x has made a decision about which half of the data is right, and the rows it ignored are the evidence for the other half. Reporting them is the only way a reader can see the choice the start made.

None of this is the robust standard error, which changes the claim made about an estimate’s spread and leaves the estimate alone. The MM-estimator changes the estimate, and inherits its start’s judgement to do so.

Measured on what #

One straight-line design, normal errors with standard deviation 0.35, and two stated kinds of contamination, with two hundred datasets at each count; the trimmed start exact in the breakdown counts and a hundred-start search in the efficiency measurement. The M-scale’s tuning and the step’s efficiency are computed by quadrature at the normal model; everything else is counted. The bisquare is the only redescending loss measured, and the least trimmed squares fit the only start besides least squares; the S-estimator, the start most MM software actually uses, is not measured here.

Still open: telling which half #

The counts locate the MM-estimator’s failure precisely: a start that chooses the half of the data holding a tight far cluster, which the efficient step then makes more confident. The natural response is not a better step but a check on the start. The exact trimmed fit’s enumeration already holds the second-best subset of eleven rows alongside the best, and when the best and the second-best lines disagree in sign while their objectives are close, the data contain two competing lines and no estimator built on one objective can choose between them honestly. How often the two best subsets disagree at seven, eight and nine far rows, how close their objectives are when they do, and whether a report that printed both lines would have flagged the 111 datasets on which the efficient step went on to entrench the wrong one, is a count these enumerations can make and have not made.