Two points that hide each other

The line that one point drew placed a single observation far out on the x axis, eight units below the true line, and watched it turn a slope of 0.495 into −0.389. The diagnostic found it without any effort at all: a Cook’s distance of 24.1, against a conventional threshold of 1, with every other observation in the dataset below 0.22.

Now add a second observation beside the first — at x = 9.3 instead of 9, the same eight units below the true line — and run exactly the same diagnostic again. The slope moves further the wrong way, to −0.511. And Cook’s distance, the measure that found the lone point at 24.1, now reports 0.966 for one of the pair and 0.772 for the other. Neither crosses 1. The dataset has become more wrong, and the instrument built to say so has gone quiet.

That is masking, and it is not a defect in the arithmetic. Every number in this essay is computed exactly, twice, by routes that share nothing but the fitting code. The diagnostic answers its own question correctly. The question is about one row at a time, and two rows standing together answer it for each other.

A second row beside the first #

Four fits of the same twenty-two rows, and the figure draws three of them.

With every row in, the slope is −0.511. Delete the row at x = 9 and it is −0.376. Delete the row at x = 9.3 instead and it is −0.389 — which is not a coincidence, because what is left is exactly the dataset of the leverage essay, one far row and twenty clean ones. Delete both, and the slope returns to 0.495, the clean answer.

So removing either member of the pair leaves the conclusion reversed, and removing both restores it. The fit is not held by one row. It is held by two, each of which is enough on its own, so that taking one away changes almost nothing. That redundancy is the whole mechanism, and everything below is a consequence of it.

Measured on the scale Cook’s distance uses — the summed squared movement of every fitted value, divided by the number of parameters and the error variance — deleting the pair together moves the fit by 53.3. That is more than twice the lone point’s 24.1, and more than fifty times the largest value either member of the pair reports alone. The influence did not go away when the second row arrived. It was pooled, and the single-row measure cannot see a pool.

The summary statistics did what the four datasets with one summary would predict, which is to improve. With one far row the fit’s R² was 0.315; with two it is 0.551, because a line passing near two distant observations accounts for a great deal of the spread in y. A worse dataset, a larger R², and — as the essay on what R² measures argues at length — no reason to have expected anything else.

What deleting one row asks #

Cook’s distance for row i is the product of two quantities, and writing it out shows exactly where the pair gets through.

$$D_i = \frac{e_i^2}{p\,s^2}\cdot\frac{h_{ii}}{(1-h_{ii})^2}$$

The first factor is how far the row sits from the fitted line, $e_i$, on the scale of the error variance. The second is how much leverage the row holds, $h_{ii}$. A row is influential when it is both far from the line and heavily weighted in deciding where the line goes.

The pair’s leverages are 0.410 and 0.444, each more than twice the conventional flag. So the second factor is large, and the only way the product can stay under 1 is for the first to be small — and it is. The line fitted to all twenty-two rows passes close to both far observations, because the two of them together have dragged it there. Their residuals are small for the same reason the line is wrong.

The deletion formula makes the rest explicit. The change in the coefficients when row i is removed is, exactly,

$$\hat\beta - \hat\beta_{(i)} = (X^\top X)^{-1} x_i\,\frac{e_i}{1-h_{ii}}$$

and every quantity on the right-hand side belongs to the full fit — the fit that has already been bent to the pair. Deleting row A asks how far the line moves when A is gone; the answer is computed from a line that B is still holding in place. The formula is not an approximation. It returns precisely the refitted coefficients, which is the trouble: it answers the question it was built for with complete accuracy, and for two rows that is the wrong question.

The figure shows the pair where the leverage essay’s single point sat, far to the right, and a long way below where that point sat on the vertical axis. The threshold at 1 passes over both.

There is a stricter convention in circulation, 4/n, which for twenty-two rows is 0.182, and by that convention both members of the pair are flagged. It would be convenient to conclude that the stricter rule solves the problem. It does not, for two reasons the next sections measure: it flags a clean row as well, and once the cluster grows past four rows it stops flagging the cluster at all. (For a straight line p = 2, so 4/n and the leverage flag 2p/n are the same number here, which is a coincidence of the model’s size and not a relationship between the two conventions.)

The yardstick without the row, fooled the same way #

Cook’s distance scales each residual by the error variance of the full fit, and a far outlier inflates that variance, which is one reason a single diagnostic is never enough. The deleted residual was designed to close that gap:

$$t_i = \frac{e_i}{s_{(i)}\sqrt{1-h_{ii}}}$$

where $s_{(i)}$ is the error scale estimated with row i removed. A row cannot inflate its own yardstick, because the yardstick is computed without it. For the lone far point of the leverage essay this statistic reads −13.29, which no convention would pass.

For the pair it reads −1.75 and −1.43. The yardstick is estimated without the row — and with its partner. The partner keeps the line close to both, so the residual in the numerator is small, and it keeps the error scale near what the full fit reports, so the denominator is not shrunk. The design that protects against one kind of self-concealment is exactly as open to concealment by a second row as Cook’s distance is.

Scanning twenty-two deleted residuals for any beyond ±2 is also twenty-two tests, and a scan of many honest statistics finds something beyond a nominal threshold far more often than the threshold suggests. The familywise version of the cut-off, Bonferroni’s, is ±3.52 for twenty-two rows at 5% — the correction that holds the familywise rate where it is promised. Neither the routine ±2 nor the corrected ±3.52 comes anywhere near the pair.

The clean row that gets the flag instead #

The same scan does report something beyond ±2, and it is not the pair.

The largest deleted residual in the data, −2.26, belongs to the clean observation at x = 0.2 — the leftmost of the twenty honest points. It is the only row beyond ±2, and by the 4/n convention its Cook’s distance of 0.208 is flagged too. Ranked by the absolute deleted residual, the two far rows come second and fifth.

The mechanism is geometric. The pair pulled the right-hand end of the line down, and a least-squares line pivots about the centre of its data, so the left-hand end went up. The fitted value at x = 0.2 is now well above where the clean observations sit, and the leftmost of them — the one furthest from the pivot on the other side — acquires the largest residual. The outlier-detection literature calls this swamping, and it is masking’s mirror: the rows that are wrong conceal themselves, and in doing so they push the suspicion onto a row that is right.

In practice this is the worse half. A scan that reads the flags will investigate the row at x = 0.2, find nothing wrong with it — there is nothing wrong with it — and conclude that the data have been checked. The panel of honest residual plots already shows that one residual around two is what correct data routinely produce, so the finding will look like noise, which in a sense it is. The signal was elsewhere and it was switched off.

Leverage does not read the outcome, so it is not fooled #

One diagnostic passes through all of this untouched, and the reason is in its formula.

Leverage depends on the x values and the sample size and on nothing else. The pair’s leverages, 0.410 and 0.444, are both far above 2p/n = 0.182, and they would be exactly the same whatever y values the two rows carried — on the true line, eight below it, or eighty. Masking works through the residual and through the refit, and both of those need the outcome. A quantity that never reads the outcome cannot be persuaded by it.

The hat matrix says something more specific than that the rows are far out. Its off-diagonal element for the pair, $h_{AB}$, is 0.426 — between the two rows’ own diagonals of 0.410 and 0.444, and close to both. An off-diagonal element measures how much one row’s observation contributes to the fitted value at the other row, so this says that, as far as the fit is concerned, the two rows occupy nearly one place: either observation moves the fitted value at both locations by almost the same amount.

The two-by-two block of the hat matrix belonging to the pair has a largest eigenvalue of 0.854, and a second that is almost zero. That eigenvalue is the leverage the pair holds jointly — how much of the fitted value at their shared location is decided by their shared observation. One far row held 0.737 of it. Two rows beside each other hold 0.854 between them, and each, examined alone, looks as if it holds about half. The design-side arithmetic states the redundancy outright, before any y is measured.

Leverage is not a sufficient diagnostic, and this is not an argument that it is. It flags a far observation that sits exactly on the line as readily as one that does not, which is why the leverage essay needed Cook’s distance at all. But it has the one property the deletion diagnostics lack against this failure: no arrangement of outcomes can talk it out of a flag. That is also its limit, as the cluster grows.

Where the mask is strongest #

Masking needs the two rows to agree, and how much they need to agree can be measured by holding one fixed and moving the other.

Put the second row on the true line and it hides nothing. The fixed row reads Cook’s distance 6.57, lower than the lone point’s 24.1 — a good far observation does steady the fit somewhat — but far above any threshold. And the good row itself reads 3.71, flagged, because the bad row beside it pulls the line away from it: a correct observation at high leverage, reported as the most influential thing in the data after the error.

Lower the second row towards the first and the fixed row’s distance falls steadily, crossing 1 just before the two are level. At the level — both eight below — the pair reads 0.966 and 0.772 and nothing is flagged. Lower the second row past the first and the roles swap: at twelve below, the moving row reads 3.81 and the fixed row, still eight below the line and still wrong by exactly as much as before, reads 0.021. It has become invisible behind a row that disagrees with the rest of the data more than it does.

So masking is not a property of two outliers. It is a property of two rows that agree with each other and disagree with everything else. Two errors made independently rarely agree that closely. Two observations that share a cause — the same instrument miscalibrated on the same afternoon, the same site with a data entry convention of its own, the same subgroup the model does not describe — agree by construction, and those are the errors most worth finding.

A crowd of far rows #

Two rows are the smallest case. Grow the cluster one row at a time, each 0.3 further out and each eight below the true line, and both families of diagnostic move.

The largest Cook’s distance any single member of the cluster reports falls from 24.1 with one row to 0.966 with two, 0.297 with three and 0.168 with four — which, against the 4/n threshold of 0.167 for twenty-four rows, is flagged by a margin of a little over a thousandth. At five rows it is 0.126 against 0.160, and from there on no member of the cluster is flagged by either convention; at eight rows the largest is 0.096. The slope, meanwhile, stays reversed at every size, and deleting the whole cluster moves the fit by 87.0 at three rows and 301.5 at eight. The deleted residuals tell the same story: the largest in the cluster never exceeds 1.75 in absolute size from two rows onward, while some clean row always exceeds 2.

Leverage holds out longer, and then it too starts letting rows through.

The cluster’s rows share what one row held. Their total leverage rises only from 0.737 to 0.970 across eight rows, so each row’s share falls roughly as one over the cluster’s size. With six rows the nearest member of the cluster holds 0.131, under the 2p/n flag of 0.154 for twenty-six rows, and by five rows every member is under the looser 3p/n. Leverage was never fooled by the outcome, and it still is not; what happens here is that the thresholds are stated per row, and a crowd divides its leverage among its members until each looks ordinary.

At some size a far cluster stops being a set of outliers and becomes a second group of observations, and there is no size at which that happens by arithmetic. Eight rows out of twenty-eight near x = 9 is a design with two regions, and whether the far region or the near one is telling the truth about the slope is not a question any diagnostic can settle from these data alone. A design that puts its leverage in a few rows makes every downstream quantity depend on those rows, and a crowd of them is the same dependence spread thinner — and where the relationship itself differs between the two regions, what a straight line estimates is decided by the design rather than by either region alone.

Deleting pairs, and what the search costs #

If deleting one row at a time cannot see a pair, deleting two at a time can, and here it does so unmistakably.

There are 231 ways to choose two rows from twenty-two. Deleting the far pair moves the fit by 53.3. The next largest pair reads 0.973, and it consists of one far row and one clean row, which is to say it is essentially the single-row deletion again; the largest pair made of two clean rows reads 0.564. Exactly one pair of the 231 crosses 1, and it is the right one, by a factor of more than fifty.

The difficulty is what that costs, and the costs are exact. A scan of every group of up to two rows is 253 refits. Up to three rows it is 1,793; up to five, 35,442; up to eleven — half the data — 2,449,867. At a hundred rows, a scan of every group of up to five is 79,375,495 refits, before any question of what threshold a group statistic should be held to when that many of them are being compared.

And the scan has to be told what size of group to look for. Against a cluster of three, the same search over pairs half-works: the only pairs over 1 are the three drawn from the cluster, at 2.77, 2.36 and 1.98, so it points at the right rows — but deleting any one of those pairs leaves the third far row in place and the slope still reversed, at no better than −0.36. The pair search flags the cluster and its remedy does not remove it; only the group of three, at 87.0, does. Every cluster size is a separate computation, and the number of such computations grows combinatorially with the data. The single-row diagnostics were attractive precisely because they cost n refits — or none, since the closed form needs only the full fit — and the property that makes them cheap is the property that makes them blind.

Two routes to every deletion #

Every deletion quantity here comes from the full fit through the hat matrix: a rank-one update for one row, and for a group of rows G the version of rank equal to the group’s size,

$$\hat\beta - \hat\beta_{(G)} = (X^\top X)^{-1} X_G^\top (\mathrm{Id} - H_{GG})^{-1} e_G$$

with the group’s Cook’s distance being that change measured in the metric of $X^\top X$. Each is required to agree with a second route that shares nothing with it but the least-squares routine: remove the rows, fit again, and measure how far every fitted value moved.

On all 22 single deletions and all 231 pairs — 253 deletions in all — the slopes agree within $10^{-10}$ and the Cook’s distances within a relative $10^{-9}$, and the deleted residuals are rebuilt from the refitted line’s prediction error at the deleted row with the same agreement. That check is the reason the numbers above can be stated to three decimals without hedging, and it is the same discipline as a number computed by two routes that share nothing.

There is no simulation anywhere in this essay. The twenty clean points were drawn once from a stated seed, in the leverage essay, and every quantity since is deterministic arithmetic on them, so there is no Monte Carlo error to report: the figures are exact to rounding, and a reader with the dataset would reproduce them to the last digit printed.

A constructed case shows how, and says nothing about how often #

This is one dataset, built to exhibit a mechanism, and what it establishes should be stated narrowly.

It shows that two rows can conceal each other from Cook’s distance and from the deleted residual while reversing a slope, that the same arrangement pushes a flag onto a clean row, and that leverage is immune to the concealment because it does not read the outcome. Those are statements about how the diagnostics work, and they hold for any data with this structure, because the formulas above are exact.

It does not show how often masking occurs in real data, and nothing here estimates that. The sweep shows the concealment weakening quickly as the two rows disagree, so two errors made independently will usually be caught one at a time; the dangerous case is correlated error, whose frequency is a fact about the process that produced the data rather than about regression.

The thresholds are conventions and are treated as such. By Cook’s distance at 1 the pair passes; by 4/n it is caught, along with a clean row, until the cluster reaches five. Neither convention has a distributional justification of the kind a p-value has, and the Bonferroni bound on the deleted residual assumes the scan is the only look at the data, which in practice it rarely is.

And it is about a straight line. The formulas generalise to any number of predictors without change, but one feature of this case does not: here a far row is far in the one direction there is, and a reader can see the pair on the scatter plot. That stops being true with several predictors, which is where the problem becomes genuinely hard.

Where this goes next #

The search over groups is exactly what the single-row diagnostics were designed to avoid, and its cost is combinatorial in the size of the data and in the size of the group. The alternative is a fit that does not need to be told which rows to delete — one that decides, as part of fitting, which observations to believe.

A robust loss and a far x takes the two standard candidates. A bounded loss limits how hard a large residual can pull; a trimmed fit uses only the half of the data that fits best. The masking measured here already suggests which of them will matter against far rows, since a far row’s residual is small precisely because it has pulled the line to itself — and a loss that reads only the residual will not see a row that has done that.

The question this essay opens and does not answer is the one the last section named. With several predictors a cluster of rows can be unremarkable along every axis separately and far out in their joint space, and — worse — a cluster can inflate the very covariance of the predictors that the hat values are computed from, pulling the centre towards itself and stretching the spread until its own members look ordinary. Then leverage, the one diagnostic this essay found impossible to fool, is masked as well. What that needs is a robust estimate of where the predictors are centred and how they spread, computed before any leverage is believed, and whether that estimate inherits the same combinatorial search this essay ran into is the next thing worth measuring.