The line that one point drew

Twenty observations lie along a line with a positive slope. Add one more, far out along the x axis and below the line, and the fitted slope becomes negative. Not attenuated — reversed.

The measurement #

Without the extra point the slope is 0.495. With it the slope is −0.389. One observation in twenty-one has moved the estimate by 0.88, which is nearly twice the slope it replaced, and has changed the qualitative conclusion from “y rises with x” to “y falls with x”.

Nothing about the added point is anomalous to look at. Its y value is not extreme in absolute terms — the data ranges more widely than the amount it sits below the line. What makes it decisive is where it sits on the x axis, and that is a quantity with a name and a formula.

Leverage, which is known before the outcome is #

For a simple regression the leverage of the i-th point is

$$h_i = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_j (x_j - \bar{x})^2}$$

Look at what is in it. The sample size, the x values, and nothing else. The outcome does not appear.

So leverage is a property of the design rather than of the data. It can be computed from a list of x values before a single measurement is taken, and it says how much the fitted line at that point is determined by that point’s own observation.

The leverages sum to 2 in a simple regression — one per fitted parameter — so the average is 2/n, which at n = 21 is about 0.095. The injected point’s leverage is 0.737.

That single number is the whole story. One observation is carrying nearly three quarters of the fitted value at its own location, where an average point carries a tenth. Whatever its y turns out to be, the line will follow it.

What twenty-one points make an ordinary leverage #

Leverage has a budget, and knowing the budget says immediately how unusual any particular value is.

The leverages of all n points sum to the number of fitted parameters. With twenty-one points and a straight line that is 2, so the average leverage is $2/21$, which is 0.095, and the conventional flag sits at twice that, 0.19.

A single point’s leverage can be anything up to 1, and at 1 the fitted line passes exactly through it whatever the other twenty points do.

So the quantity being watched is not “is this point unusual in y” but “how much of the design’s two units of leverage has one row taken”. A point holding a fifth of it — twice its share — is already flagged; a point holding half of it is deciding the answer.

That is also why the failure is invisible in the usual displays. The four datasets with one summary include exactly this case, and a panel of honest residual plots shows that a single large residual is what noise routinely produces. Neither a summary nor a residual carries leverage, and leverage is the half of the story that is known before the outcome is measured at all.

Influence, which needs the outcome too #

Leverage says how much a point could matter. Influence says how much it does, and it combines leverage with how far the point is from the line.

Cook’s distance is the standard measure: how far every fitted value moves if this observation is deleted. For the injected point it is 24.1, against a conventional threshold of 1 and against values well below 0.1 for every other point in the dataset.

The two quantities come apart in an instructive way, and the figure shows it. A point can have high leverage and low influence — if it sits at an extreme x and happens to fall on the line the rest of the data implies, it confirms the fit rather than dictating it. Dragging the injected point onto the line leaves its leverage at 0.737, unchanged, while its Cook’s distance falls.

So the correct summary is not that distant points are bad. It is that distant points are decisive: they determine the answer, and whether that is good or bad depends entirely on whether their outcome is trustworthy.

Why this is a design problem, not a data problem #

The framing that follows from leverage being computable in advance, and the reason this essay opens the field.

A design that places most of its x values in a cluster and one far away has decided, before collecting anything, that one observation will determine the slope. That is sometimes exactly right — spreading the x values out is how a slope is estimated precisely, and the standard error is proportional to one over the square root of the spread. Extreme x values are valuable.

The trouble is that the same arrangement makes the estimate fragile. Wide spread buys precision if every observation is sound, and buys a single point of failure if one is not.

That trade is quantifiable before the study runs. The leverages are computable from the intended design, and a design with one point at leverage 0.737 is one where a single measurement error, transcription slip or unrepresentative unit changes the published conclusion.

Two practical consequences.

Report the leverages, or at least the maximum. It is a property of the design, it costs nothing, and it tells a reader immediately whether the result rests on the bulk of the data or on a corner of it.

Prefer several observations at the extreme to one. Two points at the same distant x each have about half the leverage and the pair retains almost all the precision, while no single measurement can now carry the fit — though two wrong measurements there can hide each other from every single-row diagnostic, which is the price of the same redundancy. This is the cheapest robustness available in regression design and it is rarely done, because the distant condition is usually the expensive one.

What to do when a point is influential #

The question the diagnostic raises and does not answer. Four responses, in descending order of respectability.

Investigate it. An influential point is the observation most worth checking, and the check is often decisive: a transcription error, a unit mix-up, a subject who does not meet the inclusion criteria. This is the entire practical value of the diagnostic and it succeeds often enough to justify the whole apparatus.

Report both fits. If the point survives scrutiny, the honest presentation is the slope with it and the slope without it, stated plainly. A conclusion that depends on one observation out of twenty-one is a fact about the study, and concealing it is the failure this field exists to prevent.

Use a method that is less sensitive to it. Robust regression down-weights points with large residuals, and resists this one only in part — a bounded loss caps the residual and not the leverage, so how much depends on which method. That is a real option and it has a cost worth naming: a robust method will also down-weight a genuine, informative extreme observation, so it buys stability by discarding some of the precision the wide design was chosen for.

Delete it. Sometimes correct — a demonstrated error should be removed — and the most abused option in applied statistics, because the diagnostic that identifies an influential point identifies precisely the point whose removal most changes the answer. Deleting on the basis of influence alone, without an independent reason, is fitting the conclusion.

The ordering matters more than the list. Investigation is a question about the world; deletion is a decision about the analysis; and the difference between a defensible removal and an indefensible one is whether anything was learned in between.

Why the summary will not warn anyone #

The reason this needs a diagnostic rather than attention. The fit with the injected point reports a slope, a standard error, a t statistic and a p-value, and all four are computed correctly. None of them is distorted in a way that looks distorted.

The R² is not even low: the reversed fit explains a substantial share of the variance, because the injected point is far away and the line passing near it accounts for a great deal of the total spread. A large R² is produced by the same feature that makes the fit fragile, which inverts the usual reading of it — a topic with its own essay.

So there is no number in the standard output that flags the problem, and the only route to it is a diagnostic computed on purpose.

The thresholds, and how much to trust them #

Two conventions are in circulation and both deserve to be held loosely.

A leverage above 2/n or 3/n times the average is worth looking at. A Cook’s distance above 1 is worth looking at. Neither has a distributional justification of the kind a p-value has; they are rules of thumb calibrated on experience.

The reason to keep them loose is that they are thresholds on a continuum, and everything this site says about thresholds applies. A point at Cook’s distance 0.95 is not meaningfully different from one at 1.05, and a study reporting “no observation exceeded the conventional threshold” has said less than it appears to.

The more useful habit is comparative rather than absolute: look at the ranked influences and ask whether the largest is far from the rest. In the dataset here the answer is unmissable — 24.1 against a next largest below 0.2 — and no threshold was required to see it. Where the influences are all similar, no single point is carrying the fit and the diagnostic has done its job by saying so.

What the field takes from this #

Three things this essay establishes that the rest of the field builds on.

A regression summary does not determine what the data looks like. One point moved the slope through zero while the summary continued to report a well-behaved fit.

Some of the relevant diagnostics are available before the outcome is. Leverage is a function of the design, which means fragility is a choice made at the planning stage and not a misfortune discovered afterwards.

And the number that identifies the problem is not in the standard output. Slope, standard error, t and p are all correct and all silent. The diagnostic has to be computed deliberately, which means it has to be asked for, which means somebody has to know it exists.

The standard error is not measuring this #

A subtler point, and the one that explains why the fragility survives peer review.

The standard error of a slope is

$$\text{se}(\hat\beta) = \frac{s}{\sqrt{\sum (x_i - \bar{x})^2}}$$

and the denominator is the spread of the x values. A design with one point far out has a large denominator, so it reports a small standard error. The fragile fit comes with a confident interval around it.

That is not a bug in the formula. The standard error correctly describes how much the slope would vary across repeated samples if the model is right and the errors are what the model says — and under those assumptions the wide design really is more precise. The formula is answering its question correctly.

The trouble is that the question excludes the failure mode. A standard error quantifies sampling variability; it does not quantify sensitivity to a single observation being wrong in a way the model does not describe. Those are different sources of uncertainty and only the first is reported.

So the wide design produces an estimate that is genuinely more precise under its assumptions and genuinely more fragile to their violation, and only the first half appears in the output. The site’s gate checks the first half directly — the closed-form standard error is compared against the actual spread of four thousand slopes fitted to four thousand simulated datasets, and they agree — which establishes that the formula is right and leaves the fragility exactly where it was.

A correct standard error and a fragile estimate are compatible, and the diagnostic is what distinguishes them.

Two routes to the standard error #

Worth showing, because it is the check that separates “the formula is wrong” from “the formula answers a different question”, and this essay depends on the second being the case.

The closed form above is a claim: fit this model to repeated datasets and the slopes will have this standard deviation. That claim is checkable by doing it. Four thousand datasets from a known slope, four thousand fits, and the standard deviation of the resulting estimates is 0.1887 against a closed form of 0.1865 — agreeing to well within the Monte Carlo error of the comparison itself.

Two routes, sharing only the fitting code, required to meet. So the formula is right, least squares is unbiased for the slope, and everything unsettling in this essay is happening in a procedure whose advertised properties have been verified.

That ordering matters for how the field reads. The diagnostics here are not corrections to a broken method. They are answers to questions the method never claimed to address, applied to a method that does exactly what it says.

Where the same structure appears without a regression #

Leverage is presented as a regression diagnostic and the underlying idea is general: in any estimate that is a weighted average, some observations carry more weight, and the weights are usually determined by the design rather than by the data.

A mean has uniform leverage. Every observation carries 1/n, which is why a single point can shift a mean but never dominate it, and why means are comparatively robust.

A ratio estimator concentrates weight in the denominator’s small values. An observation with a small denominator dominates the ratio, and it does so for reasons visible before the numerator is measured.

A weighted survey estimate hands its leverage to the rare strata. A respondent weighted up by a factor of forty to represent an under-sampled group has forty times the influence of an ordinary respondent, and survey estimates have been reversed by one such respondent.

And any extrapolation is a leverage statement. Predicting outside the range of the x values means relying on points near the edge, and the further out the prediction, the more it is determined by the handful of observations at the extreme.

In every case the question is the same and answerable in advance: which observations, if wrong, would change the answer, and how many of them are there? For a regression the answer has a formula. Elsewhere it usually takes a moment’s thought and no more.

The general habit is the one the field is arguing for. A number’s uncertainty is routinely reported. Its dependence on individual inputs almost never is, and the second is often the larger risk.

What a reader can do with a published regression #

The essay’s diagnostics need the data, and a reader usually has only the paper. Three things can still be recovered from what is normally printed.

The range of the predictor tells most of the story. A scatter plot with a cluster and an outlier is visible at a glance, and if no scatter plot is shown for a simple regression, that omission is itself informative.

A very small standard error on a small sample suggests a wide design, by the formula above, and a wide design on a small sample is where high leverage lives.

And a conclusion that turns on the sign of a coefficient rather than its magnitude is the one most vulnerable to this, because reversing a sign takes far less influence than changing an order of magnitude — as the 0.495 to −0.389 above shows.

None of that is a substitute for the diagnostic. It is enough to know when to ask for it.

The diagnostic answers a question the standard output does not raise, and knowing when to raise it is most of the skill.

A closing note on the word “outlier” #

The vocabulary gets in the way here, and it is worth separating three things the single word is asked to cover.

An observation can be extreme in x, which is leverage and is a property of the design. It can be extreme in y given x, which is a large residual and is a property of the fit. And it can be influential, which is roughly the product of the two and is what actually changes the answer.

Calling all three “an outlier” loses the distinction that matters. A point extreme in x with a small residual is valuable and should be kept. A point with a large residual and low leverage is a curiosity and changes little. Only the combination is dangerous, and only the combination is what Cook’s distance measures.