Three mechanisms and one dataset

Four rules decide which outcomes go missing from the same two hundred rows. Each one loses the same 35% of them, each leans on whatever it reads with the same coefficient of 1.2, and the only thing that separates them is which column they look at. Three leave the fitted slope exactly where it was — the closed forms come out at 1.11 × 10⁻¹⁶, which is double-precision arithmetic saying zero — and the fourth moves it by −0.163531, which is 27.2552% of the quantity being estimated.

That is the whole taxonomy, and it is worth being precise about what it is not. It is not a statement about how much data is lost, because the four rules discard shares of it that differ by 0.08 percentage points. It is not a statement about how hard the rule bites, because the same coefficient of 1.2 drives all four. It is a statement about which variable the chance of being recorded is a function of, and — this is the part that makes the subject difficult rather than merely fiddly — that variable is the one thing a dataset with gaps in it cannot display.

A gap is not the only way a value can fail to be a number, and the distinction is worth making at the outset. A subject still event-free when a study ends is not missing: what happened to it is partly recorded, in the form of a bound, and the estimator that keeps such a subject in its own denominator recovers the truth to a fraction of a point because that bound is information. Everything here is the other case — the cell is simply empty, and the only thing carried alongside it is the fact of its being empty.

The world underneath is stated once and never estimated. Two standard normal covariates $x$ and $z$ correlated at 0.3, an outcome

$$y = 1 + 0.6\,x + 0.4\,z + \varepsilon, \qquad \varepsilon \sim N(0, 1),$$

and two hundred rows of it. The analysis being measured is least squares of $y$ on $x$ and $z$, which is the model the data were generated from: every failure below belongs to the missingness rather than to a misspecified fit, and keeping those two apart is most of what makes this subject readable.

What is held fixed, and it is nearly everything #

A row is recorded when a latent index crosses zero:

$$c_0 + c_x x + c_z z + c_y y + u > 0, \qquad u \sim N(0,1),$$

and the four mechanisms are four choices of which of $c_x$, $c_z$ and $c_y$ is 1.2 while the others are zero. Substituting the outcome equation turns the index into a constant plus a mean-zero normal of standard deviation $\tau$, so the marginal chance of being recorded is $\Phi(A)$ with $A = (c_0 + c_y)/\tau$ — and the intercept that hits a stated missing fraction can therefore be solved rather than searched for: $A = \Phi^{-1}(1 - f)$ and $c_0 = A\tau - c_y$.

The consequence is that $A$ is 0.385320 in every row of every table here, and the marginal chance of being recorded is 0.65 exactly rather than approximately. That is worth more than it sounds. A comparison of mechanisms calibrated by trial and error is a comparison in which the rows also differ, slightly, in how much they threw away — and every difference in a bias could then be read as a difference in sample size. Counted over four thousand studies the four rules keep 64.97%, 64.97%, 65.05% and 65.03% of the rows, and the spread across them is the 0.08 percentage points quoted above, which is Monte Carlo noise rather than a design difference.

What is held fixed also includes the strength. An index coefficient of 1.2 is a hard dependence, not a nudge: under the rule that reads the regressor, the chance of being recorded runs 0.036080, 0.274883, 0.726376, 0.964219, 0.998658 at $x$ equal to −2, −1, 0, 1 and 2. A row two standard deviations below the mean is recorded about one time in twenty-eight, and a row two above is recorded essentially always. That is deliberate, because the central claim here is an identity rather than a small-bias approximation, and a gentle setting would let the identity be mistaken for an approximation working well.

Which column the rule reads #

Selection here is a single linear constraint on a multivariate normal, which is exactly the situation whose truncated moments have a closed form. Writing $t$ for the standardised index and $\lambda$ for the inverse Mills ratio $\varphi(A)/\Phi(A)$, every variable splits into a part proportional to $t$ and a part independent of it, so

$$\mathbb{E}[v \mid \text{recorded}] = \mathbb{E}[v] + \operatorname{Cov}(v, t)\,\lambda,$$

$$\operatorname{Cov}(u, v \mid \text{recorded}) = \operatorname{Cov}(u, v) - \operatorname{Cov}(u,t)\operatorname{Cov}(v,t)\,\delta,$$

with $\delta = \lambda(\lambda + A)$. At the missing fraction used throughout, $\lambda$ is 0.569845 and $\delta$ is 0.544296. No integral is evaluated numerically and nothing is drawn: this is the population a complete-case analysis is actually running on.

Now solve the two-by-two normal equations in that population. The relation $\operatorname{Cov}(w, y) = \Sigma_{ww}\theta$ holds for the full population, and selection subtracts the same rank-one term from both sides, so almost everything cancels. What survives is the part of $\operatorname{Cov}(y, t)$ that does not pass through the covariates:

$$\theta_{cc} - \theta = -\,\delta \, p_\varepsilon \, V_{\text{sel}}^{-1} (p_x, p_z)',$$

where $p_\varepsilon$ is the covariance of the residual with the index. And $p_\varepsilon$ is zero exactly when the index does not read the outcome. That is the field’s central result stated as an identity: the bias under a covariate-driven rule is not small, it is zero, and no amount of strengthening the dependence can make it otherwise.

For the three rules that read a covariate, $p_\varepsilon$ is 0 and the closed bias is $\pm 1.11 \times 10^{-16}$ on the slope on $x$ and 0 or $\pm 5.55 \times 10^{-17}$ on the slope on $z$. For the rule that reads the outcome, $p_\varepsilon$ is 0.65116 and the slope converges to 0.436469 against a true 0.6, with the second coefficient converging to 0.290979 against 0.4.

The sample is wrong about nearly everything #

The figure above is the half that makes the first one surprising, and it deserves to be read slowly. Under the rule that reads the regressor, the rows that survive have a covariate mean of 0.437767 against a population mean of zero and a covariate standard deviation of 0.823879 against one. Their mean outcome is 1.315192 against a true 1 — wrong by 0.315192, counted at 1.3157 over four thousand studies. Under the rule that reads the outcome the mean outcome among the survivors is 1.617443, wrong by 0.617443, counted at 1.6186.

Any summary a reader might form from that sample is wrong. The covariate looks shifted and looks less variable than it is; the outcome looks half a unit too high; a histogram of either would mislead. And the slope is right to fourteen decimal places.

The reason is one sentence long and it is the sentence the rest of this field is built on. Least squares of $y$ on $(x, z)$ estimates the conditional law of the outcome given the covariates. Selection that acts on the covariates alone changes which covariate values appear and in what proportion — it changes the marginal law of $x$ and $z$ — and leaves the conditional law of $y$ given them untouched, because that conditional law is what the outcome equation states and the index never touches $\varepsilon$. A regression conditions on exactly the thing the selection moved, so the selection is invisible to it.

Selection that reads the outcome does something else entirely. It reads $\varepsilon$, so among the survivors at a given $x$ and $z$ the residual is no longer mean zero — it is a truncated normal whose mean depends on where that row sat — and the conditional law of the outcome given the covariates is now a different law from the one the world was built with. That is not a shortage of data. It is a different regression.

Read that way the outcome-driven case is a selection effect of a familiar shape rather than a new phenomenon. It is the mechanism behind an inflated effect among the studies that reached significance and behind the improvement of a group chosen for being worst: a filter applied to a quantity is a filter applied to the noise inside it, and the filtered noise then has a mean. What is specific here is only that the filter is invisible — the rows it removed are not in the table, and there is nothing to notice.

What the interval covers, and what it says it covers #

A bias of 0.163531 is not by itself a statement about anything a reader would act on. Turning it into one means asking what the ordinary interval does, and coverage is a count rather than an argument: build the interval in each of four thousand studies and see how often it holds the value the data were generated at.

The three covariate-driven rules give 95.15%, 95.40% and 94.80%, each within its own Monte Carlo error of the promise — the claim a 95% interval is making is a claim about a procedure run repeatedly, and here the procedure keeps it. The outcome-driven rule gives 49.70%. An interval advertised at nineteen times in twenty contains the truth about half the time, which is not a mild loss of precision; it is a procedure that has stopped doing the thing it is named after, in the same way a trial whose analysis does not know its own allocation rule has stopped testing anything.

The second coefficient tells the same story more quietly. Its interval covers 94.90%, 95.47% and 95.03% under the three harmless rules and 73.02% under the fourth, on a bias of −0.109021 closed and −0.10847 ± 0.00129 counted. It is less damaged only because the coefficient is smaller and its interval is the same width, which is a fact about the ratio rather than about the mechanism.

Two routes, and where they meet #

Every number above exists twice. The closed forms come out of the truncation identity; the counts come out of four thousand simulated studies that share none of that arithmetic; and the two are required to agree, which is the discipline this whole collection runs on. Under the outcome-driven rule the count lands on the closed form to −0.00012, against a standard error of 0.00131 — a tenth of one standard error, which is the sort of agreement that says the two routes are computing the same thing rather than two things that happen to be close.

The closed side is itself computed twice. The plim is obtained once by solving the selected normal equations directly and once through the rank-one identity written out above, and the two agree to machine precision. That matters more than a third significant figure, because the identity is the thing that generalises: it says why the covariate-driven bias is zero, where the direct solve only says that it is.

The bar that carries no useful information in either route is the one for the completely-random rule. It is there because it is the reference the other three are read against, and because its bias being zero is the only one of the four a reader would have predicted.

The line is scaled, not tilted #

There is a fact in the closed forms that nobody went looking for, and it is the most elegant thing in this field.

Under the outcome-driven rule the slope on $x$ is 0.436469 against 0.6 and the slope on $z$ is 0.290979 against 0.4. Divide each by its target: 0.727448 and 0.727448. Not two similar numbers — the same number, to every digit the arithmetic carries.

It follows from the algebra rather than from a coincidence. When the index reads the outcome and nothing else, its covariate part is $c_y(\beta x + \gamma z)$, so $\operatorname{Cov}(w, t)$ is proportional to $\Sigma_{ww}\theta$ — and applying the selected covariance’s inverse to something proportional to $\Sigma_{ww}\theta$ returns something proportional to $\theta$ itself, by the rank-one update. The bias vector is therefore parallel to the coefficient vector, and complete-case least squares does not distort the fitted relation at all: it shrinks the whole of it towards zero by one factor,

$$1 - \frac{\delta\, c_y^2 \sigma^2/\tau^2}{1 - \delta (c_y/\tau)^2\, \theta'\Sigma\theta},$$

which is a third route to 0.727448 and agrees with the other two to ten decimal places. Across index coefficients of 0.4, 0.8, 1.2, 1.6, 2.4 and 3.2 the factor reads 0.927933, 0.810025, 0.727448, 0.678543, 0.631286 and 0.611285 — a shrinkage that deepens with the dependence and never becomes a tilt.

The practical residue is a warning about diagnostics. A fitted relation that has been scaled rather than tilted looks entirely reasonable: the coefficients keep their signs, their ratio is unchanged, and a reader comparing the relative importance of two predictors would reach exactly the right conclusion. Only the levels are wrong, and only by a factor nothing in the data reveals. This is the opposite failure from a summary four different datasets share, where the numbers agree and the pictures do not; here the picture is right in shape and wrong in scale.

Which regression is protected, and which is not #

Protection is not a property of a mechanism alone. It is a property of a mechanism together with the analysis being run, and one row of the table shows it.

Under the rule that reads the second covariate $z$, the partial slope on $x$ — the coefficient in the regression that includes both covariates — is exactly right. The marginal slope of $y$ on $x$ alone converges to 0.683878 against a true 0.72, an error of −0.036122. Nothing about the mechanism changed between those two sentences. What changed is which regression was run: selection on $z$ is invisible to a regression that conditions on $z$ and is a straightforward selection effect on one that does not.

It is the same structural point three causal readings of one regression make about a covariate beside a treatment: the arithmetic does not change, and whether including a column is right is decided outside the fit.

So “missing at random” does not certify an analysis. It certifies the analyses that condition on whatever the mechanism reads, and the same dataset can carry one estimate that is exactly right and another, from the same rows, that is not. Under the outcome-driven rule the marginal slope converges to 0.503771, and both readings are wrong there.

That figure is the same point from the other side. Four estimators — the complete cases, two forms of weighting, and twenty imputations — put the marginal slope at biases of −0.00333, −0.00504, −0.00332 and −0.00293 against a target of 0.72, a spread of 0.00211 across all four. Anybody choosing between them on this evidence would be choosing between four right answers, and choosing badly, since the repairs cost spread and buy nothing. On the outcome’s mean, from the same rows and the same draws, the complete cases are wrong by 0.315192 and the other three are not. Whether a repair is worth anything is a question about the estimand, not about the mechanism.

What could have produced this without the claim being true #

Three ways, and each was closed rather than argued away.

The mechanisms could have differed in how much they discarded. They do not: the calibration is solved in closed form, the marginal chance of being recorded is 0.65 exactly in all four, and the counted shares differ by 0.08 percentage points. A bias that came from a smaller effective sample would also have shown up as a wider interval, and the intervals under the three harmless rules are the ones keeping their promise.

The zeros could have been small numbers rounded. They are not. The closed forms are $\pm 1.11 \times 10^{-16}$, which is one unit in the last place of a double near this magnitude, and the identity says they are zero for any $c_x$ and $c_z$ at any strength. Pushed to an index coefficient of six, where the survivors’ covariate variance has fallen to 0.470415, the closed bias is still exactly zero.

The counts could have agreed by luck. Four thousand studies give standard errors of about 0.0015 on each bias, and three of the four counts sit within one of them of zero while the fourth sits a hundred and twenty-five of them away. That gap is not a resolution question; it is the difference between a quantity that is zero and one that is not, and the practice of counting an advertised rate rather than asserting it is what makes it visible at all.

The row the data cannot name #

Everything above assumes the mechanism is known, and it is not. A dataset arrives with a pattern of gaps in it, and the question this whole taxonomy turns on — which column the rule read — is a question about the values that are not there.

It can be made exact. Two worlds can be constructed that produce identical covariates, identical patterns of what is recorded and identical recorded outcomes — not close, the same numbers — and whose true slopes are 0.6 and 0.315452. One is missing at random and one is not. Every statistic anybody can compute from the recorded data is the same number in both, so no test distinguishes them and no sample size ever will.

That is why this field opens on a taxonomy rather than on a repair. The three-line table above is a statement about which of four worlds a dataset came from, and the dataset is silent on the question. What is left is choosing an assumption and saying what it was worth — which is the shape of every honest answer under a resampling that cannot see past its own sample, and which the arguments that price filling the gaps, weighting them away and reporting a range instead each take up in turn.