A score that balances

On the population every number here is measured in, the difference between the treated arm’s average outcome and the control arm’s is 2.7102. The average effect of the treatment over everybody is exactly 1. The gap between those two numbers — 1.7102, nearly twice the effect itself — is not noise, is not a small-sample artefact, and does not shrink with more rows. It is what happens when the units that received the treatment were not the same units as the ones that did not.

They differ in a way that can be written down. Two covariates are recorded before assignment, both standard normal and independent of each other; the probability of being treated is $\operatorname{expit}(0.5 + x_1 + 0.75\,x_2)$, so units high on either covariate are likelier to be treated, and 59.5% of the population ends up in the treated arm. The outcome rises with both covariates whatever the treatment does. So the treated arm is a sample with higher covariates in it, and the untreated arm is a sample with lower ones, and the difference of their means is the effect of the treatment plus the effect of the covariates being different.

How different is measurable in the units imbalance is always reported in: a difference of arm means divided by the root of the average of the two arms’ variances. Integrated over the population rather than counted in any sample, the arms sit 0.8310 apart on the first covariate and 0.6015 apart on the second. In the vocabulary a randomised trial uses those are enormous — the essay that enumerated every possible split of sixteen units found a third of them landing more than half a standard deviation apart and treated that as the scandal it is.

Why the removal is exact rather than merely good #

The repair is one line of arithmetic. Give each unit a weight of one over the probability it had of receiving the treatment it actually received: $1/e(x)$ for a treated unit, $1/(1-e(x))$ for an untreated one, where $e(x)$ is the propensity score. Then compute the weighted arm means.

What that does to the covariate distribution is worth writing out, because it is the whole of the claim and it takes one step. Among treated units the covariates have density $e(x)\varphi(x)$ — the population’s own density, tilted towards whoever was likely to be treated. Weighting by $1/e(x)$ multiplies that density by exactly the factor that produced it:

$$\frac{e(x)\,\varphi(x)}{e(x)} = \varphi(x).$$

The same cancellation runs in the control arm with $1-e(x)$. Both weighted arms therefore carry the population’s own covariate density, whatever the assignment rule was. The result is not an approximation that improves as the assignment gets gentler, and it is not a bound. It is an identity, and the only thing that can spoil it is not knowing $e(x)$.

Here $e(x)$ is known, so the identity can be checked rather than trusted. Integrated over a 241-by-241 product grid carrying 58,081 points of the covariate space — the same grid every population quantity in this field is computed on, whose mass integrates to one to twelve digits — the weighted standardised difference on the first covariate is $-2.8\times10^{-17}$ and on the second $-5.7\times10^{-19}$. Those are machine precision. They are not small numbers that a reader is invited to round; there is nothing left to round.

A number that small invites the obvious suspicion: that it is an artefact of the arithmetic rather than a property of the weights. Two things rule that out. The first is that the quantity is a ratio — a difference of weighted means over the root of an average of weighted variances — so a grid whose weights happened to be slightly wrong would move the numerator and the denominator together and leave a residue at the size of the grid’s own error, which is around $10^{-13}$, not $10^{-17}$. The second is that the same integration, on the same grid, with the same code, returns 0.8310 and 0.6015 when the weighting is switched off, and 0.7057 when the weighting uses a score that omits a covariate. An integration scheme that manufactured zeros would manufacture them in all three columns.

The claim is therefore asserted at $10^{-9}$ rather than at a tolerance chosen to fit. That distinction matters more than it looks: a bound of $10^{-3}$ would be met by any quantity that merely got small, and “got small” is what a good approximation does. The whole content of the identity is that this one is not an approximation, so the assertion has to be tight enough that an approximation would fail it.

A scalar standing in for a vector, measured as a limit #

The identity above is about weighting. The claim that gives the propensity score its name is stronger and different: that the score is a balancing score — that treatment and the whole covariate vector are independent given the single number $e(x)$, so conditioning on one scalar does what conditioning on every covariate separately would do.

That claim is usually cited. It can be measured, and the measurement is a limit rather than a number. Split the population into bands of equal mass by the value of the score, read the treated-minus-control difference on the second covariate inside each band, and average those over the bands weighted by mass. If the score is a balancing score, the residual has to fall to nothing as the bands narrow, because within a narrow band of $e(x)$ the assignment is very nearly a coin toss and a coin toss balances everything.

It does. At one band — which is no conditioning at all — the residual is 0.5766. At two bands 0.2166, at five 0.0632, at ten 0.0262, at twenty 0.0115, at fifty 0.0046. A fall of 125.1 times, monotone at every step, with no sampling noise anywhere in it because every one of those numbers is an integral.

The comparison that makes this a measurement rather than an illustration is the second series. Do the same thing with bands of the first covariate instead of bands of the score. One covariate is a perfectly good thing to condition on; it simply is not a balancing score. The residual on the second covariate reads 0.5766 at one band, 0.6621 at five, 0.6611 at ten and 0.6587 at fifty. It does not fall. Over the same range in which the score’s residual drops by a factor of 125, the covariate’s residual grows by about a seventh and then sits there — a fall of 0.88, which is a rise.

Two series that start at the same place and end two orders of magnitude apart is what “a scalar suffices” means when it is counted. The reason the second series is flat is worth saying plainly: the first covariate knows nothing about the part of the assignment that the second covariate drives, so however finely it is cut, each of its bands still contains units assigned at wildly different probabilities. The score is the only function of the covariates that has been built to contain all of it.

A score that never saw the second covariate does not balance less #

Every application of this arithmetic uses a fitted score, not a known one, and the fit can be wrong. The natural expectation — the one this field opened expecting to confirm — is that a misspecified score removes some of the imbalance and leaves the rest. It does not.

The misspecification measured here is the plainest kind: a logistic model in the first covariate alone, fitted on the population itself rather than on any sample, so that what it leaves behind is a property of the misspecification and carries no sampling noise. Its coefficients come out at 0.4493 and 0.8995, which is the best a curve in $x_1$ can do at tracking a surface that depends on $x_2$ as well.

Weighted by that score, the first covariate is balanced to 0.0035 — the model was given that covariate and it uses it. The second covariate goes to 0.7057, against the 0.6015 it started at. Weighting has made the imbalance on the omitted covariate larger than doing nothing at all.

The mechanism is not mysterious once the direction is followed through. A treated unit that the short model thinks unlikely to be treated gets a large weight; the units the short model is wrong about in that direction are exactly the ones high on the covariate it cannot see. So the treated arm’s weighting pushes its average on the second covariate up. In the control arm the same score assigns large weights to units it thinks unlikely to be untreated, which are low on the hidden covariate, pushing that arm’s average down. The two moves are in opposite directions on the same axis, so they add rather than cancel, and the gap widens.

This is a specific and useful thing to know about a specific and common failure. “Adjusting for what is available” is not a partial version of adjusting for what matters: it can be worse than nothing on precisely the covariate that was left out. The essay that priced adjusting for every covariate on hand across four thousand random structures found the same shape from the other side, and the essay that ran one regression through three causal structures found the arithmetic identical and the answer right in one of the three.

In a sample nothing is exact, and the fitted weights balance best #

Everything above is a population statement. A sample of six hundred rows has its own imbalance, which no population identity can remove, so the question in a sample is not whether the residual is zero but how large it is.

Counted over six hundred draws, the arms start 0.8343 apart on the first covariate and 0.6038 apart on the second — which is the independent route to the 0.8310 and 0.6015 above, computed from samples rather than from an integral and agreeing with it to about half a per cent. That agreement is the check that the grid and the sampler describe the same world.

Weighting by the true propensity leaves a root-mean-square residual of 0.1396 and 0.1310. That is the size of a sampling error and not of an imbalance: about $1/\sqrt{n}$, which is what is left once the systematic part has gone.

And then the reading nobody plans for. Weighting by a propensity fitted from the same sample leaves 0.0837 and 0.0733 — a little over half of what the true weights leave, on both covariates, over the same draws. The weights that balance a sample best are not the true weights.

The reason is in the fitting rather than in the weighting. A logistic regression fitted by maximum likelihood sets its own score equations to zero, and for this model the score equation on a covariate is the sum over units of the treatment assigned less the probability the fit says it was assigned with, times that covariate. Setting it to zero is, arithmetically, setting the draw’s own covariate imbalance to zero. The true propensity was never asked to do that; it is right about the population and knows nothing about which units this particular draw happened to assign. The fitted score has absorbed the accident, and what is left over is smaller.

That is a fact about balance here. Read as a fact about variance it is the strongest result in this field, and it has an essay of its own to decompose it, where the same phenomenon comes back as a variance ratio below one half and a projection that says exactly what was absorbed.

The misspecified score’s sample column completes the picture: 0.0487 on the covariate it was given — better than either correct model, because it has one fewer constraint to satisfy and spends all of its freedom on that one covariate — and 0.7143 on the covariate it was not. A model can be simultaneously the best-balancing model on the table and the worst one on the table, depending on which column is read, and a diagnostic that reports only the covariates the model contains will report success.

Balance is the cheap half of this #

Nothing above cost anything. The weights removed the imbalance exactly, the removal held whatever the assignment rule was, and no assumption beyond knowing $e(x)$ was used. A reader who stopped here would take away that weighting solves the problem it addresses, which is true and is about a quarter of the story.

What it charges is observations. A weight of one over a small probability is a single row standing in for many, and a weighted average in which one row stands in for many has the variance of a much smaller sample. Kish’s effective sample size prices that, and the price is not gentle: the essay that counts what a weight leaves finds a treated arm worth 94% of itself where the assignment is nearly a toss-up and a fraction of that where it is not.

The charge has no upper bound, and the reason it has none is that the identity above needs every unit to have had some chance of either treatment. Where that nearly fails, the weights are enormous, one observation can own most of the answer, and the estimator’s spread grows without any of its exactness being lost — the essay on the region with no comparison measures that spread quadrupling across a single sweep while the procedure remains, in the population, exactly unbiased.

What the balance cannot check #

There is a second thing a balance table does not settle, and it is larger than the first.

Every number in this essay assumes that the treatment is as good as randomly assigned once the two recorded covariates are held fixed. Unconfoundedness is not a property the data can report on: a third covariate driving both assignment and outcome would leave every column above exactly where it is, since the arms would still be balanced on everything recorded. A perfectly balanced weighted comparison in an observational study is a comparison that has succeeded at the part that can be checked.

And even within the recorded covariates, the balance a fitted score achieves is a balance in its own model. The misspecified column is the demonstration: a diagnostic run on the first covariate alone reports 0.0035 and would be read as a success. What makes the failure visible is having a covariate the model was not given and looking at it — which in an application is precisely the covariate nobody has.

The estimator built from a wrong nuisance model does not have to be helpless, though, and the field’s fourth reading is about what a second model buys.

That is the essay on what having two models buys, and its finding is narrower than the theorem it tests.

Where this stops being a measurement #

Three choices in the population above are doing work, and each of them was made so that a claim could fail.

Two covariates rather than one. With a single covariate the propensity is a monotone relabelling of it, bands of the score are bands of the covariate, and the balancing-score claim is true by construction and says nothing. The whole of the strata comparison exists because there is a second covariate for the score to have to carry.

An assignment rule that is not centred. The intercept is 0.5 rather than zero, so the treated share is above a half and the region where the assignment is nearly decidable is not symmetric. A symmetric rule would make several of the later readings vanish by symmetry rather than by fact.

An effect that varies with the first covariate. The effect is $1 + 0.8x_1$, averaging to exactly one. A constant effect would make every population an estimator could be about have the same average, which removes the distinction the field’s third essay is built on. Treatment-effect heterogeneity is what makes “which population” a question with an answer.

What has not been measured here is what a covariate that is not normal would do. The balancing-score identity is exact whatever the covariate law — nothing in the cancellation above used normality — but every band of the score in this population is a band of a normal linear predictor, and a skewed or discrete second covariate would leave the exact balance untouched while moving the integrals that price it. The essay on balancing a skewed covariate does the corresponding measurement for a covariate-adaptive assignment rule and finds the rule’s guarantee intact and its cost changed; the same question here has machinery waiting for it and no answer yet.

And the comparison this whole field is an alternative to remains the honest baseline. Assigning by a coin does not deliver balance either — it delivers a known reference distribution, which is a different and better thing — and arranging the units in pairs before assigning them removes a share of the variance that is knowable in advance. Weighting removes an imbalance that has already happened, exactly, at a price that has to be counted; the designs remove it beforehand at a price that is fixed. Neither of those sentences is available without the arithmetic that says what “exactly” and “price” mean, and the price is what the rest of this field measures.