The estimated weight is the better one

On the population this field is measured in, the probability each unit had of being treated is known exactly. So the estimate can be built two ways on the same sample: weighting by the true propensity, or fitting a logistic model to the treatment indicator and weighting by that instead. One of those uses the truth and the other uses a noisy estimate of it.

Over twelve hundred paired draws, the variance of the estimate built on the fitted propensity is 0.034446 and the variance of the estimate built on the true one is 0.072230. A ratio of 0.4769. Estimating a quantity that was already known has halved the variance of the answer.

Neither estimator is meaningfully biased — 0.0197 and 0.0202 against a true effect of one — so this is not a trade of one property for another. It is a variance, cut in half, by replacing an exact input with an approximation to it.

The result is not a curiosity of one setting. Across five settings of the assignment rule, at five hundred paired draws each, the ratio reads 0.2783, 0.4896, 0.7361, 0.7556 and 0.7200 — below one everywhere, and below one for the unstabilised estimator too. The reading at strength 1 in that sweep is 0.4896 rather than the 0.4769 above because it is a different call at a different number of draws; the two are the same quantity measured twice, and they agree to about a per cent of the ratio.

What it is not #

Three ordinary explanations are available and none of them survives contact with the numbers.

It is not bias traded for variance. Both biases are about 0.02 on an effect of one, and they are the same to within a thousandth of each other. Nothing has been given up.

It is not the pairing. Both estimators are computed on the same draw — the same units, the same covariates, the same assignment — so the ratio is a comparison of two procedures on one experiment rather than a comparison of two experiments. Pairing removes noise from the comparison; it cannot create a difference that is not there, and it does not favour either side.

It is not stabilisation. The stabilised estimator — the Hájek form — divides each arm’s weighted total by its own summed weights, and the unstabilised Horvitz–Thompson form divides by the sample size; and the fitted weights win under both: 0.4769 for the stabilised and 0.5285 for the unstabilised. That second number is against the guess, and worth saying so. The obvious story is that the unstabilised estimator, having more chance imbalance to suffer from, should gain more from an input that absorbs it. It gains less.

The reason that guess is wrong is worth following, because it says what stabilisation already does. An unstabilised estimator’s variance contains a term that comes purely from its weights failing to sum to what they should — the arm’s weighted total overshooting or undershooting the sample size — and dividing by the realised sum removes that term outright. So the stabilised estimator has already had one component of the chance imbalance taken out of it by arithmetic, and what remains is the component a fitted score can absorb. The unstabilised estimator carries both, and the fitted score can only reach one of them, so the share it removes is smaller even though the quantity removed is larger. Across the overlap sweep the unstabilised ratio reads 0.2995 at the widest setting and 0.4934 at the thinnest, below one at every setting like the stabilised one, and moving about a good deal more between them.

What the fitted score does that the true one does not #

The mechanism is in the fitting rather than in the weighting, and it is one line.

A logistic regression fitted by maximum likelihood is defined by setting its score equations to zero. For a propensity model with covariates $x_i$, that condition is

$$\frac{1}{n}\sum_i \left(T_i - \hat e_i\right) x_i = 0,$$

where $T_i$ is the treatment the unit received. Read what that says. The treatment assigned, less the probability the fitted model says it was assigned with, summed against every covariate, is exactly zero. A fitted propensity is by construction the value that sets this draw’s own covariate imbalance to nothing.

The true propensity does no such thing. It is right about the population, and it knows nothing whatever about which units this draw happened to assign. Put the same quantity together with the true propensities and it is not zero — it is a random vector, mean zero across draws and not zero in any of them, and it is the accident the draw actually committed.

That accident is not new to this field. It is exactly what the essay on the balancing score measures in the language of residual imbalance: over six hundred draws the true weights leave a root-mean-square standardised difference of 0.1396 and 0.1310, and the fitted weights leave 0.0837 and 0.0733.

The same fact, twice: fitted weights balance the sample better, and fitted weights give a smaller variance. What remains is to show that those two sentences are one sentence rather than two coincidences.

The decomposition, which is the whole argument #

An assertion that the gain “is” the absorbed imbalance is worth nothing unless the absorbed imbalance can be measured and subtracted. It can.

Across the twelve hundred draws, take the true-weight estimator’s error and regress it, by least squares, on the draw’s own score — the vector above, computed with the true propensities, one component per column of the design. This is a regression across draws, not within a sample: each draw contributes one error and one score vector.

That regression explains 56.33% of the true-weight estimator’s variance and leaves a residual variance of 0.031543. The estimated-weight estimator’s own variance is 0.034446. Those two numbers are the argument: what is left of the true-weight estimator once the draw’s chance imbalance is projected out is, to within nine per cent, what the estimated-weight estimator has.

The same regression run on the estimated-weight estimator explains 0.05% of its variance. Not a small share — no share. There is nothing left in that estimator for the draw’s imbalance to predict, because the fitted score has already used it up, which is what the score equation above says in one line and what this measurement says in two routes.

And the two routes really are two. The variance ratio 0.4769 is measured directly, from two columns of numbers. The projection predicts a ratio of $1 - R^2 =$ 0.4367 if the whole of the gain is absorbed imbalance and nothing else. Those are computed from different things — one from a pair of variances, one from a regression’s residual — and they agree to 91.6%.

What that regression could have been, and was not #

An $R^2$ of 56.33% appearing exactly where an argument needs one is the kind of number to attack rather than accept, and there are two ordinary ways to manufacture it.

The first is overfitting. A regression with enough columns explains anything. This one has three — an intercept and one component of the score per covariate — fitted across twelve hundred draws. Under a null of no relationship at all, three columns on twelve hundred rows would explain about a quarter of one per cent of the variance by chance. The observed 56.33% is two hundred times that, and the control is in the same table: the identical regression, with the identical three columns on the identical twelve hundred rows, explains 0.05% of the estimated-weight estimator. That is below what three free columns pick up by accident, which is what a genuinely exhausted signal looks like, and it is not a number a fitting artefact can produce on one column and not the other.

The second is that the score might be a proxy for something else the draw contains — its sample size, its treated share, the covariate means themselves. It is those things, in the sense that the score is built out of them; that is the point rather than a confound. What matters is the direction of the claim, which is not “the error is predictable” but “the error is predictable from exactly the quantity the fitted score sets to zero, and the fitted estimator’s error is not”. A different summary of the draw would explain some other share and would say nothing about why one estimator beats the other. The score is not a proxy chosen because it worked; it is the quantity the likelihood equation names, written down before the regression was run.

The third possibility, that the whole effect is an artefact of the stabilisation, is closed off by the unstabilised column. Weighting each arm’s total by the sample size rather than by its own summed weights gives variances of 0.066054 with true weights and 0.034909 with fitted ones — the same halving, through an estimator whose denominator is not random at all.

Why the agreement is from below, and why that is forced #

The gap between 0.4367 and 0.4769 is not slack in the argument; its direction is determined in advance, and that is what makes the check a check.

A least-squares projection produces the smallest residual variance any linear correction along those directions can achieve. It is an optimum, not a rival procedure. The fitted propensity is a particular nonlinear function of the same imbalance — it solves a likelihood equation rather than minimising a squared error — so it can approach that floor and it cannot beat it.

So the prediction was never that the two numbers would be equal. It was that the residual would be at or below the estimated-weight variance, and that the ratio of the two would be near one from beneath. It is 91.6%. Had it come out above one, the claim that the fitted weights win by absorbing the imbalance would have been refused — the fitted weights would have been doing better than the best available linear correction, which means they would be doing something else, and the essay would be about what.

That refusal is the shape every claim in this field is built on. The assertion that the true weights are the better ones is written down, fed the data, and rejected. An assertion that could not have been rejected would be a restatement of the code rather than a test of it, which is the same reason the essay separating a plug-in from a maximiser computes both rather than assuming they coincide.

The gain shrinks exactly where the imbalance stops being the problem #

If the gain is absorbed imbalance, then it must fall wherever the variance stops being made of imbalance. It does, and this is the sweep that turns a result into a claim.

The share of the true-weight estimator’s variance predictable from the draw’s own imbalance falls across the overlap sweep: 73.6%, 53.3%, 32.6%, 28.4% and 27.1%. And the variance ratio rises with it — 0.2783, 0.4896, 0.7361, 0.7556, 0.7200 — so where three quarters of the variance is chance imbalance the fitted weights remove nearly three quarters of it, and where a quarter is, they remove a quarter.

The absolute numbers are worth reading beside the ratio, because a ratio hides which end is moving. At the widest setting the true-weight estimator has a variance of 0.035496 and the fitted-weight one 0.009880 — the fitted version is nearly four times more precise, on an estimator that was already the better-behaved of the two. At the thinnest setting measured both variances are more than ten times larger, and the fitted version’s advantage has narrowed to about a quarter. The gain is not a fixed discount; it is a fixed subtraction, of a quantity that is most of the variance at one end of the sweep and a minority of it at the other.

What the variance is made of instead, at the thin end, is a handful of enormous weights — one observation owning a large share of an arm, which no correction to the score’s location can do anything about.

One reading in that sweep is not monotone: the ratio goes 0.7556 at strength 2 and back to 0.7200 at 2.5, and the unstabilised version moves about more than that. Five hundred paired draws resolve a ratio near three quarters to perhaps two or three points, so those two readings are not distinguishable and the honest statement is that the ratio flattens out around three quarters rather than continuing to rise. What is distinguishable is the fall from 0.2783 to about 0.75 across the sweep, which is a factor of nearly three and the thing being claimed. The essay on the region with no comparison is where the other half of that variance is priced, and nothing in this essay touches it.

The interval built for a known weight is the wrong interval #

There is a practical consequence, and it is the reason this result matters to somebody who never intends to compare two weight sources.

The standard error usually applied to an inverse-probability estimate is derived treating the weights as known. Apply it to estimated weights and it is describing a larger variance than the estimator has. Counted here, the interval built that way covers 93.1% when the weights really are the true ones — which is about right — and 99.0% when they were estimated.

The same over-coverage appears in the four-cell table of the essay on what having two models buys as 99.2%, arriving there as an anomaly in a row nobody was reading and explained here. Over-coverage is not a failure a reader will complain about, which is precisely why it survives: an interval that is forty per cent too wide reports a study as less informative than it is, and nothing in the output announces it. It is the mirror of the trimming result, where an interval that had quietly changed its target reported a study as more certain than it was. Both come from an interval whose derivation no longer describes the procedure that was run — the essay on what the 95% refers to is the general form of the complaint, and the essay on an analysis that has to know its own assignment rule is the same defect in a design rather than in an analysis.

Where this stops holding #

Three limits, and the third is the one this field would have run next.

The propensity is fitted by maximum likelihood. The whole mechanism is the score equation, so the result belongs to that estimator rather than to “estimating the propensity” in general. A score fitted by some other criterion sets some other quantity to zero and would have to be measured separately.

The effective size cannot see any of this. Kish’s number is a function of a set of weights, and the true and fitted weight sets have almost the same spread, so it prices them almost identically while their estimators differ by a factor of two.

The essay that counts what a weight leaves is where that limitation is measured. A summary of the weights is not a summary of the estimator, and the gap between them is this essay.

And the obvious next estimator is not measured here. A propensity can be fitted to balance the covariates directly rather than to fit the treatment indicator — a different estimating equation, with the same fitted values only in the limit. Such a score would set to zero exactly the residual imbalance that the balancing arithmetic reports as 0.0837 and 0.0733, rather than setting the likelihood’s score to zero and getting the balance as a consequence. Whether that wins by more than the maximum likelihood fit does, or by less, is the question this essay leaves; a weight fitted to balance fits one on these same draws and answers it.

The comparison worth keeping is with the alternative that has no weights at all. Arranging the units before assigning them removes a share of the variance that is knowable in advance, and assigning by a coin delivers a known reference distribution rather than balance. Both of those act before the accident happens. What this essay measures is the one repair available afterwards: a score fitted to the accident removes the part of it that is a location, and leaves the part that is a handful of units nobody has a comparison for.