A threshold in the tail

A rule that balances a covariate’s mean removes part of the imbalance in any function of that covariate and leaves the rest, and how much of each is not a matter of simulation. It is a correlation, it has a closed form, and at one point it is a constant this site has already met from the other side.

The identity #

The statement below is about a perfectly balanced covariate, which no rule achieves and which is the right idealisation anyway: it is the ceiling every balancing rule is working towards, so a number computed at it is an upper bound on what any of them can be worth.

Suppose a rule balances x exactly, and the outcome depends on g(x). Write the imbalance in g as its projection onto x plus the part orthogonal to it. The rule removes the first and cannot touch the second, so

$$\operatorname{var}(\text{imbalance in } g) = (1 - \rho^2)\times\operatorname{var}(\text{imbalance in } g \text{ under a coin}), \qquad \rho = \operatorname{corr}(x, g(x)).$$

For a threshold, g(x) = 1{x > c} on a standard normal covariate, that correlation is available in one line. The covariance of x with the indicator is φ©, the density at the cut; the indicator’s standard deviation is √(p(1 − p)) with p = 1 − Φ©. So

$$\rho(c) = \frac{\varphi(c)}{\sqrt{p(1-p)}}.$$

Everything in this essay is that expression evaluated somewhere.

Where the density comes from #

The covariance is worth deriving rather than quoting, because the fact that it is the density at the cut is the whole explanation of the collapse in the tail.

$$\operatorname{cov}(x, \mathbf{1}\{x > c\}) = \mathbb{E}[x\,\mathbf{1}\{x>c\}] = \int_c^\infty x\,\varphi(x)\,dx = \varphi(c),$$

because xφ(x) is the derivative of −φ(x). So the numerator of the correlation is the height of the normal curve at the cut, and the denominator is the standard deviation of a coin with probability p = 1 − Φ©.

Both fall as the cut moves out, and the ratio is what decides the answer: the density falls like exp(−c²/2) and the indicator’s standard deviation like √p, which for the normal falls like √(φ©/c). The density wins, so the correlation falls — and it falls slowly at first, since ρ is 0.798 at the median and still 0.762 at half a standard deviation, and then quickly.

The shape of that decline is the shape of the whole field. A rule that balances a covariate’s mean protects against structure in the middle of the covariate’s range and stops protecting against structure at its edges, and the crossover is around one standard deviation, where it is doing about half the job.

The median, where it is 2/π #

At c = 0 the density is φ(0) = 0.3989 and p(1 − p) = ¼, so ρ = 2φ(0) = √(2/π) and

$$\rho^2 = \frac{2}{\pi} = 0.636620\ldots$$

Balancing the covariate’s mean removes exactly 2/π of the imbalance in a median split of it, and leaves 0.3634.

That constant has appeared on this site before, in the other direction. A median split of a normal covariate can see exactly 2/π of it: the two halves have means ±√(2/π), the between-category variance is 2/π, and a rule that balanced the two categories perfectly would still leave √(1 − 2/π) = 0.6028 of a coin’s imbalance in the number itself.

The two statements are one integral read in opposite directions. A cut sees 2/π of the number, and the number removes 2/π of the cut. Neither is an approximation and neither depends on the sample size, the rule, or anything but the normality of the covariate — and the symmetry between them is the reason the pair of essays exists.

Away from the median it collapses #

The interesting half is what happens as the cut moves. A threshold in the tail is a rarer event; its variance is smaller, its covariance with x is smaller, and the two do not shrink at the same rate.

At a threshold two standard deviations out — an eligibility rule that 2.3% of units meet, a clinical cutoff, a legal limit — balancing the covariate’s mean removes 13.1% of the imbalance and leaves 86.9%. The design has done almost nothing about the only feature of the covariate the outcome uses, and the balance table will show the covariate balanced to a hundredth.

The curve is symmetric about the median, and the symmetry is worth one sentence because it says the result is about rarity and not about direction. A threshold at −1.5 — a rule that 93.3% of units satisfy — is exactly as poorly protected as one at +1.5, since a rare event and its rare complement have the same variance and the same covariance in absolute value. What a mean-balancing rule cannot see is any feature carried by a small fraction of the units, at either end.

Several cuts, and the same integral again #

The comparison with categorising is worth making because the two approaches to a continuous covariate now have a common measure.

A rule that balances k equal-probability bins of the covariate can see Σpᵢμᵢ² of it, which is 2/π at two bins, 0.793 at three, 0.861 at four and 0.945 at eight — the share a cut throws away falling slowly with the number of categories. A rule that balances the number sees all of the number by construction, and a share ρ²© of any threshold in it.

So the two are both projections, onto a step function in one case and onto the identity in the other, and neither dominates: a rule with eight bins sees 92.5% of the covariate and would see a threshold at 2 much better than the mean-balancing rule does, because one of its cut points is near it. The choice between them is a choice about where the outcome’s structure sits, which is exactly the quantity nobody has — and it is the argument for reading more than one function of the covariate rather than choosing between two.

Two routes, and where they part company #

The identity above assumes the rule balances x exactly and that what is left behaves like a coin’s imbalance. Neither is quite true in a trial of two hundred units, so the closed form is checked against counted trials.

At n = 800, with the rule at full determinism, the counted share left is 0.3865 where the closed form says 0.3634, 0.5415 against 0.5614, and 0.7287 against 0.7309 — agreement to about two points at three thresholds, which is the noise of a ratio of two variances estimated over twelve hundred trials each.

At n = 200 the counted shares are 0.4182, 0.5765 and 0.7792: consistently above the closed form, by four points at the median and five in the tail. That gap is a finite-sample effect with a cause: the rule balances the sample mean of x rather than the population mean, so the projection it removes is estimated rather than known, and estimating it costs a degree of freedom out of the n the trial has. The closed form is the limit and it is optimistic at any real trial size.

That is the honest form of two routes agreeing. They agree in the limit; at a real trial size the closed form understates what survives, in a direction that can be explained and not merely observed.

How far into the tail, in one expression #

The table stops at two standard deviations and the expression does not, and continuing it says something the five rows only hint at.

Far out in the tail p is small, so p(1 − p) ≈ p, and the normal tail satisfies p ≈ φ©/c. Substituting both into ρ² = φ©²/(p(1 − p)) leaves

ρ²© ≈ c·φ©

— the number of standard deviations out, times the height of the density there. It is an approximation and it tightens as the cut moves: at c = 2 it gives 0.108 against the exact 0.131, at c = 3 it gives 0.0133 against 0.0146, and at c = 4 it gives 0.000535 against 0.000566.

What that expression makes obvious is the rate. φ© falls like exp(−c²/2), so the share a mean-balancing rule removes falls at a Gaussian rate, not a polynomial one. Two more rows of the table, computed exactly rather than extrapolated, are 1.46% at a threshold of 3 — met by about one unit in seven hundred and fifty — and 0.057% at 4.

So the honest generalisation is stronger than the table’s last row. Beyond about two and a half standard deviations, balancing a covariate’s mean does nothing at all for a threshold in it: not a little, not a quarter, but a fraction of a per cent. An eligibility criterion that a hundredth of the population meets, a safety limit, a rare-event endpoint — for any of those, a covariate balanced to machine precision leaves the imbalance that matters exactly where a coin would have left it.

That also disposes of a tempting middle position. Since ρ² does not merely decline but collapses, there is no regime in which a mean-balancing rule is partly adequate for a rare threshold and can be supplemented; there is a regime where it does most of the job and a regime where it does none, with the transition inside one standard deviation of the median.

The half-way threshold is one unit in five #

Where exactly the transition sits is worth pinning down, since the table brackets it and does not name it.

Setting ρ²© = ½ and solving gives c = 0.81, where 20.9% of units lie above the cut. At that threshold a perfectly balanced covariate removes half the imbalance in the indicator and leaves half.

One unit in five is a more useful landmark than “around one standard deviation”, because it is stated in the currency an experimenter has. Nobody knows where the outcome’s structure sits in standard deviations; everybody knows roughly what fraction of their units meet a criterion. If the feature that matters is carried by more than a fifth of the units, the rule is doing most of its job; if by fewer, it is doing less than half.

The symmetry noted above extends the landmark to both ends without any further arithmetic: a criterion met by four units in five is exactly as poorly served as one met by one in five, since a rare event and its rare complement share a variance and a covariance in absolute value. The quantity that decides is the smaller of the two shares, and a balancing rule protects against it in proportion to how close that share is to a half.

What it would take to do better #

The identity also says what a repair would have to be, which is a shorter list than it looks.

Nothing about the rule can help. The 1 − ρ² is what survives when x is balanced perfectly, so a better optimiser, a more deterministic rule or a larger trial cannot get underneath it: at a threshold of 2 an experimenter who balanced the covariate to machine precision would still carry 86.9% of a coin’s imbalance in the thing that matters.

Nothing about the analysis can undo it either, in the sense that matters here. An analysis that adjusts for the right function recovers the precision — that is the last essay of this field — but it recovers it from a trial that was assigned as though the imbalance did not exist, and an adjustment cannot recover information the design threw away when it was still cheap to keep.

What can help is telling the rule about the function. If the rule balances the indicator itself, the indicator’s imbalance is what it removes, and the closed form no longer applies because x is no longer the only thing being balanced. That is one line of change to a rule and it is the subject of the next essay, along with what it costs when the guess about the shape is wrong.

What the curve is for #

The practical content is that the protection a balancing rule offers depends on where in the covariate the outcome’s structure lives, and the dependence is steep.

An experimenter balancing age has in mind that age matters. If age matters linearly, the rule removes essentially all of the imbalance that matters. If what actually matters is being over 75 — 6.7% of a normally distributed population two and a half standard deviations up from a mean of 55 with a spread of 10 — the rule removes about a quarter of it. The design is doing a quarter of the job it appears to be doing and there is nothing in the trial to say so.

And the curve says something about the choice of cut too. A rule that balances a median split is balancing the function with the highest possible correlation with the covariate, which is why the two approaches to a continuous covariate — cut it and balance the halves, or read the number — agree better at the median than anywhere else. Both are optimising for the middle of the distribution, and tails are where they part company.

What happens to the estimate #

The imbalance is the mechanism; the variance of the treatment estimate is what an experimenter actually pays. The two are not the same number, because the estimate’s variance is the outcome’s noise plus whatever the imbalance carries into it.

For the outcomes measured in the previous essay — where the covariate’s contribution and the residual noise are of the same size — the rule that reads the number takes the estimate’s variance to 0.476 of a coin’s against a linear covariate and 0.755 against a threshold at 1. The imbalance ratios behind those are 0.016 and 0.614.

The dilution is worth stating in general terms: a rule that removes a share s of the imbalance in the function that matters reduces the estimate’s variance by s times the share of that variance which the imbalance is responsible for. So the closed form above is an upper bound on what any balancing rule can be worth for a threshold outcome, and every real trial is below it because the outcome has noise in it.

What is claimed here, and what is not #

This essay takes the exact share of a threshold’s imbalance that a balanced covariate removes, and the claims are the closed form, its value of 2/π at the median, its collapse in the tail, and its agreement with counted trials in the limit.

What stays out and is named as a decision: covariates that are not normal, where the correlation is still the right object and φ©/√(p(1−p)) is not the right formula — the general expression is a ratio of two integrals and is computed numerically in this field’s library, but nothing is claimed about which distributions matter in practice; several thresholds at once, where the projection is onto a space rather than a line; and the exact finite-sample correction for the gap measured above, which would be an expansion in 1/n and is named rather than derived.

The boundary against the field before this one is the covariate. That a median split can see 2/π of a normal covariate, and that a rule which balances categories leaves 0.6028 of a coin’s imbalance in the number, are established there. This is the same integral, and its appearance in both directions is the reason to state it as an identity rather than as a coincidence.

The checks, and the refusals that make them mean something #

Two claims are gated in this field’s library, and the first is checked to machine precision because it is an identity rather than a measurement: the removed share at the median is required to equal 2/π to within 10⁻¹², and to equal the between-category share a median split sees, which is the same constant arriving from the other essay’s arithmetic. A numerical integration of the correlation, which shares no line of code with the formula, is required to agree to six decimals.

The second is the two-routes check at three thresholds: the counted residual is required to be within six points of 1 − ρ² at eight hundred units, and the closed form is required to be monotone in the threshold — the further into the tail, the more survives — which is the shape of the whole result and would fail if the correlation had been computed wrongly.

The refusals for this field belong to the essays on either side of this one, and the one that bears on this essay hardest is the rule reported on the covariate it reads: a design that removes 99.4% of the imbalance in x removes 22.9% of the variance of the estimate when the outcome depends on a threshold at 1.5. The identity above is what turns that from a surprise into an arithmetic consequence.