Normal Distribution
nothing is called 95% until it has been counted
Concept

Treatment effect heterogeneity — where it appears

That a treatment's effect differs from unit to unit rather than being one number. It is what makes the population an estimate averages over part of the answer, so two procedures reporting different averages need not disagree about anything.

Named by 7 essays across 3 fields — each of them below, with the objects they name alongside it.

A weight that balances, and one that unbalances. The standardised difference between the arms on each covariate, integrated over the population rather than counted in a sample. Unweighted, the arms differ by 0.8310 on the first covariate and 0.6015 on the second, which is what makes the raw difference of arm means 2.7102 against a true average effect of 1.0000. Weighting each unit by one over its own assignment probability removes both differences exactly — -2.78e-17 and -5.69e-19, which is machine precision and not a small number — because the weighted density of the treated arm is the population's own whatever the propensity is. Weighting by a score fitted without the second covariate balances the first to 0.0035 and pushes the second out to 0.7057, further apart than doing nothing.

A score that balances

Weighting each unit by one over its own assignment probability drives the standardised difference between the arms from 0.8310 to 2.8×10⁻¹⁷ — exactly, not nearly. A score fitted without the second covariate leaves that covariate at 0.7057, further apart than doing nothing at all.

weights · Weighting The estimator has no upper bound on what it costs. What a thinning overlap does to a stabilised inverse-probability estimate of an average effect of 1.0000, over 600 samples of 600 at each of six settings. The spread rises from 0.1965 to 0.8122 and the root mean square error from 0.1964 to 0.9219, so at the thin end the error is very nearly the whole of the quantity being estimated. The lower line is the share of the arm's weighted total the single largest observation owns, averaged over the same draws: 0.69% to 9.78%, and in the worst single draw of the sweep 82.75%. Coverage of the 95% interval goes from 93.7% to 55.5%.

The region with no comparison

A trimmed interval covers the average effect over everybody 90.8% of the time at six hundred rows and 41.0% at nine thousand six hundred, while covering the average effect over the units it kept 94.3% and 96.0% throughout. An interval that gets worse as the sample grows is an interval about something else.

weights · Weighting Four numbers, and only one of them is the question. Four quantities in a population where the effect is not the same for everybody: 40.0% compliers, 25.0% always-takers and 35.0% never-takers, with the compliers carrying an effect 1.0000 larger than everybody else's. The population average effect is 0.5000. The compliers' average effect is 1.1000. A valid instrument converges on 1.1000 — the second of those, not the first, and the two differ by 0.6000. Comparing the treated with the untreated as they stand gives 1.4182, wrong by 0.9182 in the same direction, because always-takers start 1.5000 above never-takers before any treatment happens. The instrument removes the selection and changes the question at the same time, and only one of those is reported.

Whose effect it is

With a perfectly valid instrument and no violation of anything, the estimate converges on 1.1000 where the population average effect is 0.5000. The gap is exactly θ(1 − p_c), the always-takers and never-takers cancel out of both halves of the ratio, and five per cent defiers move the answer to 1.2667.

instrument · Exclusion An exact test rejecting a true hypothesis a fifth of the time. How often each analysis reports an effect when the average treatment effect is exactly zero and the effect varies between units, at 150 units with 25% treated. The permutation test on the difference in means reads 4.20% where the effect is constant — where the two nulls coincide and its exactness applies — and 22.93% where the effect varies with a standard deviation of 3. The same test on the studentised difference reads 6.27% there, and the ordinary large-sample t, which makes no exactness claim at all, reads 6.60%.

The null the exactness is for

A permutation test is exact under the hypothesis that the treatment changed nothing for anybody. Under the hypothesis it changed nothing on average, with a quarter of the units treated and the effect varying between them, it rejects a true null 22.93% of the time.

exact · Nuisance One statistic that is right under both hypotheses. Rejection rates for both statistics under both nulls, at 25% of 150 units treated, with the weak-null readings taken at an effect spread of 3. The difference in means is exact under the sharp null and rejects 22.93% of true weak nulls. The studentised difference is exact under the sharp null — 4.07% — and reads 6.27% under the weak one. The repair is a change of statistic inside the same construction: the same re-randomisations, the same fixed outcomes, a different number compared across them.

A statistic that is exact twice

Dividing the difference in means by its own separate-variance standard error before permuting takes the rejection rate under a true weak null from 20.47% to 6.07%, keeps the exactness under the sharp null at 4.07%, and costs 0.8 points of power against a real effect. At an even split it changes nothing at all, in every draw.

exact · Nuisance How often four permutation tests of a covariate-adjusted effect reject a true weak null, effect varying with the outcome, 25% of 150 treated. At an effect spread of 0, 0.5, 1, 1.5, 2: the adjusted coefficient's permutation test rejects 5.9%, 10.8%, 13.2%, 16.6%, 14.8%; over its sandwich standard error, 5.7%, 5.0%, 3.4%, 3.4%, 3.0%; the coefficient with a treatment-by-covariate interaction, 5.1%, 10.2%, 12.8%, 14.7%, 14.0%; the same over its sandwich standard error, 5.7%, 5.2%, 5.2%, 5.1%, 3.9%.

The repair, with a covariate in it

Studentising a permutation test's statistic made the difference in means exact under the sharp null and nearly right under the weak one. Carried to the covariate-adjusted estimate a trial actually reports, the repair works only if the adjustment includes the treatment-by-covariate interaction. With a quarter of 150 units treated and effects that vary with the outcome, the adjusted estimate's permutation test rejects 14.8% of true weak nulls; over its sandwich standard error, 3.0%; the interacted estimate over its sandwich, 3.9%. When the effect varies with the covariate itself, even an even split does not protect: the adjusted test rejects none of a thousand true weak nulls, and at an average effect of 0.2 its power is 5.2% against the interacted, studentised test's 40.7%.

exact · Nuisance How often an adjusted permutation test rejects a true sharp null when the covariate is chosen after randomisation, held fixed or re-chosen inside every shuffle. 150 units, half treated, six baseline covariates of which three are prognostic; 2,000 experiments and 99 re-randomisations each, read on the adjusted coefficient over its sandwich standard error. Choosing the covariate most correlated with the outcome: 5.0% held fixed, 5.0% re-chosen. Choosing the covariate with the smallest residual variance: 5.5% held fixed, 5.3% re-chosen. Choosing the covariate most imbalanced between the arms: 3.0% held fixed, 4.8% re-chosen. Choosing the covariate with the smallest adjusted p-value: 10.4% held fixed, 5.1% re-chosen. Naming the most prognostic covariate in advance: 5.5%.

The covariate chosen afterwards

A trial that chooses which baseline covariate to adjust for after randomisation has two ways to run its permutation test: hold the chosen covariate fixed across the shuffles, which is what an analysis that reports adjusting for it actually does, or re-make the choice inside every shuffle, which is exact by construction. Holding it fixed, a choice made on significance rejects 10.4% of true sharp nulls, a choice made on imbalance only 3.0% — and pays seven and a half points of power for it — while a choice that never reads the arm labels is the same test either way. The interacted, studentised statistic keeps its weak-null behaviour only when the covariate chosen is the one the effect varies with.

exact · Nuisance

Named alongside it

The objects these essays reach for when they reach for this one.

Average treatment effectExact testPermutation testRandomisation testSharp nullStatistical powerAllocation ratioAnalysis of covarianceCovariate adjustmentEstimandHeteroskedasticityHeteroskedasticity-consistent

All concepts