Normal Distribution
nothing is called 95% until it has been counted
Series

Weighting — the series

7 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. A weight that balances, and one that unbalances. The standardised difference between the arms on each covariate, integrated over the population rather than counted in a sample. Unweighted, the arms differ by 0.8310 on the first covariate and 0.6015 on the second, which is what makes the raw difference of arm means 2.7102 against a true average effect of 1.0000. Weighting each unit by one over its own assignment probability removes both differences exactly — -2.78e-17 and -5.69e-19, which is machine precision and not a small number — because the weighted density of the treated arm is the population's own whatever the propensity is. Weighting by a score fitted without the second covariate balances the first to 0.0035 and pushes the second out to 0.7057, further apart than doing nothing.

    A score that balances

    Weighting each unit by one over its own assignment probability drives the standardised difference between the arms from 0.8310 to 2.8×10⁻¹⁷ — exactly, not nearly. A score fitted without the second covariate leaves that covariate at 0.7057, further apart than doing nothing at all.

    part 1 · weights
  2. Three answers to how much sample is left. What a set of inverse-probability weights leaves of the treated arm, by three routes, at six settings of the assignment rule. The integral 1/(π∫φ/e) reads the whole covariate space and falls from 0.9392 to 2.655e-3. Kish's effective size counted in samples of 600 falls only to 0.2861, because almost all of the integral's fall is in a region a sample of six hundred never draws from. And the fraction the variance of the weighted mean actually delivers is lower again — 0.1155 — because the variance is the average of one over the effective size and the effective size averaged is not the same number. At the widest overlap all three agree to 0.05%.

    How many observations a weight leaves

    Kish's effective sample size is exact — for an outcome whose mean does not move with the covariates the weights are built from, the studentised variance reads 1.0680 where the formula says one. For the population's own outcome the same reading is 6.769, rising to 52.497.

    part 2 · weights
  3. The estimator has no upper bound on what it costs. What a thinning overlap does to a stabilised inverse-probability estimate of an average effect of 1.0000, over 600 samples of 600 at each of six settings. The spread rises from 0.1965 to 0.8122 and the root mean square error from 0.1964 to 0.9219, so at the thin end the error is very nearly the whole of the quantity being estimated. The lower line is the share of the arm's weighted total the single largest observation owns, averaged over the same draws: 0.69% to 9.78%, and in the worst single draw of the sweep 82.75%. Coverage of the 95% interval goes from 93.7% to 55.5%.

    The region with no comparison

    A trimmed interval covers the average effect over everybody 90.8% of the time at six hundred rows and 41.0% at nine thousand six hundred, while covering the average effect over the units it kept 94.3% and 96.0% throughout. An interval that gets worse as the sample grows is an interval about something else.

    part 3 · weights
  4. Either model is enough; neither is not. The bias of three estimators of an average effect of 1.0000, over 600 samples of 600 units with the assignment rule at strength 1, in each of the four cells made by getting each nuisance model right or wrong. The wrong model in both cases is one that omits the second covariate, which the outcome and the assignment both depend on. The outcome model alone is off by 0.8064 whenever it is the wrong one; weighting alone is off by 0.8190 whenever the propensity model is. The augmented estimator built from both is off by -0.0085, -0.0083 and -0.0016 in the three cells where at least one of them is right, and by 0.8118 in the fourth — which is between its two components rather than better than either.

    Either model, but not neither

    The augmented estimator's bias is −0.0085, −0.0083 and −0.0016 wherever one nuisance model is right, against components off by 0.8064 and 0.8190. One step past the overlap sweep it is the least biased estimator on the table at 0.0857 and the worst on it at 1.9265.

    part 4 · weights
  5. Estimating a weight you already know is worth doing. The variance of an inverse-probability estimate weighted by a propensity fitted from the sample, over the variance of the same estimate weighted by the true propensity, paired on the same 500 samples of 600 units at each of five settings. Every reading is below one: the stabilised estimator keeps 27.8% of its true-weight variance where the assignment is nearly a coin toss and 72.0% where it is nearly decidable, and the unstabilised one 30.0% and 49.3%. Neither estimator is materially biased, so this is a variance rather than a trade. The true weights are right about the population and know nothing about the draw; the fitted weights are the value that sets this draw's own imbalance to zero, and that imbalance was what the variance was made of.

    The estimated weight is the better one

    The propensity is known exactly here, so it can be weighted by — and estimating it from the same data and weighting by that gives a variance ratio of 0.4769 on paired draws. The reason is a projection: the draw's own imbalance explains 56.33% of the true-weight variance and 0.05% of the estimated-weight one.

    part 5 · weights
  6. Weights that balance a sample by construction. What three sets of weights leave of the standardised difference between the arms on each covariate, as a root mean square over 1200 samples of 600 units. The true propensity leaves 0.1317 and 0.1186 — a sampling error, since it is right about the population and knows nothing of the draw. A likelihood fit leaves 0.0770 and 0.0657, having absorbed part of the draw's imbalance as a side effect of fitting the treatment. Weights fitted so that each arm's weighted means are the sample's leave 1.4e-14 and 1.2e-14, which is the arithmetic's floor rather than a small number: the largest gap between a weighted arm mean and the sample mean in any draw is 9.8e-14.

    A weight fitted to balance

    Weights fitted so that each arm's weighted covariate means equal the sample's leave a difference of 1.4×10⁻¹⁴ between the arms and give the estimate a third of the variance of weights fitted by likelihood — 0.011883, within a relative 5.8% of the bound no estimator can beat. In the world where the assignment carries a square nobody named, the same exact balance leaves the square further apart than no weighting at all, and where the outcome carries it too the estimate is wrong by 0.6973 with an interval that covers 1.5%.

    part 6 · weights
  7. One weighting told the means and one told the second moments, in five worlds. The bias of the fit to balance over 600 samples of 600 units in each world, fitted to the covariates' means and fitted to their means, squares and product. Told the means it is off by -0.0004, -0.0020, 0.0103, 0.6973, 0.2698 in the worlds with no square, a square in the assignment, a square in the outcome, a square in both and a cube in both; told the second moments, by -0.0009, -0.0000, 0.0009, -0.0045, 0.3099. Its interval covers 94.0%, 94.7%, 95.0%, 1.5%, 51.0% and 93.7%, 89.8%, 94.0%, 91.0%, 48.3%.

    The moments a balance is told

    Weights fitted to balance the covariates' means were wrong by 0.6973 in the world where both the assignment and the outcome carry a square. Told the squares and the product as well, the same construction is off by −0.0045 there and its interval covers 91.0%. The failure moves up a moment rather than away: with a cube in both, the second-moment balance is off by 0.3099 and leaves the cube twice as far apart as no weighting. And where overlap is thin, 37.0% of samples have no such weights at all.

    part 7 · weights

All series