Eight groups, one population

Eight hospitals report their surgical mortality. The smallest did three operations, the largest did forty, and the reported rates run from well below average to well above it. The question is what to estimate for each hospital, and there are two answers available without thinking about it at all.

The two answers nobody argues for #

The first is to take each hospital at its word. Eight groups, eight estimates, each computed from that group’s own data and nothing else. It is what a spreadsheet does by default and it is called no pooling.

The second is to notice that these are eight hospitals doing the same operation, average everything, and report one number for all of them. That is complete pooling, and it is what happens whenever a summary is quoted without a breakdown.

Both are defensible in a sentence and both are wrong in a way that can be counted. The data here is generated from a stated rule — each group’s true rate is drawn from a population, then observed with noise proportional to how few observations the group has — so the truth is available and the error of each answer is a number rather than an argument.

Across three thousand such datasets, with eight groups of four observations each, the mean squared error per group is 2.23 if every group is taken at its word and 1.15 if they are all collapsed into one. Neither is close to the best available.

Why taking each group at its word is worse than it sounds #

The failure of no pooling is not that it is biased. It is not: the average of the estimates is the average of the truths, and any given group’s own mean is an unbiased estimate of that group’s own rate.

The failure is that it is noisy, and it is noisiest exactly where it matters. A hospital with three operations reports a rate computed from three operations. That number moves enormously with a single outcome, and the hospitals at the top and bottom of the resulting league table are almost always the small ones — not because small hospitals are unusually good or unusually bad, but because a small sample can land anywhere.

This is the winner’s curse in a different costume. The extremes of a set of noisy estimates are selected for noise as much as for signal, so the group that looks best is disproportionately likely to be a group that got lucky, and it will look less good next year without anything having changed.

The consequence for the estimate is measurable and is the 2.23 above: taking a three-observation mean as an estimate is paying full price for its noise.

Why one number for everybody is worse than it sounds #

Complete pooling has the opposite failure. It is not noisy — the grand mean of a hundred and seven observations is a precise number — and it is wrong for every group that genuinely differs from average.

The error it makes is the real spread between the groups, and that spread does not shrink as more data arrives. Ten thousand observations per hospital would leave complete pooling estimating exactly the same wrong number for a hospital that is genuinely better than average, because the estimate has nothing to do with that hospital’s data at all.

So the two answers fail in exactly opposite ways: one has all the variance and none of the bias, the other has all of the bias and none of the variance. Written that way it is obvious that the useful answer is somewhere between them, and the only question is where.

Both numbers are closed forms #

Neither of the two errors had to be simulated, and checking them says the generating rule is what it is described as.

No pooling estimates a group’s rate from its own four observations, so its squared error is $\sigma^2/n = 9/4$, which is 2.25, against 2.23 measured.

Complete pooling reports the grand mean for every group, so its squared error against a particular group’s truth is $\operatorname{Var}(\bar y) + \tau^2 - 2\operatorname{Cov}(\bar y, \theta_g)$. With eight groups, $\operatorname{Var}(\bar y) = (\tau^2 + \sigma^2/n)/8 = 0.406$ and the covariance is $\tau^2/8 = 0.125$, so the total is $0.406 + 1 - 0.25 = 1.156$, against 1.15 measured.

Both to three figures, from τ = 1 and σ = 3 and nothing else.

Which gives the crossing exactly, and it does not involve the number of groups #

The same two expressions say when one obvious answer beats the other, and the answer is cleaner than it looks.

Complete pooling’s error is $\tau^2(1 - 1/G) + (\sigma^2/n)/G$ and no pooling’s is $\sigma^2/n$. Setting them equal, the factor $(1 - 1/G)$ appears on both sides and cancels, leaving

$$\tau^2 = \sigma^2/n .$$

Complete pooling beats no pooling exactly when the population spread is smaller than a group’s own standard error, whatever the number of groups.

Here τ = 1 and the standard error is 1.5, so collapsing everything into one number is the better of the two obvious answers — by a factor of nearly two — and it would still be at eight hundred groups or at three.

That is worth having because the intuition runs the other way. More groups feels like more reason to report them separately; the arithmetic says the number of groups changes both errors in step and decides nothing. What decides it is a comparison between two quantities a reader can estimate before choosing: how different the groups really are, and how well any one of them is measured.

The estimate between them #

The middle estimate is a weighted average of the group’s own mean and the population’s:

estimate = B × (population mean) + (1 − B) × (the group’s own mean)

and everything in this field is a statement about B. It is not a tuning parameter, not a prior belief, and not a compromise chosen for diplomatic reasons. It is

B = se² / (se² + τ²)

where se is the standard error of the group’s own mean — what that group’s data does not know — and τ is the real spread between the groups.

Read as a ratio, it says: move to the middle in proportion to how much of what separates this group from the others is noise. A group whose own mean is mostly noise moves almost all the way; a group measured precisely stays where it is; and a set of groups that genuinely differ by a lot barely moves at all, because then the differences between them are real.

Against the same three thousand datasets, this estimate costs 0.88 where the two obvious answers cost 2.23 and 1.15. That is not a marginal improvement over the better of them — it is a quarter of the error of one and three quarters of the error of the other, from the same data, with no additional observations and no additional assumptions beyond the one that the groups are of a kind.

The weight has nothing to do with the value #

The property that makes this a statement about information rather than a nudge toward the middle: how far a group moves does not depend on where it sits.

A hospital reporting a very high rate and a hospital reporting a very low one, with the same number of operations, move by the same fraction. Nothing in B refers to the group’s estimate. What it refers to is se, which is fixed by the group’s size, and τ, which is a property of the whole set.

At the spread in the opening figure, the group of three moves 75% of the way to the population mean and the group of forty moves 18%. Those two numbers are the field in miniature: the small group is mostly borrowing, the large group is mostly reporting, and neither was told which to do.

Where τ comes from, which is the part that makes it honest #

An obvious objection: the weight requires τ, the real spread between the groups, and if that were supplied by the analyst then the whole thing would be an elaborate way of asserting an answer.

It is not supplied. It is estimated, from the only place it could come from — the observed spread between the group means, which contains both the real spread and the noise that would separate the groups even if they were identical. The real one is the difference:

τ̂² = max(0, spread between the group means − average noise)

That is a moment estimator, it is nearly unbiased for τ², and it has one property worth noticing immediately. When the groups are genuinely identical, the observed spread is on average exactly the noise, the difference is negative about half the time, and the estimate is then zero — which says pool completely, which is correct.

Estimating τ rather than knowing it is not free, and the size of the fee is worth stating plainly. At a true spread of τ = 1 the estimate that knows τ costs 0.88 and the estimate that infers it costs 1.05 — still far better than either obvious answer, and worse than the idealised version by about a fifth. A later essay is about that estimate, its downward bias, and what it means that the data is being asked to supply its own prior.

The crossing point #

The figure above has one feature worth naming, because it is the answer to “when does this matter”.

At the left, where the groups are genuinely alike, complete pooling wins outright and by a lot. At the right, where they differ wildly, it loses catastrophically — its error grows without limit while every other estimate flattens. The two obvious answers cross somewhere in the middle, and the useful observation is that the crossing point is not knowable from either of them.

An analyst committed to one of the two extremes is making a bet about where τ sits, without measuring it, and the cost of losing that bet is unbounded in one direction. The middle estimate does not make the bet: at small τ it is complete pooling, at large τ it is no pooling, and in between it is a weighted average that costs less than both.

More data per group narrows the gap, which it should: a group with plenty of its own data has less to borrow. What it does not do is make the two obvious answers converge. Complete pooling is still unboundedly wrong at large τ however much data each group has, because its error is the spread, not the noise.

Everyone benefits, when everyone is from the population #

The natural worry about pooling is that it protects the small groups by damaging the large ones, or that it flatters the middle at the expense of whoever is genuinely unusual. The first is checkable directly: the per-group error under pooling against each group’s own mean, group by group.

When all eight groups really are drawn from the same population, every one of them is estimated better by pooling than by its own mean — the large ones by a little, the small ones by a lot. Nobody pays for anybody.

That “when” is load-bearing, and it is the whole content of the assumption this field makes. The groups have to be exchangeable: knowing nothing else about them, any one could have been any other. A group that is genuinely not from the population is estimated worse by pooling — measurably, by a factor of six at six population widths out — and that failure has an essay of its own, because it is the case where the method’s confidence is least deserved.

Four names for one estimator #

The estimate above is called a random effects model, a mixed model, a multilevel model, a hierarchical model and empirical Bayes, depending on the literature. In the case here they are the same arithmetic.

The names differ because the arguments that reach the estimate differ. The random-effects account says the group parameters are draws from a distribution rather than fixed unknowns. The Bayesian account says the population distribution is a prior and the estimate is a posterior mean. The decision-theoretic account, which reaches almost the same estimator by an argument mentioning no population at all, is Stein’s — and the fact that three incompatible-sounding stories arrive at the same weighted average is the strongest evidence available that the weighted average is not an artefact of any one of them.

For a reader, the practical consequence is that software offering “random effects” and software offering “partial pooling” is offering the same thing, and that the choice worth arguing about is not which name to use but whether the groups are exchangeable.

What changes at scale #

The comparison so far uses eight groups. The number of groups matters in a specific way that is worth separating from the number of observations in each.

More observations per group makes every group’s own mean better, so B falls and the estimates move less. More groups makes τ̂ better, so the weight itself is more reliable — and with very few groups the weight is being estimated from very little, which is a real limitation rather than a technicality. Three groups do not tell much about the spread of the population they came from.

The practical floor is somewhere around five to eight groups before the estimated τ is worth trusting, and below that an analyst is choosing a weight rather than measuring one. That is not a reason to fall back to no pooling, which is a weight of zero chosen with even less evidence; it is a reason to say which weight was used and why.

What to look for in a published analysis #

A reader met with a table of group-level rates — hospitals, schools, regions, product lines, sales territories — can ask two questions that separate the careful analyses from the rest.

Are the extremes the small groups? If the top and bottom of the ranking are the groups with the fewest observations, the table is a ranking of sample sizes as much as of performance, and the analysis has done no pooling at all.

Is there a stated spread between groups? An analysis that has fitted a hierarchical model can report τ̂, and that number says how much of the visible variation between groups is real. An analysis that cannot report it has not measured the thing that decides whether the differences it is discussing exist.

Neither question requires the data, and both are answerable from a methods section.

The first question has a consequence worth spelling out, because it is where this arithmetic reaches a decision rather than a report. League tables of schools, hospitals, surgeons and police forces are published as rankings of unpooled group means, and the units at the ends of those rankings are systematically the small ones. Pooling changes the order — it moves the small groups towards the middle and leaves the large ones alone — so the ranking a hierarchical model produces is not a politer version of the raw one, it is a different ranking, and the difference is largest exactly where the raw table is least reliable.

That is worth stating as an instruction rather than a caution: a ranking of groups of very different sizes is not a ranking of the groups. It is a ranking that mixes performance with sample size, and the mixture is heaviest at the two ends, which are the only rows anybody reads.

What this field will establish #

Four essays, and the ladder is short because the subject is one idea taken seriously.

This one has been about the two obvious answers and the fact that a measured third exists, at 0.88 against 2.23 and 1.15. The next is about B itself — that it is exactly a posterior mean, that the same estimator arrives from a decision-theoretic argument with no population in it, and that shrinking towards the middle beats every group’s own mean even when the truths were never drawn from anything.

Then where τ comes from, which is where this field meets the Bayesian one — the prior stops being a belief and becomes a component the data supplies, with a cost that can be counted like any other.

And last, where it fails: the group that is genuinely not from the population, the handful of groups too few to estimate a spread from, and the direction of the error when exchangeability is assumed and false.

Every number in all four is counted the same way. The truth is generated from a stated rule, each estimator is run against thousands of datasets drawn from it, and the squared error of each is totalled — which is the only way to compare estimators that are all defensible in a sentence.