Pooling a proportion

Ten clinics, ten patients each, and a count of events in each. Three saw two events, one saw five, one saw none at all. What is the best estimate of the event rate at the clinic that saw none?

Zero is the answer its own data gives, and zero is wrong in a way that has nothing to do with optimism. It is wrong because ten observations cannot distinguish a rate of zero from a rate of 3%, and the other ninety observations in the study contain information about which is more likely.

Partial pooling is the machinery for using that information, and applying it to a proportion breaks in two places before it starts.

Two reasons the raw scale does not work #

An estimate could leave the interval. The pooling operation is B·μ + (1 − B)·y, a weighted average. On a proportion scale that is safe as long as both μ and y are between 0 and 1 — but the population centre μ is estimated, the weights come from estimated standard errors, and a hierarchy with a covariate in it (which is the next essay) shrinks towards a fitted value that has no such guarantee. A method whose output can be a probability of −0.04 is a method with a special case in it.

The information in a count depends on where it is. A proportion near a half is measured much less precisely than one near zero: the variance of an observed proportion is p(1 − p)/n, which is four times larger at 0.5 than at 0.1. So the standard error that decides how much a group borrows is a function of what the group observed, not only of how much it observed — and two groups of ten that saw 5 and 1 events have genuinely different precisions.

The second point is the one that matters for the argument this field is making. For a mean, precision came from sample size. For a slope, from the design. For a proportion, from the value itself.

The log-odds scale #

The repair is standard and it is one substitution. Work with

y = log(k / (n − k))

whose standard error is approximately √(1/k + 1/(n − k)). On that scale the quantity runs over the whole line, a weighted average of two points on it is another point on it, and the normal-normal machinery applies unchanged. Transform back with the inverse logit and every estimate is a probability by construction.

The price is at the boundary. A group with zero events has k = 0, its log-odds is −∞, and its standard error is infinite. There is nothing to shrink.

The correction, and how much work it is doing #

The usual fix adds a half to both counts:

y = log((k + ½) / (n − k + ½)), se = √(1/(k + ½) + 1/(n − k + ½))

which makes 0 of 10 into log(0.5/10.5) = −3.04 with a standard error of 1.46. Both are finite and the machinery runs.

It is worth being clear about what has happened there, because it is easy to read the correction as tidying. At zero events, the correction is supplying the entire estimate. The group’s own data says −∞ with infinite uncertainty, which is a statement carrying no information at all; what the arithmetic receives instead is a number invented by adding a half. The estimate that comes out — 21.2% in the figure — is very nearly the population’s estimate, and that is the right answer, but it is right because the standard error came out large and the weight came out near one, not because the invented log-odds was accurate.

Whether that is acceptable is a question the correction cannot answer about itself, so it has to be checked against something that does not use it.

The exact route #

The two-level model can be written down without any normal approximation at all:

$$k_j \sim \text{Binomial}(n_j,\; \text{expit}(\theta_j)), \qquad \theta_j \sim \text{Normal}(\mu,\; \tau^2)$$

and the posterior for one group’s θ is that binomial likelihood times that normal prior, which a grid of six thousand points integrates in a millisecond. No standard error appears anywhere. No correction is needed, because the binomial likelihood at k = 0 is perfectly well defined — it is (1 − p)ⁿ, a decreasing function of p, and multiplying it by the population’s normal density gives a posterior with a finite mean.

That is the second route, and comparing the two is the only way to find out whether the log-odds approximation is one.

The number to take from that figure is not the size of any one gap. It is the shape: the two routes agree where the count is comfortable and part company at the ends.

At three, four and five events out of ten the two agree to within 0.007. At zero they are 3.3 percentage points apart — 16.4% against 13.0%, which is a relative difference of a quarter. The approximation is systematically the higher of the two at the low boundary, because adding a half to a count of zero moves it towards the middle, and the exact route does no such thing.

Why that shape is the finding #

A check written only at four events out of ten would report that the approximation is excellent, and would be telling the truth about a case nobody needs pooling for. A group that saw four events out of ten has a perfectly serviceable estimate of its own.

The case pooling is reached for is the group that saw nothing, and that is the case where the approximation is worst. So the assertion in the library is two-sided on purpose: it requires the two routes to agree in the middle and requires them to disagree at the boundary by more than they differ anywhere in the middle. A single-sided version would have certified the method on the region where it is not used.

That second figure is worth a moment because the direction is not the intuitive one. More data usually improves a normal approximation. Here it does not, at the boundary, because the boundary case is precisely where the count carries no information about the shape of its own likelihood, and adding a constant half is a fixed intervention whose relative size does not shrink in the way that matters.

The prior this has been using all along #

There is a version of this operation already on the site, and seeing that it is the same operation is worth more than either half alone.

What a prior is worth puts a Beta prior on a single proportion and measures what it costs in observations. What a credible interval covers counts the coverage of the interval that results. In both, the prior is supplied — Jeffreys’, or flat, or a deliberately confident wrong one.

Partial pooling is that machinery with the prior estimated from the other groups. The population distribution over log-odds is a prior for each group, its centre and spread are recovered from the nine clinics that are not this one, and the shrinkage weight is the number of observations that prior is worth. A group that borrows 80% is a group whose own ten observations have been given a fifth of the say, because a prior worth about forty observations was assembled out of the rest of the study.

That reframing settles a question that otherwise looks like a matter of taste. The objection to a prior on a proportion is that somebody chose it. Here nobody did: it is a summary of the other groups, and it is checkable against them. The objection that survives is a different one — the population model might be wrong about this group, which is what when borrowing goes wrong measures — and it is a claim about structure rather than about belief.

What the pooling is worth #

Three estimators, run against two thousand datasets of ten groups of ten, scored on squared error on the probability scale:

Partial pooling is 37% better than either. And 13.7% of all groups in that simulation saw zero events — so more than one group in eight is a case where the unpooled estimate is not merely imprecise but outside the range the truth can occupy.

Note also that the middle row is worse than the first here, which was not true for slopes. Complete pooling on this data is the worst of the three, because the groups’ true rates genuinely differ by enough that one number for all of them is a poor summary — and the raw proportions, bad as they are, at least point in the right directions.

What the group that saw nothing should be told #

The question this essay opened with has an answer now, and the answer is not a number.

At a population spread of 0.8 on the log-odds scale, the clinic that saw no events is estimated at 21.2%. At a spread of 3.2 it is estimated at 6.4%. Same clinic, same zero out of ten, same nine other clinics in the study — and the estimate differs by a factor of three, because what changed is how different the clinics are from each other.

That is the correct behaviour and it is worth stating as the general rule. A group with no information of its own is estimated entirely from the population, so its estimate is a statement about the population rather than about the group. Where the clinics are alike, the best guess for one that saw nothing is close to the overall rate. Where they differ wildly, seeing nothing is itself weak evidence of being one of the low ones, and the estimate follows it down.

Both of those come out of the same weighted average with no special-casing, which is the argument for having the machinery at all. And both depend on τ, which is estimated from ten numbers and carries every difficulty the field beside this one documents. The estimate for the clinic that saw nothing is the estimate most sensitive to τ, because it is the one being supplied entirely by the population — so it is also the one whose interval the plug-in gets most wrong.

How much of the answer the correction is supplying #

The claim that the added half is doing the work at zero events can be sized rather than asserted, and the figure already prints everything needed for it — twice, by two routes that do not share any arithmetic.

The first route is the weight. A count of zero out of ten becomes a log-odds of −3.04 with a standard error of 1.46, and B = se²/(se² + τ²) at τ = 0.8 is 2.132/(2.132 + 0.64) = 0.769. The second route reads the answer backwards: the population sits at logit(0.313) = −0.786, the group is estimated at 21.2%, which is logit −1.313, and a weighted average that lands there from −3.04 has a weight of 1.727/2.254 = 0.766. Three thousandths apart, from the standard error on one side and from the plotted estimate on the other.

So the honest figure is that the population supplies about three-quarters of the zero group’s estimate and the invented half-event supplies the remaining quarter. That is a great deal less than all of it, and it is a great deal more than nothing: a quarter of a number that gets published as a clinic’s event rate is coming from a constant somebody added to make the logarithm finite.

The second figure says the share moves, and moves the wrong way. At τ = 3.2 the same standard error gives B = 2.132/(2.132 + 10.24) = 0.172, so the group now keeps five-sixths of its own corrected value — and expit(−3.04) is 4.6%, which is most of the 6.4% reported. The correction supplies a quarter of the answer when the groups are alike and five-sixths of it when they are not.

That is worth stating beside the reassuring reading of the same picture. It is true that a group seeing nothing among widely differing groups is more plausibly a genuinely low-rate group, and it is true that the estimate should follow that down. What it follows down to is a number with no data behind it, since a count of zero says only “small” and the half says how small. The exact route is not merely a check on the approximation there — it is the only one of the two that has an opinion about the boundary derived from the model rather than from a convention.

Which extreme is worse says where τ sits #

The three-row table has an ordering that the slope case did not: complete pooling at 0.01946 is 14% worse than making no use of the population at all, at 0.01705, while partial pooling beats the better of them by 37% and the worse by 45%.

That ordering is readable. Ignoring the population costs roughly a group’s own sampling variance; ignoring the group costs roughly the spread between groups. Whichever extreme is worse is the one whose neglected quantity is larger, so complete pooling losing here says the between-group spread exceeds the within-group noise on this data — the clinics genuinely differ by more than ten observations each can resolve.

Reading it that way makes the table a diagnostic rather than a scoreboard. The gap between the two extremes estimates the same ratio the shrinkage weight is built from, and it does so without fitting a hierarchical model at all, which makes it worth computing before one is fitted.

The interval, and the two ways it can be wrong #

Everything above is about the estimate. The interval inherits two separate problems and they are worth separating, because only one of them is new.

The first is the one the field beside this documents: τ is estimated from ten numbers and then treated as known, so every interval is too short and the shortfall is worst for the groups doing the most borrowing. On this data that is the group that saw nothing — which is also the group whose estimate is most likely to be quoted.

The second is older and belongs to proportions rather than to pooling.

A pooled interval for a proportion is therefore built on top of a scale whose own intervals oscillate by several percentage points as the truth moves, for reasons of discreteness that no amount of hierarchical machinery removes. At ten observations per group, an interval that claims 95% is making a claim the sample space cannot quite support in either direction.

The practical reading is that the pooled estimate for a small group is a real improvement over its own proportion and the pooled interval for a small group should be read as approximate. Those are different statements and it is worth not letting the first carry the second.

The scale is a modelling choice, and it shows #

There is one more thing the log-odds detour costs, and it is visible in the first figure if the arrows are read carefully.

Shrinkage is linear on the log-odds scale and therefore not linear on the probability scale. Two groups the same distance apart in raw proportion do not move by the same amount; a group at 0.1 and a group at 0.5, each shrunk by the same weight towards the same centre, travel different distances in probability. The picture of equal-length arrows that partial pooling produces for means is not available here and should not be expected.

That is not a defect. The whole reason for the transformation is that probability is the wrong scale on which to add and subtract, and a consequence of using the right scale is that the wrong one no longer looks tidy. But it does mean that a claim like every group moved a third of the way in has no meaning without saying on what scale, and that reporting the shrinkage weights alongside the estimates is more useful here than it is for a mean.

It also means the choice of scale is a modelling assumption with consequences. Log-odds is the natural choice for a binomial and is what makes the normal-normal algebra apply; a complementary log-log or a probit scale would give slightly different estimates from the same data. The exact route sidesteps the question entirely by never leaving the probability scale — which is a further argument for having it, beyond checking the approximation. Where the two disagree, the exact route is not a better approximation. It is the model, computed.