Eight groups, τ = 1 against a within-group spread of 3

Each row is a group. The hollow circle is the group's own mean, the filled one is the estimate after pooling, and the small mark is the truth the data was generated from. The group of 3 moves 75% of the way to the population mean of 0.10; the group of 40 moves 18%.

Each curve is one population spread τ. A group's estimate moves B = se²/(se² + τ²) of the way to the population mean, where se = σ/√n is what the group's own mean does not know. At τ = 1 a group of 9 observations sits halfway.

Each point is 1,200 datasets of 8 groups. At τ = 0 the groups are identical and complete pooling is best at 0.29 against 2.23; at τ = 6 they are unrelated and it is worst at 31.5 against 2.23. Partial pooling is at or below both at every point.

Squared error for each group under partial pooling, with each group's own mean beside it. Seven of the eight are estimated better by pooling. The eighth, which was never from the population, is estimated 2.9 times worse — 6.5 against 2.3.

The population spread is not supplied to a hierarchical model — it is estimated from how far apart the group means are, after subtracting the noise that would separate them anyway. It averages 1.41 here against a true 1.5, and comes out exactly zero on 5% of datasets.

Each point is 4,000 datasets, with the population spread estimated from the data rather than supplied. Partial pooling first beats BOTH of the estimators it sits between at 5 groups; below that, complete pooling — which estimates nothing at all — is the better answer. Its own cost falls from 1.751 at 2 groups to 0.795 at 40.

The factor is one minus (J − 3) times the within-group variance over the sum of squares when the centre is the data's own mean, and one minus (J − 2) times the same ratio when the centre is given. At three groups the first constant is zero, so the factor is exactly 1 and nothing moves; at two it is negative and the estimator expands on 100% of datasets. Each point of the estimated-centre curve sits close to the given-centre curve one group to its left.

Eight groups whose sizes span a factor of 13.3. The smallest gains 2.076 of squared error, which is 36.8% of the total reduction; the largest gains 0.030, which is 0.5%. 2 of the eight account for half of everything pooling buys.

Every group's truth is in one of two clusters, and the population's total spread is exactly τ = 1, so the analysis recovers τ̂ = 1.515 and every shrinkage weight is what it would be for a single normal population. The estimates are pulled towards the grand mean, which is the middle of the gap — a place 0 of the 12 truths are and 6 of the estimates end up.

The middle half of the space between the two clusters holds 6.6% of the group truths at every spread, by construction. The observed group means land there 20.1% of the time at τ = 1 and the pooled estimates 46.3% — so pooling more than doubles the number of estimates in a region the population has all but vacated.

Two populations of the same total spread and completely different shape, put through the same analysis 3,000 times each. The reported spread averages 0.883 from the clusters and 0.841 from the normal — a difference of 0.042 against a between-dataset spread of 0.733, which is 6% of it and is not a difference a single study could read.

Pooling's expected squared error for a group, divided by its own mean's, against how far the group truly sits from the centre. It is ×0.25 at the centre and crosses ×1 at 1.732 population widths, beyond which 8.33% of a normal population lies; capping the shift at one standard error holds every group under ×2.

Uncapped, pooling's total error is ×0.500 of the unpooled total and its worst group reaches ×9.25 by six population widths. A cap of one standard error gives ×0.528 and ×2.00; a cap of zero is no pooling, ×1 and ×1.

Over 20,000 sets of eight groups, each group's standard error equal to the population's spread, with that spread estimated from the eight means. Pooling's squared error is ×0.647 of the unpooled error over all eight. For the group whose true effect is furthest out it is ×1.218, worse than its own mean in 61.6% of sets; for the group whose observed mean is furthest out it is ×0.417, worse in 26.8%.

Beyond two population widths above the centre lie 2.28% of the true values, 7.86% of the raw means, 0.234% of the posterior means, and 2.28% of the constrained estimates.

The truth has 2.28% of groups beyond 2 widths. The raw means count 7.86%, the posterior means 0.234% and the constrained estimates 2.28%; their mean squared errors are 1.000, 0.500 and 0.586 of a population variance.

A hundred groups with sizes from 4 to 400, of which 36% have twenty units or fewer. Small groups make up 36.3% of the true top ten, 62.1% of the top ten by raw means, 13.4% by posterior means and 22.9% by the posterior chance of being in the top ten. The three rankings recover 4.43, 5.38 and 5.47 of the true top ten.

In truth a group of any size is in the top ten one time in ten. Ranked by raw means, the smallest groups (4–10 units) are placed there 20.0% of the time and the largest (101–400) 4.4%; ranked by posterior means, 2.1% and 15.6%; by the posterior chance of being in the top ten, 4.8% and 12.5%.

Groups ordered by posterior mean. The bar spans the 5th to 95th percentile of each group's rank over 1,000 draws of the whole table from the posterior. The leading group, of 39 units, could rank anywhere from 1 to 31; the twentieth from 4 to 65.