Eight groups, τ = 1 against a within-group spread of 3

Each row is a group. The hollow circle is the group's own mean, the filled one is the estimate after pooling, and the small mark is the truth the data was generated from. The group of 3 moves 75% of the way to the population mean of 0.10; the group of 40 moves 18%.

Power of the low-kurtosis test at 5% against two clusters with the stated total spread, groups of four observations with noise 3; the observed means and the pooled estimates give the same power. Spread 1: 6.5%, 6.0%, 6.0%, 8.1%. Spread 2: 12.2%, 19.4%, 28.9%, 45.5%. Spread 4: 45.3%, 69.4%, 91.9%, 99.5% — at 8, 16, 32, 64 groups. The analyst's own check against the fitted normal population, dots: 6.2%, 6.2%, 6.4%, 7.5%; 13.0%, 18.4%, 28.1%, 45.9%; 43.6%, 67.7%, 91.1%, 99.5%.

The share of datasets whose estimated spread between groups is zero, so that every pooled estimate equals the grand mean, under two clusters and under a normal population of the same spread one: 29.9% and 31.5% at 8 groups, 18.3% and 19.7% at 16 groups, 8.6% and 10.0% at 32 groups, 2.1% and 2.6% at 64 groups.

Groups of 2, 4, 8 and 16 observations in turn, spread 2, noise 3; power of the low-kurtosis test at 5%. Observed means: 16.3%, 23.4%, 34.7%, 45.5%. Pooled estimates: 18.9%, 29.4%, 47.6%, 75.0%. Standardised means: 17.6%, 26.6%, 44.5%, 68.3%. Check against the fitted model: 16.3%, 28.1%, 44.0%, 65.7% — at 8, 16, 32, 64 groups. The fitted-model check's size: 4.9%, 5.3%, 5.0%, 4.8%.

A hundred groups from four to four hundred patients, ranked from their posterior under a fitted quadratic size relation whose uncertainty is drawn once per table. The groups of twenty patients or fewer have rank intervals 30.1 places wide on average, and 34 of their 36 true ranks fall inside; the groups of over a hundred, 32.1 places, with 27 of 30 inside.

Groups of twenty patients or fewer. Coverage of the 90% rank interval, with its width in places, and of the 95% effect interval: fitted line, plugged in 95.6% over 36.9 places and 83.1%; fitted quadratic, plugged in 87.5% over 28.1 places and 85.5%; quadratic, each group inflated 92.8% over 32.8 places and 93.7%; quadratic, centre drawn once 89.1% over 30.0 places and 93.7%; the true relation 91.6% over 27.3 places and 94.8%.

Over 300 tables. Groups of twenty or fewer: fitted line plugged in 95.6%, quadratic with its centre drawn once 89.1%, true relation 91.6%. Groups of over a hundred: 87.6%, 89.5% and 91.6%.

Average width over 300 tables. Under a straight relation and the true centre, the group of four spans 56.5 places and the group of four hundred 18.8; under a steepening relation, 16.0 and 16.9, and 25.8 for the group of four under a fitted line.

Each row is a group. The hollow circle is the group's own mean, the filled one is the estimate after pooling, and the small mark is the truth the data was generated from. The group of 3 moves 75% of the way to the population mean of 0.10; the group of 40 moves 18%.

Each curve is one population spread τ. A group's estimate moves B = se²/(se² + τ²) of the way to the population mean, where se = σ/√n is what the group's own mean does not know. At τ = 1 a group of 9 observations sits halfway.

Each point is 1,200 datasets of 8 groups. At τ = 0 the groups are identical and complete pooling is best at 0.29 against 2.23; at τ = 6 they are unrelated and it is worst at 31.5 against 2.23. Partial pooling is at or below both at every point.

Squared error for each group under partial pooling, with each group's own mean beside it. Seven of the eight are estimated better by pooling. The eighth, which was never from the population, is estimated 2.9 times worse — 6.5 against 2.3.

The population spread is not supplied to a hierarchical model — it is estimated from how far apart the group means are, after subtracting the noise that would separate them anyway. It averages 1.41 here against a true 1.5, and comes out exactly zero on 5% of datasets.

Each point is 4,000 datasets, with the population spread estimated from the data rather than supplied. Partial pooling first beats BOTH of the estimators it sits between at 5 groups; below that, complete pooling — which estimates nothing at all — is the better answer. Its own cost falls from 1.751 at 2 groups to 0.795 at 40.

The factor is one minus (J − 3) times the within-group variance over the sum of squares when the centre is the data's own mean, and one minus (J − 2) times the same ratio when the centre is given. At three groups the first constant is zero, so the factor is exactly 1 and nothing moves; at two it is negative and the estimator expands on 100% of datasets. Each point of the estimated-centre curve sits close to the given-centre curve one group to its left.

Eight groups whose sizes span a factor of 13.3. The smallest gains 2.076 of squared error, which is 36.8% of the total reduction; the largest gains 0.030, which is 0.5%. 2 of the eight account for half of everything pooling buys.

Every group's truth is in one of two clusters, and the population's total spread is exactly τ = 1, so the analysis recovers τ̂ = 1.515 and every shrinkage weight is what it would be for a single normal population. The estimates are pulled towards the grand mean, which is the middle of the gap — a place 0 of the 12 truths are and 6 of the estimates end up.

The middle half of the space between the two clusters holds 6.6% of the group truths at every spread, by construction. The observed group means land there 20.1% of the time at τ = 1 and the pooled estimates 46.3% — so pooling more than doubles the number of estimates in a region the population has all but vacated.

Two populations of the same total spread and completely different shape, put through the same analysis 3,000 times each. The reported spread averages 0.883 from the clusters and 0.841 from the normal — a difference of 0.042 against a between-dataset spread of 0.733, which is 6% of it and is not a difference a single study could read.

Pooling's expected squared error for a group, divided by its own mean's, against how far the group truly sits from the centre. It is ×0.25 at the centre and crosses ×1 at 1.732 population widths, beyond which 8.33% of a normal population lies; capping the shift at one standard error holds every group under ×2.

Uncapped, pooling's total error is ×0.500 of the unpooled total and its worst group reaches ×9.25 by six population widths. A cap of one standard error gives ×0.528 and ×2.00; a cap of zero is no pooling, ×1 and ×1.

Over 20,000 sets of eight groups, each group's standard error equal to the population's spread, with that spread estimated from the eight means. Pooling's squared error is ×0.647 of the unpooled error over all eight. For the group whose true effect is furthest out it is ×1.218, worse than its own mean in 61.6% of sets; for the group whose observed mean is furthest out it is ×0.417, worse in 26.8%.

Beyond two population widths above the centre lie 2.28% of the true values, 7.86% of the raw means, 0.234% of the posterior means, and 2.28% of the constrained estimates.

The truth has 2.28% of groups beyond 2 widths. The raw means count 7.86%, the posterior means 0.234% and the constrained estimates 2.28%; their mean squared errors are 1.000, 0.500 and 0.586 of a population variance.

A hundred groups with sizes from 4 to 400, of which 36% have twenty units or fewer. Small groups make up 36.3% of the true top ten, 62.1% of the top ten by raw means, 13.4% by posterior means and 22.9% by the posterior chance of being in the top ten. The three rankings recover 4.43, 5.38 and 5.47 of the true top ten.

In truth a group of any size is in the top ten one time in ten. Ranked by raw means, the smallest groups (4–10 units) are placed there 20.0% of the time and the largest (101–400) 4.4%; ranked by posterior means, 2.1% and 15.6%; by the posterior chance of being in the top ten, 4.8% and 12.5%.

Groups ordered by posterior mean. The bar spans the 5th to 95th percentile of each group's rank over 1,000 draws of the whole table from the posterior. The leading group, of 39 units, could rank anywhere from 1 to 31; the twentieth from 4 to 65.

Groups of 4 to 400, true effects rising with log size at a slope of 0.6 population widths. The true bottom ten is 80.6% small groups. Ranked by their own means the bottom ten holds 4.28 of the true ten; pooled to one centre, 3.19, with 42.6% small groups; pooled to a centre that rises with size, 4.35, with 83.0%.

Hollow points are the groups' own means, clipped at three population widths; filled points are the estimates pooled to zero; the line is that prediction. For the groups of twenty or fewer the pooled estimate sits on average 0.456 above the truth.

The small groups' average true effect is −0.259. Their estimates are off on average by −0.012 (own means), +0.276 (one fitted centre), −0.173 (fitted line in log size), −0.025 (fitted quadratic in log size), −0.003 (the true relation).

The fitted line has slope 0.255 and the between-group spread it estimates is 1.150; the quadratic's curvature is 0.236. At the smallest size the true relation is −0.259, the line predicts −0.448 and the quadratic 0.214.

straight: line 0.750, quadratic 0.774, true relation 0.719; flattening below the median size: line 0.871, quadratic 0.866, true relation 0.817; steepening below the median size: line 0.539, quadratic 0.490, true relation 0.398.

Groups of twenty patients or fewer, 600 tables of a hundred groups. Coverage: one centre, plugged in 58.0% (average width 2.81); one centre, its uncertainty added 58.5% (average width 2.83); a fitted line, plugged in 83.1% (average width 1.58); a fitted line, its uncertainty added 88.4% (average width 1.77); a fitted quadratic, plugged in 85.5% (average width 1.53); a fitted quadratic, its uncertainty added 93.7% (average width 1.92); the true relation and spread 94.8% (average width 1.55).

Over 600 tables. The smallest group, of four patients: fitted line plugged in 71.2%, with its uncertainty 80.8%, fitted quadratic with its uncertainty 94.8%, true relation 95.0%. The largest, of four hundred: 93.3%, 94.3%, 93.5% and 95.7%.

The 17 groups of eight patients or fewer. A fitted line with its centre treated as known misses 4 of their true effects; a fitted quadratic with its centre's uncertainty added misses 0.

Each interval's coverage of groups of twenty or fewer and its average width, 600 tables per shape. straight: one centre, plugged in 86.9% at 3.19, one centre, its uncertainty added 87.2% at 3.22, a fitted line, plugged in 92.8% at 2.78, a fitted line, its uncertainty added 93.9% at 2.89, a fitted quadratic, plugged in 92.1% at 2.78, a fitted quadratic, its uncertainty added 94.2% at 3.00, the true relation and spread 94.9% at 2.80. flattening below the median size: one centre, plugged in 93.9% at 3.32, one centre, its uncertainty added 94.1% at 3.34, a fitted line, plugged in 92.9% at 3.18, a fitted line, its uncertainty added 93.7% at 3.28, a fitted quadratic, plugged in 92.8% at 3.16, a fitted quadratic, its uncertainty added 94.3% at 3.35, the true relation and spread 94.8% at 3.19. steepening below the median size: one centre, plugged in 58.0% at 2.81, one centre, its uncertainty added 58.5% at 2.83, a fitted line, plugged in 83.1% at 1.58, a fitted line, its uncertainty added 88.4% at 1.77, a fitted quadratic, plugged in 85.5% at 1.53, a fitted quadratic, its uncertainty added 93.7% at 1.92, the true relation and spread 94.8% at 1.55.

One table with a straight size relation. The centre's share of the centre-aware interval's variance for the group of four: one centre 1.3%, a line 10.4%, a quadratic 26.0%; for the group of four hundred, 0.1%, 0.4% and 0.7%.

Over 400 tables of a hundred groups each. The quadratic's curvature coefficient is significant at 5% on 5.3% of tables whose relation is straight, 14.5% of tables whose relation is flattening below the median, 29.3% of tables whose relation is steepening below the median; its average AIC weight is 0.374, 0.456, 0.558.

Per table of a hundred, over 400 tables. straight: always a line 4.59 (holding 4.25 of the true ten), chosen by the curvature test 4.57 (holding 4.22 of the true ten), always a quadratic 4.42 (holding 4.18 of the true ten), told the shape 4.59 (holding 4.25 of the true ten); flattening below the median: always a line 2.86 (holding 4.48 of the true ten), chosen by the curvature test 2.61 (holding 4.51 of the true ten), always a quadratic 2.19 (holding 4.50 of the true ten), told the shape 2.19 (holding 4.50 of the true ten); steepening below the median: always a line 3.44 (holding 6.55 of the true ten), chosen by the curvature test 3.44 (holding 6.55 of the true ten), always a quadratic 3.40 (holding 6.59 of the true ten), told the shape 3.40 (holding 6.59 of the true ten).

A steepening relation; the curvature statistic is −1.32, so the test keeps the line, and the quadratic's AIC weight is 0.46. Flagging every group whose chance exceeds a half, the line flags 6 groups, 4 of them truly in the bottom ten; the mixture flags 11, 6 of them truly there.

Per table of a hundred, over 400 tables of 200 draws each. straight: the chosen centre, drawn flags 2.49, 1.31 truly in the bottom ten, the choice carried by AIC weights flags 2.55, 1.34 truly in the bottom ten, the told centre, drawn flags 2.40, 1.33 truly in the bottom ten; flattening below the median: the chosen centre, drawn flags 3.12, 2.04 truly in the bottom ten, the choice carried by AIC weights flags 3.15, 2.07 truly in the bottom ten, the told centre, drawn flags 3.44, 2.22 truly in the bottom ten; steepening below the median: the chosen centre, drawn flags 7.29, 5.04 truly in the bottom ten, the choice carried by AIC weights flags 7.93, 5.47 truly in the bottom ten, the told centre, drawn flags 8.35, 5.68 truly in the bottom ten.