Field

The spread, and its own uncertainty

Partial pooling estimates the population spread from the group means and then uses it as though it were known. It is not known: eight groups leave τ anywhere in a range that spans a factor of several, and on a third of eight-group datasets the usual estimate of it is exactly zero. Carrying that uncertainty rather than dropping it is the difference between an interval that covers 95% and one that covers 79%.

What eight groups say about τ, when the truth is 1. The posterior density for the population spread after eight groups whose standard errors run from 0.5 to 2.1. The shaded band is the central 95% interval, from 1.02 to 4.44; the posterior median is 1.99 and the mean 2.18. The vertical mark at 1.07 is the moment estimate that empirical Bayes substitutes and then treats as known.

What the plug-in forgets

The shrinkage weight needs a population spread, and the population spread has to be estimated from eight numbers. Empirical Bayes estimates it, substitutes it, and proceeds as though it were known — and the interval that comes out covers 79% rather than the 95% it claims.

Three priors on the spread, at a true τ of 0.5. The posterior for τ under a flat prior (mean 1.66), a half-Cauchy of scale 1 (1.32) and one of scale 0.25 (1.15). The three answers differ by 30% of the widest. The prior does visible work when eight groups cannot separate a small spread from none, and almost none when they can.

A prior on the spread

Integrating over the population spread means putting a prior on it, which sounds like the objection rather than the repair. The prior's effect is measurable, it is invisible where the groups are clearly different, and the reflex choice for a scale parameter turns out not to have a posterior at all.

How often "there is no spread between the groups" is reported about data that has one. Every dataset here was generated with a real population spread of 1. The moment estimator is the difference between the observed spread and what noise alone would produce, clamped at zero, and the difference comes out negative often: at eight groups it reports exactly zero on 32.6% of datasets, which is an instruction to pool completely and give all eight groups the same estimate. The rate falls to 4.2% at 48 groups.

When the spread estimates to zero

The usual estimate of a population spread is a difference of two positive quantities, clamped at zero. On a third of eight-group datasets with a real spread in them the difference comes out negative, the estimate is exactly zero, and every group is pooled completely on data that said no such thing.

Counted coverage of two 95% intervals, over 2,000 datasets. Each point is one of the eight groups, at its own standard error. The integrated interval covers 95.2% overall against its stated 95%; the plug-in covers 78.8%, and its shortfall grows with the group's standard error — from 86.1% at se 0.5 to 77.0% at se 2.1. The mean widths are 3.48 and 2.66.

The interval that integrates

A credible interval for one group in a hierarchy has to average over every value the population spread might take. That averaging is what makes it cover — 95.2% against the plug-in's 78.8% — and it costs 31% more width, a heavier tail, and a mixture rather than a normal.

How often a trial that borrows its control from an earlier one declares a treatment with no effect a success, by the earlier control's drift. Fifty patients an arm, an earlier control of two hundred, a one-sided 2.5% threshold. Without borrowing the rate is 2.5% at every drift. Pooling outright reaches 73.1% at a drift of half a standard deviation. A half-normal prior of scale 0.05 on the between-trial spread peaks at 35.2% and is still at 31.8% at a drift of 1.5; a half-Cauchy of the same scale peaks at 11.0% near a drift of 0.4 and falls back to 3.4%.

A control borrowed from the last trial

A trial of fifty patients an arm can borrow its control group from an earlier trial of two hundred by treating the two control means as draws from one population with a spread between trials. With two trials there is one difference to learn that spread from, so the prior on it decides how much is borrowed — and its tail decides whether the borrowing stops when the two trials disagree. A half-normal prior of scale 0.05 buys 17.4 points of power and, at a drift of 1.5 standard deviations, still declares a treatment with no effect a success 31.8% of the time. A half-Cauchy of the same scale buys 13.6 points, peaks at 11.0% and falls back to 3.4%.

How often a trial borrowing from 16 earlier trials that agree declares a treatment with no effect a success, by its own control's drift. Fifty patients an arm, each earlier control of two hundred, a one-sided 2.5% threshold. With 16 earlier trials, the half-normal prior peaks at 88.1%, the half-Cauchy at 80.7% near a drift of 0.6, and the half-Cauchy with a fifth of the prior on a vague component at 16.9%. At a drift of 1.5 the three read 73.3%, 25.5% and 3.7%.

A history that agrees with itself

Borrowing a control from sixteen earlier trials that agree with each other should be safer than borrowing from one, and it is the opposite. Sixteen agreeing trials estimate the spread between trials as small, and a small spread estimated confidently is a licence to pool: under a half-Cauchy prior that let go of a single disagreeing trial, a current control 0.6 standard deviations from a sixteen-trial history turns a treatment with no effect into a success 80.7% of the time, against 10.5% at worst with one earlier trial. Moving a fifth of the prior onto a vague component brings the worst case to 16.9% and keeps power at 91.0%, against 70.5% without borrowing.

How often a trial borrowing from 8 earlier trials on a trend declares a treatment with no effect a success. Fifty patients an arm, each earlier control of two hundred, the current trial one step after the last. At slopes 0, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1: exchangeable 2.7%, 4.4%, 6.6%, 8.3%, 9.9%, 8.3%, 6.7%; with a fifth of the prior vague 2.4%, 3.6%, 5.5%, 6.9%, 7.6%, 6.5%, 5.7%; with a line in time 2.9% at every slope.

A history that has been moving

Earlier control arms borrowed as exchangeable draws are pooled towards the history's middle, and when the control mean has been rising the current trial sits at its end. A drift of 0.02 standard deviations a trial — a step between neighbouring trials of under a third of either one's standard error — turns a treatment with no effect into a success 9.9% of the time from sixteen earlier trials, and a drift of 0.05 overstates the effect by 0.130 standard deviations from eight. A line in time removes the bias exactly, at every slope, and pays for it in borrowing: from eight earlier trials it borrows 175 control patients where the exchangeable model borrows 439.

The bias each model puts into the treatment effect when sixteen earlier controls stepped once. Sixteen earlier trials of two hundred, the first eight a step below the last eight, which sit at the current control's level; the current trial has fifty patients an arm and no treatment effect; 1,500 trials at each step. Exchangeable trials: -0.003, 0.020, 0.041, 0.056, 0.063, 0.065, 0.054. A line in time: -0.003, -0.016, -0.029, -0.041, -0.052, -0.067, -0.075. A change point: -0.003, 0.003, 0.003, -0.001, -0.002, -0.003, -0.003 — at steps of 0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5.

A history that stepped

When sixteen earlier control arms stepped once — the first eight a fifth of a standard deviation below the last — borrowing them as exchangeable flatters a treatment with no effect by 0.063 and stops borrowing, and fitting a line through them penalises it by 0.052. A model with a change point is unbiased at every step size, borrows more the larger the step, 512 patients at a fifth, and puts two thirds of its belief on the right place for the change. Against a steady drift it is the one that fails, so each of the three is safe only against the shape of history it assumes.

All essays