What eight groups say about τ, when the truth is 1

The posterior density for the population spread after eight groups whose standard errors run from 0.5 to 2.1. The shaded band is the central 95% interval, from 1.02 to 4.44; the posterior median is 1.99 and the mean 2.18. The vertical mark at 1.07 is the moment estimate that empirical Bayes substitutes and then treats as known.

The change-point model's posterior probability that the current regime began at the ninth of sixteen earlier trials, where it did, averaged over 1,500 histories at each step size: 0.06 at 0.05, 0.19 at 0.1, 0.43 at 0.15, 0.67 at 0.2, 0.92 at 0.3, 1.00 at 0.5. With sixteen positions to choose among, a uniform prior puts 1⁄16 on each.

Sixteen earlier trials, 1,500 histories each; the false-positive rate at a nominal one-sided 2.5%. No change: exchangeable trials 2.7%, a line in time 2.3%, a change point 2.5%. A step of 0.1: exchangeable trials 4.1%, a line in time 1.8%, a change point 2.3%. A step of 0.2: exchangeable trials 4.9%, a line in time 1.1%, a change point 2.2%. A drift of 0.02 a trial: exchangeable trials 9.0%, a line in time 2.3%, a change point 5.7%. A drift of 0.05 a trial: exchangeable trials 7.1%, a line in time 2.3%, a change point 8.7%.

The posterior density for the population spread after eight groups whose standard errors run from 0.5 to 2.1. The shaded band is the central 95% interval, from 1.02 to 4.44; the posterior median is 1.99 and the mean 2.18. The vertical mark at 1.07 is the moment estimate that empirical Bayes substitutes and then treats as known.

The posterior for τ under a flat prior (mean 2.18), a half-Cauchy of scale 1 (1.78) and one of scale 0.25 (1.69). The three answers differ by 23% of the widest. The prior does visible work when eight groups cannot separate a small spread from none, and almost none when they can.

Four curves and only two calculations. Under a flat prior the two grids give the same answer — posterior mean 1.389 against 1.389, a difference of 0.0%. Under a 1/τ prior they do not: 1.172 against 1.031, a fall of 12.0%, with the density at the smallest τ on the grid rising from 0.53 to 0.72. There is nothing to converge to: the likelihood is finite at τ = 0 and 1/τ is not integrable there.

Two posteriors for one group's value. The narrow one conditions on the moment estimate τ̂ = 1.07 and treats it as known; the wide one integrates over everything the eight groups leave open about τ. The intervals are 4.11 and 6.03 wide, a factor of 1.47, and their centres differ by only 0.257.

Each point is one of the eight groups, at its own standard error. The integrated interval covers 95.2% overall against its stated 95%; the plug-in covers 78.8%, and its shortfall grows with the group's standard error — from 86.1% at se 0.5 to 77.0% at se 2.1. The mean widths are 3.48 and 2.66.

Every dataset here was generated with a real population spread of 1. The moment estimator is the difference between the observed spread and what noise alone would produce, clamped at zero, and the difference comes out negative often: at eight groups it reports exactly zero on 32.6% of datasets, which is an instruction to pool completely and give all eight groups the same estimate. The rate falls to 4.2% at 48 groups.

The eight groups' own means as hollow circles, the integrated posterior's interval as a bar, and the plug-in estimate as a filled mark. The moment estimate of τ came out exactly zero on this dataset, so every plug-in estimate is the same number, 0.14. The integrated posterior, on the same observations, still gives the groups different values, because its interval for τ runs from 0.02 to 1.82 and it never has to pick one.

Fifty patients an arm, an earlier control of two hundred, a one-sided 2.5% threshold. Without borrowing the rate is 2.5% at every drift. Pooling outright reaches 73.1% at a drift of half a standard deviation. A half-normal prior of scale 0.05 on the between-trial spread peaks at 35.2% and is still at 31.8% at a drift of 1.5; a half-Cauchy of the same scale peaks at 11.0% near a drift of 0.4 and falls back to 3.4%.

Pooled outright, the earlier control would carry weight 0.800 and move the estimate by that share of every disagreement. Under a half-normal prior of scale 0.05 on the spread, a disagreement of 0.5 moves the estimate by 0.297 and one of 1.5 by 0.292; under a half-Cauchy of the same scale, by 0.131 and 0.029.

Not borrowing has power 70.5% and a worst rate of 2.5%. Pooling outright has 89.6% and 100.0%. Half-normal priors from scale 0.02 to 0.5 run from 89.2% at 95.6% to 76.5% at 4.3%; half-Cauchy priors from 87.2% at 15.8% to 76.4% at 4.2%.

Both put most of their mass below about 0.10. The half-normal gives a spread above 0.5 a probability of 1.5 × 10^−23; the half-Cauchy gives it 0.063.

Fifty patients an arm, each earlier control of two hundred, a one-sided 2.5% threshold. With 16 earlier trials, the half-normal prior peaks at 88.1%, the half-Cauchy at 80.7% near a drift of 0.6, and the half-Cauchy with a fifth of the prior on a vague component at 16.9%. At a drift of 1.5 the three read 73.3%, 25.5% and 3.7%.

Under a half-Cauchy prior of scale 0.05, the 90th percentile of the spread's posterior is 0.181 from one earlier trial (which says nothing beyond the prior), 0.084 from four and 0.063 from sixteen, drawn with no true spread.

Under a half-Cauchy prior of scale 0.05 on the spread, a current control 1.5 standard deviations above the history is pulled back by 0.029 with one earlier trial, 0.070 with four and 0.266 with sixteen; with sixteen and a fifth of the prior on a vague component, by 0.029. At 0.5 the four read 0.131, 0.362, 0.482 and 0.043.

With no vague component the analysis has power 94.3% and a worst false-positive rate of 80.7%. A vague share of 0.1 gives 92.6% and 22.1%; 0.2, 91.0% and 16.9%; 0.5, 86.4% and 9.2%. The trial alone has 70.5%.

Fifty patients an arm, each earlier control of two hundred, the current trial one step after the last. At slopes 0, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1: exchangeable 2.7%, 4.4%, 6.6%, 8.3%, 9.9%, 8.3%, 6.7%; with a fifth of the prior vague 2.4%, 3.6%, 5.5%, 6.9%, 7.6%, 6.5%, 5.7%; with a line in time 2.9% at every slope.

Eight earlier control means, each from two hundred patients, on a slope of 0.05 standard deviations a trial; the current trial's true control mean is zero. The exchangeable model's prior for it has mean -0.213 and standard deviation 0.138; the model with a line in time, mean 0.034 and standard deviation 0.071.

Median over 1,500 histories of the posterior median of τ from the earlier trials alone, half-Cauchy prior of scale 0.05. At a slope of 0.1: 4 trials 0.090, 8 trials 0.231, 16 trials 0.472; the spread of a line across the trials' times is 0.112, 0.229, 0.461.

Average estimated effect less the true one, in outcome standard deviations, at slopes 0, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1: exchangeable 0.007, 0.046, 0.082, 0.110, 0.130, 0.118, 0.102; a fifth vague 0.006, 0.038, 0.068, 0.089, 0.104, 0.094, 0.080; a line in time 0.007 at every slope.

One over the current control's posterior variance less its own fifty patients, averaged over 1,500 trials with the earlier trials identical in truth: 3 earlier trials, exchangeable 183.2 and a line in time 44.1; 4 earlier trials, exchangeable 238.1 and a line in time 69.7; 6 earlier trials, exchangeable 342.7 and a line in time 123.0; 8 earlier trials, exchangeable 438.8 and a line in time 174.9; 12 earlier trials, exchangeable 612.0 and a line in time 275.0; 16 earlier trials, exchangeable 760.1 and a line in time 370.4.