20 adaptive trials, 45% against 25%

Each line is one trial allocating patients one at a time by the arm's own posterior. The average final share on the better arm is 84.7%, with a standard deviation of 10.3 points across these 20 trials. The rule does not deliver a fixed advantage: it delivers one that depends on how the first few patients came out.

One modestly ahead: 39%, 45%, 52%, 58%, 62%, 67%, 68%, 70%; One clearly ahead: 71%, 82%, 89%, 94%, 96%, 98%, 98%, 98%; Two clearly ahead: 81%, 88%, 93%, 96%, 97%, 99%, 99%, 99%; Spread evenly: 36%, 42%, 44%, 49%, 51%, 54%, 53%, 55% — at 16, 24, 32, 40, 48, 56, 60, 64 patients an arm. The effect given up against the best arm, on average: one modestly ahead, 0.153 at sixteen an arm and 0.081 at sixty; one clearly ahead, 0.144 at sixteen an arm and 0.009 at sixty; two clearly ahead, 0.078 at sixteen an arm and 0.006 at sixty; spread evenly, 0.108 at sixteen an arm and 0.055 at sixty.

The published estimate's root mean squared error about the chosen arm's own effect: 0.091, 0.097, 0.103, 0.113, 0.125, 0.139, 0.152, 0.165; about the best arm's effect: 0.269, 0.214, 0.177, 0.154, 0.146, 0.147, 0.154, 0.164 — at 16, 24, 32, 40, 48, 56, 60, 64 patients an arm before the choice. The first falls the more patients go after the choice; the second is smallest at 48 an arm.

Fixed second stages of 10, 25, 40, 55, 70 patients: both stages combined err by 0.172, 0.155, 0.143, 0.133, 0.126; the mixture-shrunk estimate by 0.208, 0.183, 0.164, 0.151, 0.140. The rule that sizes the second stage by the interim spends 41.4 on average and errs by 0.142 and 0.159; its second stage alone errs by 0.225.

Average second stage per group: 8 arms, all alike, 60.2; 8 arms, winner chosen, 41.1; 8 arms, another arm chosen, 58.0; 16 arms, all alike, 55.1; 16 arms, winner chosen, 24.2; 16 arms, another arm chosen, 51.0. With one arm ahead by half a standard deviation, the interim chose it in 98.2% of trials at eight arms and 96.5% at sixteen.

Bias of the chosen arm's estimate: fixed second stage, by information +0.014 (error 0.157); sized, combined by information +0.022 (error 0.158); sized, combined with planned weights +0.012 (error 0.165); sized, second stage alone +0.000 (error 0.319). The fixed stage has 25 patients a group, the sized rule's average.

One arm ahead by half a standard deviation. At 8 arms the interim chose it in 98.2% of trials; there the published estimate errs by 0.1397 with a fixed stage of 41 and 0.1406 sized, and in the rest by 0.2370 and 0.2117. At 16 arms the interim chose it in 96.5% of trials; there the published estimate errs by 0.1513 with a fixed stage of 25 and 0.1544 sized, and in the rest by 0.2675 and 0.2298.

Each line is one trial allocating patients one at a time by the arm's own posterior. The average final share on the better arm is 84.7%, with a standard deviation of 10.3 points across these 20 trials. The rule does not deliver a fixed advantage: it delivers one that depends on how the first few patients came out.

Both arms have a success rate of 30% and it rises by the same amount over the course of the trial, so there is no difference anywhere to find. A fair coin finds none: 5.4% at no drift and 5.6% at 0.5, the nominal rate throughout, and that is what randomisation buys. The adaptive rule starts at 8.4% with no drift at all — the allocation is a function of the outcomes, so the two are not independent — and reaches 75% at a drift of 0.5, because the early patients went to whichever arm looked good early and the late ones went to the other. Comparing within 8 blocks of arrival time removes the part the drift caused, bringing 75% down to 1.3%, and leaves the part the adaptation caused: 7.6% at no drift.

8,000 trials in which every arm has an effect of exactly 0. The arm carried forward is reported by its first stage at +0.187 above the truth — it was chosen for being ahead — and by its second stage at -0.0032, which is unbiased by construction because the selection could not see it. The combination that gets published is at 0.123, exactly the share of the first stage's bias the first stage contributes. Root mean squared error: 0.240, 0.261, 0.182 — the unbiased estimate is the least accurate of the three.

8,000 trials with no effect in any arm. Stage one runs 8 arms at 60 each, the best is carried forward, and stage two adds 30 more to it and to the control. The histogram is where the final statistic lands and the curve is the standard normal it is being read against — shifted right, because the arm was chosen for being ahead. 10.5% of these trials clear 1.96 against a claimed 5%, and the value that actually holds the rate for this design is 2.313.

The same 60 observations, estimated two ways. Keeping the arms separate gives 0.995, which is σ. Pooling them without separating the arms — the price of staying blind to the comparison — gives 1.114, against the identity √(1 + Δ²/4σ²) = 1.118. The sample size is proportional to the variance, so a blinded design at this effect asks for 25% more units than it needs, and it does so systematically rather than by chance.

Eight arms of sixty, the best carried forward with thirty more. Average error: both stages, as published +0.123; stage two alone −0.001; stage one shrunk, then both +0.020. The arms' spread estimates to zero on 57.1% of trials.

all eight alike: published 0.182, stage two 0.261, shrunk 0.133; spread evenly from 0 to 0.5: published 0.160, stage two 0.261, shrunk 0.145; one arm ahead by 0.25: published 0.165, stage two 0.261, shrunk 0.157; two arms ahead by 0.4: published 0.152, stage two 0.261, shrunk 0.152; one arm ahead by 0.5: published 0.151, stage two 0.261, shrunk 0.189.

With the arms alike the weight on the arms' mean averages 0.879 and is exactly one — the spread estimated as zero — on 56.6% of trials. With one arm ahead by 0.5 it averages 0.416.

Eight arms of sixty, the best carried forward with thirty more, 20,000 trials. Average error: both stages, as published +0.006; shrunk under a normal prior −0.103; shrunk under a t prior, 3 df −0.101; normal prior with a fifth vague −0.073. Root-mean-square error: 0.151, 0.189, 0.192, 0.176.

20,000 trials each. all eight alike: published 0.182, normal 0.133, t 0.131, mixture 0.135; spread evenly from 0 to 0.5: published 0.160, normal 0.145, t 0.146, mixture 0.145; one arm ahead by 0.25: published 0.165, normal 0.157, t 0.158, mixture 0.154; two arms ahead by 0.4: published 0.152, normal 0.152, t 0.157, mixture 0.150; one arm ahead by 0.5: published 0.151, normal 0.189, t 0.192, mixture 0.176.

One arm ahead of seven alike, 4,000 trials a point. At leads of 0.25, 0.5, 0.75, 1, 1.5: published +0.071, +0.004, −0.002, −0.002, −0.002; normal prior −0.039, −0.104, −0.090, −0.073, −0.052; t prior, 3 df −0.045, −0.102, −0.072, −0.050, −0.032; a fifth vague −0.023, −0.075, −0.057, −0.043, −0.031.

The mixture prior's posterior weight on its vague component for the chosen arm. With the arms alike its average is 0.086 and it exceeds one half in 0.0% of trials; with one arm ahead by half a standard deviation, 0.285 and 0.4%.

4,000 trials a point. With one arm ahead by 0.5, normal shrinkage errs by −0.055, −0.082, −0.104, −0.131, −0.156 and the mixture by −0.047, −0.065, −0.075, −0.069, −0.063 at 4, 6, 8, 12, 16 arms. With all alike, +0.023, +0.021, +0.019, +0.023, +0.019 and +0.027, +0.028, +0.029, +0.038, +0.039.