Normal Distribution
nothing is called 95% until it has been counted
Series

Curse — the series

6 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. 12,000 studies of a real effect of 0.3, n = 16. Power is 21%. The studies that reached significance report a mean effect of 0.621 — 2.07 times the truth. Every one of them is honest; the selection did the inflating.

    The winner's curse

    Filter honest studies down to the ones that reached significance and the effects they report are systematically too large. At low power the inflation is a factor of two, nobody has done anything wrong, and the selection did all of it.

    part 3 · testing
  2. Three estimates of the same effect, and the honest one is the worst. 8,000 trials in which every arm has an effect of exactly 0. The arm carried forward is reported by its first stage at +0.187 above the truth — it was chosen for being ahead — and by its second stage at -0.0032, which is unbiased by construction because the selection could not see it. The combination that gets published is at 0.123, exactly the share of the first stage's bias the first stage contributes. Root mean squared error: 0.240, 0.261, 0.182 — the unbiased estimate is the least accurate of the three.

    The estimate after the choice

    An arm chosen for being ahead is ahead by more than it should be, and the trial then publishes the average of the stage that chose it and the stage that did not. The unbiased estimate is the one built from a third of the data — and it is the least accurate of the three.

    part 4 · adaptive
  3. How far three estimates of the chosen arm sit from its true effect, with all eight alike. Eight arms of sixty, the best carried forward with thirty more. Average error: both stages, as published +0.123; stage two alone −0.001; stage one shrunk, then both +0.020. The arms' spread estimates to zero on 57.1% of trials.

    Shrinking the arm that was chosen

    A trial that carries the best of eight arms forward reports it as better than it is — by +0.123 when the eight arms are alike — and its second stage alone is unbiased but the least accurate of the estimates. Pulling the chosen arm's first stage back towards the eight arms' mean before combining removes most of the bias when the arms are alike (+0.020) and gives the smallest error of any estimate, 0.133 against the published 0.182. When one arm really is ahead by half a standard deviation, the same shrinkage pulls the true winner back towards the losers and understates it by 0.103, with more error than the published estimate. The weight is estimated from eight numbers, and it is right when the selection was noise and wrong when the selection was right.

    part 5 · adaptive
  4. How far four estimates of the chosen arm sit from its true effect, with one arm ahead by 0.5. Eight arms of sixty, the best carried forward with thirty more, 20,000 trials. Average error: both stages, as published +0.006; shrunk under a normal prior −0.103; shrunk under a t prior, 3 df −0.101; normal prior with a fifth vague −0.073. Root-mean-square error: 0.151, 0.189, 0.192, 0.176.

    A prior that lets one arm stand out

    Shrinking the chosen arm of eight under a normal prior understates a genuine winner by 0.103. A t prior on three degrees of freedom with the same spread understates it by 0.101 — the tail is too far out to reach a lead of three and a half standard errors. A prior with a fifth of its weight on a vague component understates it by 0.073, costs 0.010 of bias when the arms are alike, and still has more error than the unshrunk estimate. With sixteen arms the same mixture recovers three fifths of the loss, because a bell of fifteen shows its outlier; with eight, the arms put more than half their posterior weight on 'one stands out' in 0.4% of trials where one does.

    part 6 · adaptive
  5. The error of the chosen arm's estimate against a fixed second stage's size, and a second stage sized by the interim, 8 arms, one ahead by half a standard deviation. Fixed second stages of 10, 25, 40, 55, 70 patients: both stages combined err by 0.172, 0.155, 0.143, 0.133, 0.126; the mixture-shrunk estimate by 0.208, 0.183, 0.164, 0.151, 0.140. The rule that sizes the second stage by the interim spends 41.4 on average and errs by 0.142 and 0.159; its second stage alone errs by 0.225.

    A second stage sized by the first

    A drop-the-losers trial can size its second stage by what the first stage says about the arms' shape — more patients when the chosen arm's lead looks like luck, fewer when it looks real. The rule does what it was built to do: at sixteen arms with a genuine winner it spends 51.0 patients a group when the interim chose the wrong arm and 24.2 when it chose the right one. And it buys nothing. Against a fixed second stage of the same average size, every estimate of the chosen arm errs by the same to within a few thousandths; the published combination's bias rises from 0.014 to 0.022, and the second stage alone, still unbiased, errs by 0.319 against 0.281.

    part 7 · adaptive
  6. How far the published estimate of the chosen arm sits from the best arm's effect, by how six hundred patients are split between choosing and estimating. Eight arms and a control, six hundred patients: a stated number an arm in the first stage and the rest split equally between the chosen arm and the control in the second. Root mean squared error of the published estimate about the best arm's true effect, over 5,000 trials a split. All eight alike: best at 16 an arm (0.092), 0.183 at the familiar sixty; One modestly ahead: best at 56 an arm (0.151), 0.153 at the familiar sixty; One clearly ahead: best at 48 an arm (0.146), 0.154 at the familiar sixty; Two clearly ahead: best at 40 an arm (0.128), 0.151 at the familiar sixty; Spread evenly: best at 40 an arm (0.131), 0.146 at the familiar sixty. The split that guards every world puts 32 an arm before the choice and costs at most 21.3% over each world's best.

    Patients spent before the choice

    A trial that picks the best of eight arms and then confirms it splits its patients between choosing and estimating, and the familiar shape — sixty an arm before the choice, thirty after — spends ninety per cent of a six-hundred-patient budget on the choice. Measured against the effect a reader is told about, the best arm's, that split is nearly right when one arm is modestly ahead and twice as wrong as it needs to be when the arms are alike. A split of thirty-two an arm before the choice is never more than 21.3% worse than the best split for whichever world holds; sixty an arm can be 98.8% worse. The two targets an estimate could serve — the chosen arm's effect and the best arm's — pull the split in opposite directions, and only one of them is what gets published.

    part 8 · adaptive

All series