Testing at 0.05 every time the data is looked at

The null is true in every one of these trials and the test is correct every time it is run. Looking once rejects 4.9% of the time, as it should; looking ten times rejects 19.2% of the time. Nothing changed except permission to look.

Adding the look with chance one half when the blinded variance is high spends 0.069, 0.042, 0.030, 0.021, 0.015, 0.009 points above 5% with 20, 50, 100, 200, 400, 1,000 patients at the interim. On logarithmic axes the points fall with slope −0.51: the residue shrinks as one over the square root of the interim size, because the interim statistic makes up one degree of freedom of all those the blinded variance is summed over, and the trigger's sensitivity to it is the chi-square's density at the threshold.

With the threshold set so the look is added half the time under the null: with 20 patients at the interim the chance is 43.4% at an interim statistic of zero and 70.7% at plus or minus two; with 100, 47.2% and 58.6%; with 1,000, 49.1% and 52.7%. A trigger on the within-arm variance adds it with chance one half at every interim value.

Under the effect the trial is powered for, with the trigger set to add the look in half of null trials among those still running: blinded variance, 20 patients, 67.1%; blinded variance, 50 patients, 60.5%; blinded variance, 100 patients, 57.4%; blinded variance, 200 patients, 55.2%; blinded variance, 400 patients, 53.6%; blinded variance, 1,000 patients, 52.3%; within-arm variance or calendar, 50.0%; the interim trend, 92.1%. The within-arm variance and the calendar add it 50.0% of the time whatever the effect, so they carry none of it.

Straight to the end: power 89.01% at 338.9 expected observations. A look at three quarters always: 88.38% at 300.7. Adding the look when the interim |z| is at least 0.5, 1, 1.5, 2, 2.5, with boundaries corrected to spend exactly 5%: 88.41% at 302.0; 88.45% at 302.8; 88.56% at 306.3; 88.76% at 315.5; 88.95% at 330.1. With the spending boundaries alone the same rules spend 5.059%, 5.156%, 5.232%, 5.187%, 5.058%.

The spending function alone puts the added look's boundary at 2.298 whatever the interim value. Raising it for every triggered trial by one factor to spend exactly 5% puts it at 2.345. Preserving each interim value's planned chance of crossing puts it at 2.290 for a trial triggered at |z| = 1.50 and 2.459 for one at 2.77, next to the first boundary.

straight to the end, fixed: error rate 5.0000%, power 89.01%, 338.9 expected observations with an effect; a look at three quarters, fixed: error rate 5.0000%, power 88.38%, 300.7 expected observations with an effect; added look, spending boundaries alone: error rate 5.2323%, power 88.94%, 304.9 expected observations with an effect; added look, its own boundaries raised: error rate 5.0000%, power 88.56%, 306.0 expected observations with an effect; added look, every later boundary raised: error rate 5.0000%, power 88.55%, 305.4 expected observations with an effect; added look, planned chance preserved: error rate 5.0000%, power 88.56%, 306.3 expected observations with an effect.

At triggers of 0.5, 1, 1.5, 2, 2.5, the rule with the spending boundaries alone spends 0.059, 0.156, 0.232, 0.187, 0.058 points above 5%, and correcting it to spend exactly 5% costs 0.075, 0.247, 0.379, 0.304, 0.096 points of power at an effect of 0.16 — 1.26, 1.59, 1.63, 1.63, 1.64 points of power for each point of excess error.

The null is true in every one of these trials and the test is correct every time it is run. Looking once rejects 4.9% of the time, as it should; looking ten times rejects 19.2% of the time. Nothing changed except permission to look.

Forty trials in which nothing is happening, monitored at 5 interim points. The boundary is 1.96, 1.96, 1.96, 1.96, 1.96. 3 of the forty cross it somewhere and would be reported as significant.

test at 0.05 every look: 1.96, 1.96, 1.96, 1.96, 1.96. Pocock — a constant, higher boundary: 2.41, 2.41, 2.41, 2.41, 2.41. O'Brien–Fleming — strict early, nearly nominal at the end: 4.55, 3.22, 2.63, 2.27, 2.03. Bonferroni across looks: 2.58, 2.58, 2.58, 2.58, 2.58. Every one except the first spends the same total error rate; they differ in when they spend it.

test at 0.05 every look: rejects a true null 14.4%, finds a real effect 92%, uses 204 observations on average. Pocock: rejects a true null 4.9%, finds a real effect 82%, uses 256 observations on average. O'Brien–Fleming: rejects a true null 4.9%, finds a real effect 88%, uses 299 observations on average. Bonferroni across looks: rejects a true null 3.1%, finds a real effect 77%, uses 274 observations on average.

The dashed line is the smallest effect a trial can report and still stop at each look: 0.510 at 80 observations, 0.255 at 160 observations, 0.170 at 240 observations, 0.128 at 320 observations, 0.102 at 400 observations. The true effect is 0.16, so at 3 of the five looks a trial cannot stop without reporting more than it. 29 of these forty trials stop before the last look, each marked where it stopped.

Each column is one look of an O'Brien–Fleming trial; above the boundary a trial stops there. Highlighted are the outcomes that count as at least as extreme as the observed one when outcomes are ordered stagewise: at 80, z ≥ 4.56 (probability 2.54 × 10⁻⁶ with no effect); at 160, z ≥ 3.30 (probability 4.82 × 10⁻⁴ with no effect); at 240, none; at 320, none; at 400, none. The two-sided p-value is 9.69 × 10⁻⁴.

The upper line is the benefit boundary (4.56, 3.23, 2.63, 2.28, 2.04); the lower line is where the rule stops a trial for futility (0.40 at 80, 0.66 at 160, 0.95 at 240, 1.31 at 320). Of forty trials with a real effect, 29 cross for benefit and 11 are stopped for futility.

Exact. At an interim |z| of 1.0: end only 3.64%, +0.6 3.41%, +0.75 3.20%, +0.9 3.41%, every 0.125 3.00%, every 0.05 2.88%. At 2.5: end only 38.30%, +0.6 45.80%, +0.75 45.36%, +0.9 41.70%, every 0.125 50.08%, every 0.05 53.24%. The heavy line is the largest of the six at each z.