Stopping rules
When the looking happens
A p-value is defined relative to a sampling plan, so the same data means different things under different stopping rules. Testing five times at the nominal level rejects a true null 14% of the time, and no observation in the dataset changed.
Spending the error rate
The repair for interim testing is to spend 5% across the looks rather than at each one. The boundaries are solvable rather than quotable, and a trial that can stop early uses 298 observations where a fixed design uses 400 — at a cost of half a point of power.
The effect a stopped trial reports
An O'Brien–Fleming trial at 88.45% power holds its error rate exactly and reports an effect 9.6% too large on average. The 11.39% of trials that stop at the second look report 1.83 times the truth, the ones that cross at the last look report 0.80 times it, and pooling every trial by its size gives the truth back to the last digit.
The outcomes a trial could have stopped with
A trial that stops at its second look with z = 3.3 has a two-sided p-value of 0.000969, 0.000987, 0.00187 or 0.0421, depending on how the outcomes it could have stopped with are ordered. One of the four orderings does not change when the looks the trial never reached are replanned, and the same one gives a trial that ran to the end with z = 6 a p-value of 0.0256.
A boundary for giving up
Adding "stop if z is below zero" to an O'Brien–Fleming trial costs 5.20 points of power at the effect it was designed for and halves the observations a trial with no effect uses. Stopping when conditional power at the observed trend falls under 10% costs 13.23 points and stops 21.28% of trials with a real effect. Making that rule binding lowers the benefit boundary from 2.040 to 1.901, and a binding rule that is then ignored rejects a true null 3.523% of the time instead of 2.5%.
A look the trend asked for
Under an O'Brien–Fleming-type spending function, every schedule of looks fixed in advance spends exactly 5.0000%. A committee that adds a look at three quarters of the trial whenever the interim z is 1.5 or more spends 5.2323% — 5.315% counted over a hundred thousand trials — and the most a committee choosing among six schedules could spend is 5.4390%.
A look added by a rule
A monitoring committee's rule — add a look at three quarters of the trial when the interim |z| is at least 1.5 — spends 5.2323% under O'Brien–Fleming-type boundaries. Written into the protocol it can be given boundaries that spend exactly 5%, three ways: raise the added look's boundaries by 2.05%, raise every later boundary by 0.99%, or preserve at each interim value the chance of crossing the plan would have given. All three cost the same, 0.38 points of power, from 88.94% to 88.56%, because the power the uncorrected rule seemed to keep was the excess error. Made exact, the rule is a design between the two fixed schedules — and at every trigger slightly better than randomising between them.
The variance that contains the effect
A monitoring committee that adds an interim look because the variance came in higher than planned is reading a nuisance quantity, and a look added on a nuisance quantity should leave the spending function's 5% alone. It does, exactly, when the variance is the within-arm one, and when the trigger is the calendar. It does not when the variance is the one computed with the arm labels hidden: that variance is the within-arm sum of squares plus the interim statistic squared, so it is high more often when the trend is strong. The excess is 0.069 points at a twenty-patient interim and falls as one over the square root of the interim's size — a tenth of what a trigger on the trend itself spends, and never zero.