Normal Distribution
nothing is called 95% until it has been counted
Field

What a sample-size calculation was given

A sample-size calculation takes a standard deviation, an effect and an outcome as given, and each is usually a choice or an estimate. Sized from a pilot of ten's standard deviation, 55.9% of trials have less than their planned 80% power and 11.1% less than half, although the planned sample is right on average; sizing from the pilot's 80% upper confidence limit leaves 19.8% short at 1.65 times the patients. With the effect believed to be half a standard deviation give or take a quarter, a trial of sixty-four per arm succeeds 69.2% of the time, and no trial of any size can succeed more often than the planners believe the effect is positive. Cutting the outcome at a threshold keeps at most 63.7% of its information, and a responder threshold one standard deviation out needs twice the patients.
The power trials actually have when sized for 80% from a pilot of 10. Four thousand pilots of 10 observations, each sizing a trial for 80% power at half a standard deviation from its own standard deviation. 55.9% of the trials have less than 80% power and 11.1% less than 50%; the median trial has 76.8%.

The spread a pilot supplies

A trial sized for 80% power from a pilot's standard deviation is sized from an estimate that is too small more often than not. With a pilot of ten, 55.9% of the trials it sizes have less than 80% power and 11.1% less than 50%, although the planned sample is right on average. Sizing from the pilot's 80% upper confidence limit instead leaves 19.8% short, at 1.65 times the sample; from its 90% limit, 10.0% short at 2.12 times.

 The chance a trial succeeds against its size, when the expected effect of 0.5 is uncertain by four amounts. With the effect known, 80% is reached at 63 per arm. With the effect uncertain by 0.25 standard deviations it takes 113; by 0.5, 1268; by 0.75, no sample size at all, because the chance can never exceed the 74.8% prior probability that the effect is positive.

The chance a trial succeeds

A trial of sixty-four per arm has 80% power at an effect of half a standard deviation. If the effect is only believed to be about half a standard deviation, give or take a quarter, the chance the trial reaches significance is 69.2%; give or take a half, 61.4%. Reaching 80% then takes 113 per arm, or 1,268 — and when the belief is uncertain by three quarters of a standard deviation no number of patients reaches 80%, because the chance can never exceed the 74.8% probability that the effect is positive at all.

 How much of a normal outcome's information survives cutting it into two, by where the cut is. For a small shift, a cut at the mean keeps 63.7% of the information, so the trial needs 1.57 times the sample. A cut at the top tenth keeps 34.2% and needs 2.92 times; a cut two standard deviations out keeps 13.1%.

An outcome cut in two

Replacing a measured outcome with whether it crossed a threshold keeps 63.7% of the information when the cut is at the mean, 34.2% at the top tenth and 13.1% two standard deviations out. A trial that needs 63 patients per arm on the measured outcome needs 102 cut at the mean and 185 cut at one and a half standard deviations. The responder rates that result read as a share of patients who respond — 50.0% against 69.1% — when every patient moved by the same amount; and a cut chosen after looking turns a 5% test into a 17.7% one.

 The sample a trial needs for the same power, as a share of the unadjusted analysis's, by how strongly a baseline covariate predicts the outcome. Adjusting for the covariate as measured needs one minus the squared correlation of the unadjusted sample; cut at the median, one minus 2/π of the squared correlation. At a correlation of 0.7 those are 0.510 and 0.688; cut in three, 0.611; the change from baseline, 0.600. The change score needs fewer patients than the median-cut adjustment once the correlation passes 0.624.

A baseline cut in two

Adjusting a trial's result for a baseline measurement that predicts the outcome shrinks the sample it needs to 1 − ρ² of the unadjusted one. Adjusting for whether that measurement was above its median shrinks it only to 1 − (2/π)ρ² — the cut keeps 63.7% of what the covariate could remove, exactly the fraction a cut outcome keeps. In patients the loss grows with the covariate's strength: a cut costs 12% more patients at a correlation of 0.5, 35% at 0.7 and 2.55 times as many at 0.9, where a simple change from baseline would have done better than the cut adjustment.

 What four adjusted analyses report about their own precision and what they have, 50 patients an arm, baseline correlated 0.7. Reported against actual variance of the treatment estimate, as shares of the unadjusted analysis's, over 20,000 trials of 50 patients an arm. Adjusted for the baseline as measured: 0.529 reported, 0.518 actual. Median cut fixed in advance: 0.716 and 0.700. The decile cut with the best fit: 0.671 and 0.736. The decile cut with the smallest p-value: 0.762 and 1.615.

The cut that fitted best

A trial that adjusts for its baseline cut at whichever of the sample's nine deciles fits the outcomes best — without ever looking at the treatment difference — reports a variance 0.671 of the unadjusted analysis's where its estimates actually have 0.736, which is worse than the 0.700 a median fixed in advance delivers. The sample says the chosen cut kept 72.4% of what the baseline could remove; the population says it kept less than a median. A true null is rejected 5.87% of the time at fifty patients an arm and 9.31% at ten, and a cut chosen for its p-value rejects 14.22%.

 The variance of the baseline-adjusted estimate for four arrangements of visits, against the correlation between visits. At a correlation of 0.5 between visits, the adjusted analysis has 0.750 of the unadjusted variance with one visit before treatment and one after, 0.667 with two before, 0.500 with two after and 0.417 with two of each.

A visit before or a visit after

A trial that can afford one more measurement per patient can take it before treatment, to sharpen the baseline adjustment, or after it, to average the outcome. With every pair of visits correlated 0.5, the second visit after treatment removes a quarter of the variance and the second visit before it removes a twelfth, and it is never the other way round: the baseline's contribution is ρ²(1 − ρ)/(1 + ρ), which peaks at ρ⁵ = 0.090 when ρ is the golden ratio's reciprocal. Below a correlation of one half, a single extra follow-up beats any number of baselines. When the correlation fades with the time between visits, averaging an earlier baseline into the adjustment makes the trial less precise, not more.

 The extra cost of each follow-up schedule over the cheapest, across every correlation curve with adjacent visits at 0.7, when a patient costs 5 visits to recruit. With one baseline, 1 follow-up costs at most 28.0% more than the cheapest schedule, 2 follow-ups cost at most 11.1% more than the cheapest schedule, 3 follow-ups cost at most 24.1% more than the cheapest schedule, 4 follow-ups cost at most 34.8% more than the cheapest schedule, over every stable share from 0 to 0.7. The schedule whose worst case is smallest is 2 follow-ups; its worst case is at a stable share of 0.00. Under compound symmetry the cheapest is 3; under a stable share of 0.4 it is 1.

The correlation a pilot can see

Two models of repeated visits agree that adjacent visits correlate 0.7 and disagree about how many follow-ups a trial should buy: three under equal correlation, one when the correlation fades. Planning on the wrong one costs 28.0% or 9.9%. Two follow-ups cost at most 3.2% more than the best under either, and at most 11.1% under any curve with that adjacent correlation. A pilot of twenty patients at four visits chooses worse on average than that rule when the correlation fades; it takes about forty to draw level with it, and at an effect of half a standard deviation those forty patients' extra visits cost fifty times what the better schedule saves.

 What a trial's own first patients buy when they choose the follow-up schedule for the rest, powered for an effect of 0.2. Internal pilots measured on two follow-ups, the schedule that needs no pilot, whose correlation curve chooses the schedule for every later patient. Extra cost over the cheapest schedule at stable shares 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7: no pilot, two follow-ups throughout, 11.1%, 9.8%, 8.1%, 5.9%, 2.9%, 0.0%, 0.0%, 3.2%; an internal pilot of 20, 10.5%, 9.8%, 9.3%, 7.9%, 5.9%, 3.7%, 4.2%, 5.2%; an internal pilot of 40, 5.9%, 5.7%, 5.6%, 4.7%, 3.4%, 2.0%, 2.4%, 2.5%; an internal pilot of 80, 3.6%, 3.4%, 3.2%, 2.8%, 1.9%, 1.0%, 1.6%, 1.6%. Worst cases: no pilot 11.1%, 20 patients 10.5%, 40 patients 5.9%, 80 patients 3.6%.

A schedule the trial chooses for itself

A separate pilot cannot pay for itself in choosing a trial's follow-up schedule, because its patients are recruited, measured and discarded. A trial can instead measure its own first patients on the schedule that needs no pilot — two follow-ups — fit the correlation curve to them, and give everyone after them the schedule that curve makes cheapest. Those first patients are counted in the analysis and cost no extra visit. Forty of them cut the trial's worst-case excess cost across every curve with adjacent correlation 0.7 from 11.1% to 5.9% in a trial of two hundred an arm, and eighty cut it to 3.6%; twenty buy almost nothing. And because the curve is read from within-arm deviations, the choice cannot see the effect: the pilot's effect estimate is the same whichever schedule it chose.

All essays