The same 40 units, arranged two ways

Both designs estimate the same effect of 0.5 and both are unbiased — 0.488 and 0.497. The blocked design's estimate has standard deviation 0.318 against 0.692, a variance ratio of 0.21 where the model predicts 0.20.

Over four hundred pilots each, the fitted stable share has a median of 0.40 when the truth is 0.4 and 0.68 when it is 0.7; the two histograms share 37% of their mass. A pilot drawn from the fading model is fitted with a share above 0.6 14% of the time.

The truth's cheapest schedule is 1 follow-up. Choosing from a pilot of ten patients costs on average 9.1% more at 3 visits, 8.0% more at 4 visits, 6.8% more at 6 visits; from a pilot of 160, 1.10%, 0.75%, 0.57%. The minimax schedule, 2 follow-ups chosen with no pilot, costs 2.9% more.

Internal pilots measured on two follow-ups, the schedule that needs no pilot, whose correlation curve chooses the schedule for every later patient. Extra cost over the cheapest schedule at stable shares 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7: no pilot, two follow-ups throughout, 11.1%, 9.8%, 8.1%, 5.9%, 2.9%, 0.0%, 0.0%, 3.2%; an internal pilot of 20, 10.5%, 9.8%, 9.3%, 7.9%, 5.9%, 3.7%, 4.2%, 5.2%; an internal pilot of 40, 5.9%, 5.7%, 5.6%, 4.7%, 3.4%, 2.0%, 2.4%, 2.5%; an internal pilot of 80, 3.6%, 3.4%, 3.2%, 2.8%, 1.9%, 1.0%, 1.6%, 1.6%. Worst cases: no pilot 11.1%, 20 patients 10.5%, 40 patients 5.9%, 80 patients 3.6%.

The worst extra cost over the cheapest schedule across every correlation curve with adjacent correlation 0.7. No pilot, two follow-ups throughout: 11.1%. Pilot patients on two follow-ups, powered for 0.2: 10.5%, 5.9%, 3.6%. Pilot patients on three follow-ups, powered for 0.2: 9.0%, 5.8%, 5.5%. Pilot patients on two follow-ups, powered for 0.1: 10.4%, 5.4%, 2.2%. Pilot patients on two follow-ups, powered for 0.3: 10.5%, 6.6%, 6.0% — at pilots of 20, 40, 80 patients.

Stable share 0, cheapest 1 follow-up: 1, 75%; 2, 16%; 3, 5%; 4, 0%; five or more, 4%. Stable share 0.4, cheapest 1 follow-up: 1, 56%; 2, 24%; 3, 18%; 4, 0%; five or more, 2%. Stable share 0.6, cheapest 2 follow-ups: 1, 24%; 2, 29%; 3, 45%; 4, 1%; five or more, 1%. Stable share 0.7, cheapest 3 follow-ups: 1, 7%; 2, 15%; 3, 78%; 4, 0%; five or more, 0%.

Four thousand internal pilots of forty patients, measured on two follow-ups, with a true effect of 0.5. The pilot's effect estimate averages 0.503 when it chose 1 follow-up (56.0% of pilots), 0.514 when it chose 2 follow-ups (23.6% of pilots), 0.503 when it chose 3 follow-ups (17.1% of pilots), 0.505 when it chose 4 follow-ups (1.0% of pilots), 0.456 when it chose 8 follow-ups (2.0% of pilots); its standard deviation across pilots is 0.248. The correlation between the schedule chosen and the effect estimate is −0.021.

Pilots at four visits a patient, against taking the minimax schedule with no pilot, when the true stable share is 0.4. At an effect of half a standard deviation the net is -78 visits for 40 patients, -151 visits for 80 patients, -310 visits for 160 patients. At an effect of 0.05 it is 90, 675, 663.

Both designs estimate the same effect of 0.5 and both are unbiased — 0.488 and 0.497. The blocked design's estimate has standard deviation 0.318 against 0.692, a variance ratio of 0.21 where the model predicts 0.20.

All 12,870 assignments, enumerated. The spread of the standardised imbalance is exactly 2/√n = 0.500, whatever the covariate's own distribution, and 33.3% of assignments differ by more than 0.5 standard deviations. Randomisation does not deliver balance; it delivers a known distribution of imbalance.

Under the null that the treatment does nothing, every unit's value is what it is whichever arm it landed in — so the 252 possible assignments give 252 possible differences, and that set IS the null distribution. Rejecting the most extreme 5% of them gives a test whose size is 4.76%, counted rather than assumed.

Both designs spend the same number of runs. The factorial estimates every main effect from every run; one-at-a-time estimates each from two conditions. The ratio of variances is (k+1)/2 — 2.0 at 3 factors — measured here at 2.00 over 2,000 experiments.

The curve is the non-central t on 2n − 2 degrees of freedom with δ = d√(n/2); the dots are 4,000 experiments run at each size. Reaching 80% power needs 64 per arm.

Four thousand pilots of 10 observations, each sizing a trial for 80% power at half a standard deviation from its own standard deviation. 55.9% of the trials have less than 80% power and 11.1% less than 50%; the median trial has 76.8%.

From the pilot's own standard deviation, 55.9% of trials fall short of 80% power at an average size ×0.99 of the known-spread size. From its 80% upper limit, 19.8% fall short at ×1.65; from its 90% limit, 10.0% at ×2.12.

With the effect known, 80% is reached at 63 per arm. With the effect uncertain by 0.25 standard deviations it takes 113; by 0.5, 1268; by 0.75, no sample size at all, because the chance can never exceed the 74.8% prior probability that the effect is positive.

For an expected effect of half a standard deviation: 63 per arm with the effect known, 113 at an uncertainty of 0.25, 1268 at 0.5, and no finite number past 0.594, where the prior probability of a positive effect falls to 80%.

For a small shift, a cut at the mean keeps 63.7% of the information, so the trial needs 1.57 times the sample. A cut at the top tenth keeps 34.2% and needs 2.92 times; a cut two standard deviations out keeps 13.1%.

The outcome as measured reaches 80% power at 63 per arm; cut at the control mean, at 102, 1.62 times as many.

Sixty-four per arm, no difference between the arms, candidate cuts spread between one standard deviation below and above the mean, and the most significant comparison reported. With one cut fixed in advance the false-positive rate is 4.2%; choosing among five it is 17.7%, and among seventeen 27.4%.

The treated arm is the control arm moved half a standard deviation, for every patient. Above the threshold lie 15.9% of control patients and 30.9% of treated ones — a difference of 15.0% that reads as a share of patients who respond, when every patient responded by the same amount.

Adjusting for the covariate as measured needs one minus the squared correlation of the unadjusted sample; cut at the median, one minus 2/π of the squared correlation. At a correlation of 0.7 those are 0.510 and 0.688; cut in three, 0.611; the change from baseline, 0.600. The change score needs fewer patients than the median-cut adjustment once the correlation passes 0.624.

Adjusting for a median cut rather than the covariate as measured needs 1.036 times the patients at a correlation of 0.3, 1.121 at 0.5, 1.349 at 0.7 and 2.549 at 0.9. Cut in three: 1.199 at 0.7 and 1.881 at 0.9.

A cut at the median keeps 2/π, 63.7%, of the variance the covariate would remove; one at the top tenth keeps 34.3%. At a correlation of 0.7 that is 1.349 and 1.631 times the patients the covariate as measured needs.

For 80% power at an effect of half a standard deviation the unadjusted analysis needs 63 patients an arm, the change from baseline 38, adjustment for the median cut 44 and adjustment for the covariate as measured 33.

Reported against actual variance of the treatment estimate, as shares of the unadjusted analysis's, over 20,000 trials of 50 patients an arm. Adjusted for the baseline as measured: 0.529 reported, 0.518 actual. Median cut fixed in advance: 0.716 and 0.700. The decile cut with the best fit: 0.671 and 0.736. The decile cut with the smallest p-value: 0.762 and 1.615.

Among the sample's nine deciles, the cut that leaves the smallest residual variance is the median on 21.8% of trials and one of the two outermost deciles on 1.5%; the five middle deciles take 88.1%.

At ten patients an arm the best-fitting cut rejects a true null 9.31% of the time and the smallest-p cut 16.46%; at two hundred, 5.03% and 14.27%. The median fixed in advance holds 4.78% and 4.56%.

In its own sample the best-fitting decile cut appears to keep 1.044 of the variance the baseline as measured removes at a correlation of 0.3, 0.724 at 0.7 and 0.665 at 0.9; the median fixed in advance appears to keep 0.625, 0.631 and 0.632, against 2/π, 0.637, in the population.

At a correlation of 0.5 between visits, the adjusted analysis has 0.750 of the unadjusted variance with one visit before treatment and one after, 0.667 with two before, 0.500 with two after and 0.417 with two of each.

With ρ the correlation between visits, a second follow-up visit removes (1 − ρ)/2 of the variance and a second baseline visit removes the square of ρ times (1 − ρ)/(1 + ρ). At ρ = 0.5 those are 0.2500 and 0.0833; at 0.8, 0.1000 and 0.0711. The baseline's contribution peaks at 0.0902 near ρ = 0.62.

A stable patient level carries 0.4 of the variance and adjacent visits correlate 0.7. With one baseline the adjusted variance is 0.510. Adjusting for the mean of two baselines gives 0.540 and of six 0.608; adjusting for each separately gives 0.503 and 0.486.

Under compound symmetry the cheapest design takes 3 follow-up visits, at 0.782 of the cost of one; with a stable level carrying 0.4 of the variance and the rest fading between visits, it takes 1, and 8 follow-ups cost 1.421 of one.