Normal Distribution
nothing is called 95% until it has been counted
Concept

Pilot study — where it appears

A small run made before the main experiment in order to estimate something the main experiment's design depends on. Everything the main experiment's design depends on is then a guess with a standard error, and the design's efficiency is a function of a quantity that was estimated on a handful of runs.

Named by 8 essays across 4 fields — each of them below, with the objects they name alongside it.

The power trials actually have when sized for 80% from a pilot of 10. Four thousand pilots of 10 observations, each sizing a trial for 80% power at half a standard deviation from its own standard deviation. 55.9% of the trials have less than 80% power and 11.1% less than 50%; the median trial has 76.8%.

The spread a pilot supplies

A trial sized for 80% power from a pilot's standard deviation is sized from an estimate that is too small more often than not. With a pilot of ten, 55.9% of the trials it sizes have less than 80% power and 11.1% less than 50%, although the planned sample is right on average. Sizing from the pilot's 80% upper confidence limit instead leaves 19.8% short, at 1.65 times the sample; from its 90% limit, 10.0% short at 2.12 times.

planned · Power The chance a trial succeeds against its size, when the expected effect of 0.5 is uncertain by four amounts. With the effect known, 80% is reached at 63 per arm. With the effect uncertain by 0.25 standard deviations it takes 113; by 0.5, 1268; by 0.75, no sample size at all, because the chance can never exceed the 74.8% prior probability that the effect is positive.

The chance a trial succeeds

A trial of sixty-four per arm has 80% power at an effect of half a standard deviation. If the effect is only believed to be about half a standard deviation, give or take a quarter, the chance the trial reaches significance is 69.2%; give or take a half, 61.4%. Reaching 80% then takes 113 per arm, or 1,268 — and when the belief is uncertain by three quarters of a standard deviation no number of patients reaches 80%, because the chance can never exceed the 74.8% probability that the effect is positive at all.

planned · Power One experiment finding out where to look. A single run of the fully sequential design: 40 runs, the first 8 placed at the guess K = 1, then the model refitted and the design revised after every 2. The marks are the settings the runs were made at. The horizontal lines are where a design built at the truth K = 3 would have put them — 1.875 and 10.00 — and the rule walks onto them without being told: its estimate of K after the first eight runs was 2.694, and by the end 2.765 against a truth of 3. The whole experiment is 96.5% as efficient as the design that knew the answer, where running all 40 at the guess would have been 81.1%.

The design that stops guessing

Every repair so far protects a guess. The alternative is to run part of the experiment, estimate the parameter from it, and design the rest at the estimate — which recovers most of what a threefold wrong guess costs, and has a best moment to stop guessing that is earlier than anyone expects.

robust · Local design What a pilot buys, σ = 1 against 3. Each point is 6,000 two-stage experiments of 100 units: a pilot of m per arm, then the rest split by the pilot's own estimate of the two spreads. Above the line the pilot has made the experiment worse than not bothering. The best pilot here is 8 per arm at 0.809, against 0.800 for a designer who knew the spreads — so the rule recovers 96% of what knowing them is worth. A larger pilot estimates the ratio better and has less left to apply it to, which is why the curve turns.

Allocating on a guess

Every allocation rule in this field is a function of quantities the experiment is being run to find out. Fed a pilot's estimate of them, the rule that minimises the variance makes the experiment worse than not bothering — until the arms differ by about a factor of two, which is further than anyone would guess.

allocation · Allocation What the guess is worth, when it is worth anything. The variance cost of an even split relative to the variance-minimising one for a risk difference, against the first arm's proportion, with the second at 0.3. The cost is a pure number: it does not depend on the trial's size. It is exactly zero at 0.3 and at 0.70, where the two arms have the same p(1 − p); it is 0.19% at a half and 4.36% at a tenth. Across the whole range from a tenth to nine tenths it never exceeds 4.36%, which is what the variance-minimising rule is worth here — and what it is worth is the reason it is safe to use with a guess.

The arm whose variance is its answer

With a binary outcome the allocation rule is a function of the proportions the trial exists to estimate. It costs at most 4.36% of variance to ignore it anywhere between a tenth and nine tenths, because √(p(1−p)) stays within a factor of two of its peak across 98% of the unit interval.

allocation · Allocation What two exact companions are worth to a simulated rate: the lognormal t test's size post-stratified three ways, by the number of cells. To first order, bands of the logarithms' t statistic are worth at most 2.77 times the draws; a 32 × 32 grid in the pair 4.68; bands of a score fitted to the pair on a thousand samples 4.64 at sixteen bands and 4.69 at sixty-four. Counted over runs of 4,000 samples, the grid of 1,024 cells was worth 1.97, with 94 cells a run left empty.

A companion with two coordinates

The logarithms' t statistic, cut into bands, is worth at most 2.77 times the draws to a simulation of the lognormal t test's size. It is one function of two exactly known statistics, the logarithms' mean and spread, and the pair is worth 4.64 when a score fitted to it on a pilot of a thousand samples is cut into sixteen bands whose probabilities are integrated exactly. A grid of cells in the pair needs a thousand cells to reach the same and, counted on runs of four thousand samples, is worth 1.97. Band weights estimated from a pilot instead of computed are worth less than the pilot's draws would have been had they simply been counted — exactly nothing at a pilot the size of the run.

splits · Routes The extra cost of each follow-up schedule over the cheapest, across every correlation curve with adjacent visits at 0.7, when a patient costs 5 visits to recruit. With one baseline, 1 follow-up costs at most 28.0% more than the cheapest schedule, 2 follow-ups cost at most 11.1% more than the cheapest schedule, 3 follow-ups cost at most 24.1% more than the cheapest schedule, 4 follow-ups cost at most 34.8% more than the cheapest schedule, over every stable share from 0 to 0.7. The schedule whose worst case is smallest is 2 follow-ups; its worst case is at a stable share of 0.00. Under compound symmetry the cheapest is 3; under a stable share of 0.4 it is 1.

The correlation a pilot can see

Two models of repeated visits agree that adjacent visits correlate 0.7 and disagree about how many follow-ups a trial should buy: three under equal correlation, one when the correlation fades. Planning on the wrong one costs 28.0% or 9.9%. Two follow-ups cost at most 3.2% more than the best under either, and at most 11.1% under any curve with that adjacent correlation. A pilot of twenty patients at four visits chooses worse on average than that rule when the correlation fades; it takes about forty to draw level with it, and at an effect of half a standard deviation those forty patients' extra visits cost fifty times what the better schedule saves.

planned · Power What a trial's own first patients buy when they choose the follow-up schedule for the rest, powered for an effect of 0.2. Internal pilots measured on two follow-ups, the schedule that needs no pilot, whose correlation curve chooses the schedule for every later patient. Extra cost over the cheapest schedule at stable shares 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7: no pilot, two follow-ups throughout, 11.1%, 9.8%, 8.1%, 5.9%, 2.9%, 0.0%, 0.0%, 3.2%; an internal pilot of 20, 10.5%, 9.8%, 9.3%, 7.9%, 5.9%, 3.7%, 4.2%, 5.2%; an internal pilot of 40, 5.9%, 5.7%, 5.6%, 4.7%, 3.4%, 2.0%, 2.4%, 2.5%; an internal pilot of 80, 3.6%, 3.4%, 3.2%, 2.8%, 1.9%, 1.0%, 1.6%, 1.6%. Worst cases: no pilot 11.1%, 20 patients 10.5%, 40 patients 5.9%, 80 patients 3.6%.

A schedule the trial chooses for itself

A separate pilot cannot pay for itself in choosing a trial's follow-up schedule, because its patients are recruited, measured and discarded. A trial can instead measure its own first patients on the schedule that needs no pilot — two follow-ups — fit the correlation curve to them, and give everyone after them the schedule that curve makes cheapest. Those first patients are counted in the analysis and cost no extra visit. Forty of them cut the trial's worst-case excess cost across every curve with adjacent correlation 0.7 from 11.1% to 5.9% in a trial of two hundred an arm, and eighty cut it to 3.6%; twenty buy almost nothing. And because the curve is read from within-arm deviations, the choice cannot see the effect: the pilot's effect estimate is the same whichever schedule it chose.

planned · Power

Named alongside it

The objects these essays reach for when they reach for this one.

Sample sizeExperimental designStatistical powerMonte CarloVariance reductionAllocation ratioAnalysis of covarianceAutocorrelationClosed formMaximin designPlug in estimateStandard deviation

All concepts