Statistical power — where it appears
Named by 91 essays across 38 fields — each of them below, with the objects they name alongside it.
Choosing n after looking
Re-estimating the sample size from an interim is the one adaptation with a defence, and the defence is exactly what it costs: an analyst kept blind to the arms measures a spread that contains the effect, so the design overshoots by 1 + Δ²/4σ². Re-estimating the effect instead breaks the error rate.
Not half and half
The same units, the same measurements, the same analysis — and a different variance, decided before anything is measured. When the two arms have different spreads the best split is σ₁ : σ₂, equal allocation costs 2(σ₁²+σ₂²)/(σ₁+σ₂)², and at three to one that is a quarter of the experiment.
The design that cannot see a curve
A two-level factorial has every run at a corner, where every squared term equals one — so the column that would estimate curvature is a copy of the intercept, and the design has no information about it at all. A few runs at the centre buy one number back, and only one.
The spread a pilot supplies
A trial sized for 80% power from a pilot's standard deviation is sized from an estimate that is too small more often than not. With a pilot of ten, 55.9% of the trials it sizes have less than 80% power and 11.1% less than 50%, although the planned sample is right on average. Sizing from the pilot's 80% upper confidence limit instead leaves 19.8% short, at 1.65 times the sample; from its 90% limit, 10.0% short at 2.12 times.
Five times in six
A 95% interval is read as a 95% chance that a replication's estimate will land inside it. With the spread known and a replication of the same size, the chance is 83.42% — five times in six — because both estimates are uncertain. An original that landed two standard errors from the truth captures a replication 48.40% of the time; a replication a tenth the size lands inside 44.54% of the time; and among significant originals from studies with 17% power, 66.94%.
A probe nobody chose
On a set that is definitively in two pieces, seven of twenty-four outcomes report nothing at all. Every covariate probe reports it. What separates them is not accuracy — it is that one of them can be chosen and the other is what happened.
A taper and a critical value
Two constructions whose tapers visibly differ give the same critical value, and two that share a taper exactly do not. Adding a construction whose taper is a decision rather than an accident says which half of that is true.
Randomising towards the winner
Allocating more patients to the arm that is doing better is the humane thing to want and it buys nothing statistically: at a fixed total it costs thirty points of power. And because the allocation is a function of the outcomes, the ordinary test on it rejects a true null 7.8% of the time before any time trend is applied — and 58% after one.
The charge that is not a sum
Charging two searches what each costs on its own is conservative, and conservative here means the test never fires. At the largest break measured it declares nothing, on every draw, while a calibrated threshold reaches 29%.
What a p-value does not say
The same p of 0.04 corresponds to a large effect in ten observations and a negligible one in two thousand. A p-value alone cannot be interpreted, and the number that makes it interpretable is almost never printed beside it.
When one model contains the other
The comparison a forecaster most often wants is between a model and the same model with one more term. That is exactly the comparison the standard test cannot make — and it fails by declaring the smaller model significantly better, more confidently the more data it is given.
The two worlds that look the same
Three causal structures were fitted to one covariance matrix and agree with it to 4.4·10⁻¹⁶. The regression returns 0.5000 under all three; the effect they hold is 0.5000, 0.8481 and 0.8481. What separates structures is a missing edge, and the signature of one is a correlation of exactly zero.
What naming it in advance costs
Preregistration is argued for as free. Against an effect of two standard errors hiding in one of twenty analyses, naming the right one detects it 51.5% of the time and naming the wrong one detects it 4.7% of the time; correcting all twenty detects it 22.5% wherever it is. The two are worth the same when the chance of having named correctly is 38%.
Two intervals that overlap
Two 95% intervals that just touch are read as a difference at the edge of significance. With equal standard errors their difference has p = 0.0056, not 0.05; two intervals can overlap by 58.6% of an arm and still differ at exactly 5%; standard-error bars that just touch mark p = 0.157; and when the two estimates are correlated at 0.8, touching intervals conceal a difference of 6.2 standard errors. Read as a test, non-overlap needs 1.66 times the sample for the same power.
The chance a trial succeeds
A trial of sixty-four per arm has 80% power at an effect of half a standard deviation. If the effect is only believed to be about half a standard deviation, give or take a quarter, the chance the trial reaches significance is 69.2%; give or take a half, 61.4%. Reaching 80% then takes 113 per arm, or 1,268 — and when the belief is uncertain by three quarters of a standard deviation no number of patients reaches 80%, because the chance can never exceed the 74.8% probability that the effect is positive at all.
One control, many arms
The control appears in every comparison, so it is worth √k treatment arms — and the same sharing makes the k tests correlated at n/(n+n₀), which is the quantity Bonferroni ignores. Both facts come out of one design decision, and it is the size of the control.
The analysis after three arms
An unadjusted analysis after a two-arm balancing rule rejects 0.6% of true nulls where it claims 5%. With three arms and a deterministic rule it rejects none at all — and the repair is the same repair, which is a sentence and a column in the model.
The analysis has to know the rule
A trial balanced by minimisation and analysed by comparing the two arms' means rejects a true null 0.6% of the time where it claims 5%, and at full determinism 0.0%. That is not an error anybody complains about — it is a test that has stopped working, paid for by a balance the analysis then refused to use.
The models that were never in the running
A reference distribution for a set has to assume something about every candidate in it. Assuming that all of them are as good as the benchmark is what makes the reality check honest, and it is what sixteen hopeless candidates use to destroy it.
The price of control
Every correction is paid for in power, and the exchange rate can be measured. Holm buys familywise control for 33 percentage points of power; Benjamini–Hochberg buys a weaker guarantee for 10. Neither is free and neither is a matter of taste.
The winner's curse
Filter honest studies down to the ones that reached significance and the effects they report are systematically too large. At low power the inflation is a factor of two, nobody has done anything wrong, and the selection did all of it.
What a chosen probe finds
On a chain of eight hundred draws the probe the earlier fields use misses 44% of the sets that are split. Its own residual off the rule's span misses 12%, for one least-squares fit.
What the balanced trial is worth
A rule that reads the covariate removes three quarters of the imbalance. An analysis that does not know it happened prices the imbalance anyway, rejects one true null in two hundred instead of one in twenty, and finds a real effect less often than a coin-tossed trial does.
A coverage table with its own error
Twenty cells estimating the coverage of an interval that is exactly 95%, at a thousand replications each, read from 93.9% to 96.5% — and a table like that flags at least one of its correct cells on 69.9% of honest runs. Ten times the replications does not repair it: at ten thousand the same table still flags one 63.3% of the time.
The effect a stopped trial reports
An O'Brien–Fleming trial at 88.45% power holds its error rate exactly and reports an effect 9.6% too large on average. The 11.39% of trials that stop at the second look report 1.83 times the truth, the ones that cross at the last look report 0.80 times it, and pooling every trial by its size gives the truth back to the last digit.
The correction that makes the estimate worse
Correcting for twenty analyses repairs the p-value by demanding a larger statistic, and a larger statistic is a more selected one. At two standard errors the surviving estimate averages 1.35 times the truth before the correction and 1.69 times it after — so the honest error rate is bought with a more inflated effect.
Significant in one, not in the other
Two studies of exactly the same effect, each with 50% power, disagree about significance half the time — and when they do, the test of the difference between them is significant in 9.75% of cases. A p of 0.01 beside a p of 0.20 is a difference with p = 0.36. Among four subgroups sharing one effect, at least one significant and one not happens 87.5% of the time, and the test that would tell a real difference apart needs four times the sample the effect itself needed.
An outcome cut in two
Replacing a measured outcome with whether it crossed a threshold keeps 63.7% of the information when the cut is at the mean, 34.2% at the top tenth and 13.1% two standard deviations out. A trial that needs 63 patients per arm on the measured outcome needs 102 cut at the mean and 185 cut at one and a half standard deviations. The responder rates that result read as a share of patients who respond — 50.0% against 69.1% — when every patient moved by the same amount; and a cut chosen after looking turns a 5% test into a 17.7% one.
A distribution drawn from the null
Between nested models the ordinary comparison statistic has a null distribution centred at minus one and a 95% point of a quarter. A correction to its mean repairs the centre and leaves the shape; simulating the null repairs both.
How many subjects
Sixty-four per arm for 80% power at half a standard deviation — a power figure that could only be simulated, with nothing to disagree with, until the non-central t was written. Two routes now, agreeing to within the simulation's own error.
The check before the standard error
One number decides whether every interval in an analysis is trustworthy, and the check for it flags a lag-one correlation of 0.5 nine times in ten — and one of 0.2 only one time in five, where the interval already covers 88.6% instead of 95%.
The corner the test is calibrated at
"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.
The reference the covariates supply
Hold the outcomes fixed, re-run the rule that assigned them, count. The same construction cost nineteen points of power in the adaptive field, because its rule chased outcomes and its critical value depended on a rate nobody has. Here the rule reads only what was recorded before anything happened, and the same unadjusted statistic goes from 20.3% power to 55.0% by being read against the right distribution.
What the exactness buys
Against a z test calibrated to reject exactly 5% of true nulls on this design, the randomisation test loses nineteen points of power. What it buys is that the calibration needs the success rate — which moves the critical value from 1.668 to 2.718 and is the quantity the trial was run to find out.
When the constraints run out
Every function added to a basis is a constraint the assignment has to satisfy with the same units. At sixteen units and a stated tolerance the admissible assignments run 3,874, then 1,006, then 314, then none — and the count is exact, because the assignment space is finite.
False discoveries that arrive together
Correlate twenty tests and Benjamini–Hochberg still holds its false discovery rate — 1.66% at a correlation of 0.9 with ten real effects, against 2.55% when the tests are independent. What changes is how the errors come. A family of true nulls reports anything 2.34% of the time instead of 5.08%, and when it does, it reports 16.56 false findings out of twenty.
The verdicts watch the threshold, not the gap
Two studies whose effects really differ are read through their verdicts as often as through their difference. Halves of an 80%-power study whose effects differ by the whole effect split 72.87% of the time against 49.99% when they do not — a split is never more than 1.86 times as likely under any difference up to twice the effect. And a halving of the effect in two large studies leaves both significant 97.93% of the time, while the test of the difference finds it 80.74%.
A baseline cut in two
Adjusting a trial's result for a baseline measurement that predicts the outcome shrinks the sample it needs to 1 − ρ² of the unadjusted one. Adjusting for whether that measurement was above its median shrinks it only to 1 − (2/π)ρ² — the cut keeps 63.7% of what the covariate could remove, exactly the fraction a cut outcome keeps. In patients the loss grows with the covariate's strength: a cut costs 12% more patients at a correlation of 0.5, 35% at 0.7 and 2.55 times as many at 0.9, where a simple change from baseline would have done better than the cut adjustment.
Naming a handful in advance
A preregistration that names three of twenty analyses sits between naming one and naming none, and whether it beats both depends on one thing: how much of the analyst's belief the three hold. An analyst who is 50% sure of a first choice, and whose belief halves down the list, gets 34.8% expected power from naming three, 28.3% from naming one and 22.7% from exploring all twenty. Spread the same 50% evenly over the rest and naming three is the worst of the three plans. The rule for adding one more to the list needs no belief at all: the next analysis must be at least 20.8% as likely as the first to be worth naming.
Two instruments that disagree
The overidentification test keeps its size at 5.0% and reaches 86.4% power against a violation carried by one instrument. Against the same error carried by both in proportion to their first stages it rejects on 4.6% of draws — its own size — while the estimate is wrong by 0.3000, which is 94.2% of the confounding the instruments were brought in to remove.
When the order matters
Three ways of breaking exchangeability cost 4.93, 11.07 and 1.07 points of coverage, and the ordering by cost is the reverse of the ordering by how soon a test would have caught them. The departure practitioners check for is the cheapest one.
Right for the wrong reason
A robust standard error costs no coverage where the risk is absent — 95.52% against 95.06% at twenty rows. It costs a 6.89% wider interval and a variance estimate 2.572 times as variable, and the pre-test that would avoid paying recovers 15.9% of what the insurance is worth.
The miscalibration a perfect forecaster shows
A forecaster whose true reliability is exactly zero shows a calibration error of 0.1252 on fifty forecasts and 0.0090 on ten thousand. Every one of 1,200 blameless hundred-forecast records exceeds the 0.02 routinely read as evidence of a problem, and the mean does not fall under it until 1,976 forecasts.
The p-value a replication gets
Under a true null a p-value is flat. Under a real effect its distribution is closed form and wide — a study with 80% power returns anything from 4.4×10⁻⁵ to 0.13 in eight runs of ten — and the chance that an exact replication of a p = 0.05 result is significant again is exactly one half, under both of the models people use without naming them.
A simulation that stops when it looks settled
A simulation of an interval that covers exactly 95%, checked every 250 replications for a significant departure and stopped when it finds one, flags that correct interval on 29.54% of runs. Stopped instead as soon as its estimate reaches 95%, it reports an interval that covers 94% as meeting its level on 37.21% of runs. Stopped when the estimate stops moving, it reports the right number — and has quietly chosen to run about fifteen hundred replications.
A boundary for giving up
Adding "stop if z is below zero" to an O'Brien–Fleming trial costs 5.20 points of power at the effect it was designed for and halves the observations a trial with no effect uses. Stopping when conditional power at the observed trend falls under 10% costs 13.23 points and stops 21.28% of trials with a real effect. Making that rule binding lowers the benefit boundary from 2.040 to 1.901, and a binding rule that is then ignored rejects a true null 3.523% of the time instead of 2.5%.
Estimating how many nulls are true
Benjamini–Hochberg at 5% delivers 2.55% when half of twenty nulls are false, because it cannot tell how many are. Storey's estimate of that share, read off the p-values above one half, spends the rest and finds 81.93% of the real effects instead of 74.70% on independent tests. Correlated at 0.9, the same procedure reports a finding in 19.29% of families in which every null is true.
The cliff that is a slope
A regression between two independent series is called significant 4.9% of the time at no persistence, 52.4% at a lag-one correlation of 0.9, and 83.4% at a unit root. The rule the field offers asks whether the last of those holds, and at 0.9 the unit-root test correctly refuses one 87.2% of the time.
The rank is a decision
The sequential procedure's 5% bounds one of its two errors. Over-counting reads between 4.2% and 7.2% at every sample length from fifty observations to three hundred; under-counting reads 69.5% at fifty and 0.0% at three hundred, and nothing in the procedure bounds it.
A subgroup inside its own trial
A trial with 80% power, significant overall and not in the fifth of it that is women, has told the reader almost nothing about women: that pattern turns up 57.45% of the time when women have the full effect and 58.26% when they have none. The subgroup is part of the whole, so the two verdicts are correlated, and the one test that answers the question — the subgroup against everyone else — can be recovered from the two printed intervals and nothing more.
A number for the shape
Five one-number summaries of a quantile plot's departure from its line, each a correct 5% test on forty normal observations, and no two agree about what matters. Straightness catches a skewed source 83% of the time and light tails 31%; kurtosis catches light tails 72% and skewness 0.2%. Reading all five and reporting whichever looks bad rejects 13.9% of genuinely normal samples. Correcting that search to 5% costs almost half the power against the departure it would have caught best — and a line-up of twenty panels gets the same correction for nothing.
The cut that fitted best
A trial that adjusts for its baseline cut at whichever of the sample's nine deciles fits the outcomes best — without ever looking at the treatment difference — reports a variance 0.671 of the unadjusted analysis's where its estimates actually have 0.736, which is worse than the 0.700 a median fixed in advance delivers. The sample says the chosen cut kept 72.4% of what the baseline could remove; the population says it kept less than a median. A true null is rejected 5.87% of the time at fifty patients an arm and 9.31% at ten, and a cut chosen for its p-value rejects 14.22%.
A relation supplied in advance
When theory supplies the long-run coefficient, the gap between two series can be tested directly and the search the Engle–Granger test pays for disappears: its 5% point moves from −3.38 to −2.89, and at two hundred observations the slowest return found four times in five goes from a half-life of 5.05 steps to 6.88. The price is in what a rejection means. With a gap that halves in three steps, a coefficient a fifth too large is confirmed 99.1% of the time and one half again too large 81.9%, though every such gap contains a random walk — and at a thousand observations the second is confirmed more often, not less.
A control borrowed from the last trial
A trial of fifty patients an arm can borrow its control group from an earlier trial of two hundred by treating the two control means as draws from one population with a spread between trials. With two trials there is one difference to learn that spread from, so the prior on it decides how much is borrowed — and its tail decides whether the borrowing stops when the two trials disagree. A half-normal prior of scale 0.05 buys 17.4 points of power and, at a drift of 1.5 standard deviations, still declares a treatment with no effect a success 31.8% of the time. A half-Cauchy of the same scale buys 13.6 points, peaks at 11.0% and falls back to 3.4%.
One test for eight nulls
A combination of eight forecasts can be tested against its parts one comparison at a time or all at once. All at once is a Wald test on the vector of eight encompassing means — which has seven degrees of freedom, not eight, because the means sum to zero on every origin when weighted by the combination. On sixty origins it rejects a true null 26.6% of the time, Hotelling's correction brings that to 15.1%, and handed its own true critical value it is less powerful than Bonferroni over the largest of the eight against both wrong combinations measured: 12% against 18%, and 47% against 96%.
The threshold the recentring reads
A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.
The power a calibration buys
Charging two searches separately makes a joint test conservative, and calibrating the joint statistic on its own null repairs that. Against a real break, on the pair of a break search and a window search, the repair is worth 47 points of power — 62.0% against 14.5%. On the pair of a break search and a column search it is worth nothing, and worse: a real column is found by the calibrated joint test on 11.8% of samples and by a plain Bonferroni split on 73.0%, because a joint statistic takes its threshold from the noisier of its two searches.
The hazard ratio the follow-up chose
A treatment that halves the hazard for one year and then does nothing has a Cox hazard ratio of 0.5000 if the trial stops at one year, 0.7617 at three and 0.8194 at eight. Nothing about the treatment differs between those numbers. When hazards are not proportional the hazard ratio is an average, and the length of follow-up and the dropout rate choose its weights.
Two ways to combine p-values
Fisher's and Stouffer's combinations are both exactly right when every null is true, for the single reason that each p-value is flat. Under a real effect they disagree about which evidence counts: with Stouffer held at 50% power across ten studies, Fisher is the more powerful while the signal sits in six or fewer of them and the less powerful from seven.
A look the trend asked for
Under an O'Brien–Fleming-type spending function, every schedule of looks fixed in advance spends exactly 5.0000%. A committee that adds a look at three quarters of the trial whenever the interim z is 1.5 or more spends 5.2323% — 5.315% counted over a hundred thousand trials — and the most a committee choosing among six schedules could spend is 5.4390%.
An order that spends the error rate
Test twenty hypotheses in a declared order, each at the full 5% and each only if every one before it was rejected, and the first is found 85.3% of the time where Holm finds it 52.5%. The tenth is found 20.4% of the time, the product of the powers before it. Move one true null to the head of the list and every real effect behind it is found no more than 4.3% of the time.
The repair that keeps the question
A regression between two independent trending series is significant 82.9% of the time on random walks and 100.0% on trend-stationary ones. Subtracting a fitted line leaves 74.2% and 33.5%; differencing leaves 5.0% and 5.2% and throws away the trend the study was about.
A statistic that is exact twice
Dividing the difference in means by its own separate-variance standard error before permuting takes the rejection rate under a true weak null from 20.47% to 6.07%, keeps the exactness under the sharp null at 4.07%, and costs 0.8 points of power against a real effect. At an even split it changes nothing at all, in every draw.
A visit before or a visit after
A trial that can afford one more measurement per patient can take it before treatment, to sharpen the baseline adjustment, or after it, to average the outcome. With every pair of visits correlated 0.5, the second visit after treatment removes a quarter of the variance and the second visit before it removes a twelfth, and it is never the other way round: the baseline's contribution is ρ²(1 − ρ)/(1 + ρ), which peaks at ρ⁵ = 0.090 when ρ is the golden ratio's reciprocal. Below a correlation of one half, a single extra follow-up beats any number of baselines. When the correlation fades with the time between visits, averaging an earlier baseline into the adjustment makes the trial less precise, not more.
A slow return across many pairs
One pair of two hundred observations is found tied four times in five only if its gap halves within 5.1 steps. Sixteen independent pairs sharing the same half-life, tested by the average of their Engle–Granger statistics, are found four times in five up to a half-life of 15.0 — and at a half-life of twelve, 99.8% of the time against one pair's 20.0%. What the widened test rejects is a statement about the panel: with half the pairs tied it rejects 55.4% of the time, with a quarter 21.2%, and a pair-by-pair reading of a panel in which all sixteen are tied finds 2.65 of them, against 0.78 when none are.
A history that agrees with itself
Borrowing a control from sixteen earlier trials that agree with each other should be safer than borrowing from one, and it is the opposite. Sixteen agreeing trials estimate the spread between trials as small, and a small spread estimated confidently is a licence to pool: under a half-Cauchy prior that let go of a single disagreeing trial, a current control 0.6 standard deviations from a sixteen-trial history turns a treatment with no effect into a success 80.7% of the time, against 10.5% at worst with one earlier trial. Moving a fifth of the prior onto a vague component brings the worst case to 16.9% and keeps power at 91.0%, against 70.5% without borrowing.
Sixteen subgroups and one effect
A trial of four hundred with 80% power tabulates its result by eight binary characteristics — sixteen subgroups — and the treatment's effect is the same for every patient. Among trials significant overall, 93.5% report at least one subgroup where it is not, and the median one reports seven. Some subgroup-against-the-rest interaction reaches 5% in 34.6% of trials — eight independent chances — and when the characteristics overlap as two age thresholds do, 18.6%, about four. The effective count is set by the correlation between the characteristics' split indicators, (2/π) arcsin of their latent correlation, and a Bonferroni reading of the eight holds the rate at 5% whatever the overlap.
Residuals that look too normal
The quantile plot most often read is of a regression's residuals, and residuals look more normal than the errors behind them. Each residual is a mix of all the errors, and the mix moves towards a bell as the fit takes more columns. On forty rows with skewed errors, the straightness test flags 82.9% of samples of the errors themselves, 79.1% of the residuals of a line, 71.4% with five columns, 51.2% with ten and 23.0% with twenty — far below the 49.5% that twenty errors would give, so the fit costs more than the observations it uses. Light tails vanish faster: 73.0% on the errors, 4.7% on twenty-column residuals. The band's level holds at every design, and one point with leverage 0.73 costs almost nothing; what costs is the number of columns.
Two windows and a ratio
Two counts from two windows give an interval for the ratio of their rates through the conditional binomial: given the total, the first window's count is binomial with a proportion that depends only on the ratio. The arrival times in both windows supply that binomial with a continuous coin at every total from one up, and the interval then covers exactly 95% plus a twentieth of the chance of no events at all. A total of zero can only report the whole of (0, ∞). And the exact conditional test's familiar conservatism is not the zero total's: the zero is the larger cause only while the expected total is below ln 2, and at an expected total of ten the exact interval still covers 97.8% for reasons that have nothing to do with it.
The smallest of three combinations
Reporting whichever of Fisher's, Stouffer's and Tippett's combinations is smallest is a test of its own, and on ten studies of nothing it rejects 9.66% of the time — not 5%, and nowhere near the 15% the three sizes add to, because the statistics are correlated at up to 0.903. Read at 2.448% each it is exact, and then it trails the best single combination by at most 7.45 points and leads the worst by at least 10.30.
A horizon chosen after looking
A difference in restricted mean survival read at whichever of eleven horizons looks most convincing rejects 11.24% of trials in which the treatment does nothing, against 4.70% at a horizon fixed in advance. The correlation of the differences across horizons is closed, and the Gaussian process it defines prices the choice at a critical value of 2.317 — which brings the counted size back to 4.99% and keeps 96.92% of the power that a horizon nobody could have known to fix would have had.
How slow a return a sample can see
At two hundred observations the test finds a gap that halves in five steps four times in five, one that halves in eight 37.3% of the time, and one that halves in fifty 4.95% of the time — which is the rate at which it finds pairs with no mechanism at all. The boundary moves with the sample, not with its square root.
A detector built for the ordering
The best of three checks for a drifting scale fires at half the growth factor the standard one needs — 2.12 against 4.31 — and still leaves 6.50 points of coverage gone before it does, against 0.51 for serial correlation. The reversal was not a property of the test.
A shock every pair shares
Sixteen pairs of series whose shocks are half common are worth 5.37 independent pairs to the panel's average statistic, and sixteen that share nine tenths are worth 1.44. Read against the 5% point computed for independent pairs, the pooled test then calls a panel with no relation tied 15.5% and 31.6% of the time; read against its own point, it finds a relation shared by every pair 22.3% of the time at nine tenths common, against one pair's 20.0%. Removing each date's cross-sectional average gives back the level and all sixteen pairs when every pair feels the shock equally, and leaves 13.3% false alarms when the loadings run from a half to one and a half.
A history that has been moving
Earlier control arms borrowed as exchangeable draws are pooled towards the history's middle, and when the control mean has been rising the current trial sits at its end. A drift of 0.02 standard deviations a trial — a step between neighbouring trials of under a third of either one's standard error — turns a treatment with no effect into a success 9.9% of the time from sixteen earlier trials, and a drift of 0.05 overstates the effect by 0.130 standard deviations from eight. A line in time removes the bias exactly, at every slope, and pays for it in borrowing: from eight earlier trials it borrows 175 control patients where the exchangeable model borrows 439.
One characteristic that really matters
A trial of four hundred tabulates eight characteristics and one of them genuinely changes the treatment's effect — the whole effect on one side, none on the other. Tested alone at 5% the real interaction is found 81.0% of the time; at the Bonferroni threshold that protects the table, 53.6%. It is the table's largest interaction in 82.4% of trials when the characteristics are independent and in 54.4% when they overlap like nested age bands, and in that case another characteristic crosses the Bonferroni line in half of all trials and is the top finding in a quarter. A family-wise threshold counts false rows; it cannot say which of two overlapping rows is the cause.
Residuals made independent
Recursive residuals — each row predicted from the fit to the rows before it — are exactly independent normals under the null for any design, so a quantile plot of them has an exact band. They see a skewed error worse than the ordinary residuals do, not better: 25.9% against 51.2% at ten columns on forty rows, 9.1% against 23.0% at twenty. BLUS residuals, the independent set closest to the errors, reach 38.3% and 16.2%. Every such set is a rotation of the ordinary residuals, so none holds information they lack, and independence is bought by mixing some residuals almost entirely out of other rows' errors — the least pure recursive residual at twenty columns is 0.4% its own.
The correlation a pilot can see
Two models of repeated visits agree that adjacent visits correlate 0.7 and disagree about how many follow-ups a trial should buy: three under equal correlation, one when the correlation fades. Planning on the wrong one costs 28.0% or 9.9%. Two follow-ups cost at most 3.2% more than the best under either, and at most 11.1% under any curve with that adjacent correlation. A pilot of twenty patients at four visits chooses worse on average than that rule when the correlation fades; it takes about forty to draw level with it, and at an effect of half a standard deviation those forty patients' extra visits cost fifty times what the better schedule saves.
A look added by a rule
A monitoring committee's rule — add a look at three quarters of the trial when the interim |z| is at least 1.5 — spends 5.2323% under O'Brien–Fleming-type boundaries. Written into the protocol it can be given boundaries that spend exactly 5%, three ways: raise the added look's boundaries by 2.05%, raise every later boundary by 0.99%, or preserve at each interim value the chance of crossing the plan would have given. All three cost the same, 0.38 points of power, from 88.94% to 88.56%, because the power the uncorrected rule seemed to keep was the excess error. Made exact, the rule is a design between the two fixed schedules — and at every trigger slightly better than randomising between them.
The repair, with a covariate in it
Studentising a permutation test's statistic made the difference in means exact under the sharp null and nearly right under the weak one. Carried to the covariate-adjusted estimate a trial actually reports, the repair works only if the adjustment includes the treatment-by-covariate interaction. With a quarter of 150 units treated and effects that vary with the outcome, the adjusted estimate's permutation test rejects 14.8% of true weak nulls; over its sandwich standard error, 3.0%; the interacted estimate over its sandwich, 3.9%. When the effect varies with the covariate itself, even an even split does not protect: the adjusted test rejects none of a thousand true weak nulls, and at an average effect of 0.2 its power is 5.2% against the interacted, studentised test's 40.7%.
A contrast chosen after the data
A two-arm trial with a binary outcome that reports whichever of the risk difference, the log risk ratio and the log odds ratio comes out most significant has tested three statistics that ask one question. At an equal split the choice is no choice at all: the difference has the largest z in every one of twenty thousand trials, so the selected test is the difference's, at its 5%. At a 30–70 split the choice is real and costs a point and a half of false positives, 6.5% at the rarest outcome. Pre-registering the difference and the odds ratio as a pair with Bonferroni's correction — the cautious alternative — costs eight to twelve points of power at every split.
The covariate chosen afterwards
A trial that chooses which baseline covariate to adjust for after randomisation has two ways to run its permutation test: hold the chosen covariate fixed across the shuffles, which is what an analysis that reports adjusting for it actually does, or re-make the choice inside every shuffle, which is exact by construction. Holding it fixed, a choice made on significance rejects 10.4% of true sharp nulls, a choice made on imbalance only 3.0% — and pays seven and a half points of power for it — while a choice that never reads the arm labels is the same test either way. The interacted, studentised statistic keeps its weak-null behaviour only when the covariate chosen is the one the effect varies with.
The variance that contains the effect
A monitoring committee that adds an interim look because the variance came in higher than planned is reading a nuisance quantity, and a look added on a nuisance quantity should leave the spending function's 5% alone. It does, exactly, when the variance is the within-arm one, and when the trigger is the calendar. It does not when the variance is the one computed with the arm labels hidden: that variance is the within-arm sum of squares plus the interim statistic squared, so it is high more often when the trend is strong. The excess is 0.069 points at a twenty-patient interim and falls as one over the square root of the interim's size — a tenth of what a trigger on the trend itself spends, and never zero.
A schedule the trial chooses for itself
A separate pilot cannot pay for itself in choosing a trial's follow-up schedule, because its patients are recruited, measured and discarded. A trial can instead measure its own first patients on the schedule that needs no pilot — two follow-ups — fit the correlation curve to them, and give everyone after them the schedule that curve makes cheapest. Those first patients are counted in the analysis and cost no extra visit. Forty of them cut the trial's worst-case excess cost across every curve with adjacent correlation 0.7 from 11.1% to 5.9% in a trial of two hundred an arm, and eighty cut it to 3.6%; twenty buy almost nothing. And because the curve is read from within-arm deviations, the choice cannot see the effect: the pilot's effect estimate is the same whichever schedule it chose.
A history that stepped
When sixteen earlier control arms stepped once — the first eight a fifth of a standard deviation below the last — borrowing them as exchangeable flatters a treatment with no effect by 0.063 and stops borrowing, and fitting a line through them penalises it by 0.052. A model with a change point is unbiased at every step size, borrows more the larger the step, 512 patients at a fifth, and puts two thirds of its belief on the right place for the change. Against a steady drift it is the one that fails, so each of the three is safe only against the shape of history it assumes.
A quantile over the recent scores
A noise scale that grows across the sample takes a split conformal interval from 95% to 85.63%, and the best detector cannot see most of that loss. Take the quantile over the last thirty-nine calibration scores instead of all 199 and the worst coverage across the whole sweep is 93.42%, for 3.7% more width when nothing has drifted. The same window covers 91.30% where all 199 cover 93.53% under strongly correlated errors — and switching to it only when the detector fires is worse than either, because the detector fires on correlation too.
One test for a curve that might cross
The log-rank statistic and the differences in restricted mean survival at eleven horizons are one Gaussian vector under no effect, and its correlations are closed — from −0.461 at half a year to −0.949 at three, where the log-rank statistic is nearly the restricted mean itself. So one critical value, 2.374, prices the largest of all twelve. At 100 a side it keeps 93.6% of the log-rank test's power when the hazard is halved throughout and 98.5% of the best horizon's when the curves cross, where the log-rank test has none. Running both tests and keeping either rejection rejects 8.67% of trials with no effect.
The shape the estimates keep
Partial pooling was suspected of destroying the evidence of its own misfit: compressing a two-cluster population of group effects towards one centre before any test could see the clusters. With groups of equal size it destroys nothing, because every estimate moves the same fraction of the way to the same place and the estimates keep the means' shape exactly — except when the fit pools completely, as it does on 29.9% of eight-group datasets at a spread of one. The evidence was never there: a shape check has 6.5% power at eight groups. With groups of unequal size the pooled estimates see the clusters better than the raw means do.
A factor estimated from the panel
When sixteen pairs feel a common shock with loadings from a half to one and a half, removing the cross-sectional average leaves the pooled test rejecting an unrelated panel 13.3% of the time. Estimating the factor and each pair's loading instead brings that to 8.9% — and no further, because each loading comes from the same two hundred dates however many pairs there are — and it costs power against the average at the right critical value, 73.8% against 82.5%. Removing the shocks as observed restores 4.5% exactly and finds a shared relation 7.5% of the time: a series tied to its partner carries the partner's factor in its level, not its own shock.
Residuals of a model the data chose
Five columns picked by a forward search from twenty candidates that have nothing to do with the outcome leave residuals that show a skewed error 36.8% of the time, where five columns fixed in advance show it 65.4% — the power of a fit of 13.4 fixed columns. Against light-tailed errors the chosen fit's plot is blind, 8.5% against 34.5%. Simulating the search on normal errors gives a band that holds 5%, which the ordinary band already nearly did; the level was never what broke. A reader told the final model has five columns is reading a plot with the power of thirteen.
A restriction the data can check
A rank-two system of three series is identified by exactly two restrictions on its relations, and any two that describe the estimated plane fit it equally well — their forecasts agree with the unrestricted fit's to the last digit, so the data can neither confirm nor refuse them. Supply one relation in full instead, one restriction more than identification needs, and the surplus becomes a test: a relation a tenth wrong is rejected 37.8% of the time at two hundred observations and 98.5% at eight hundred, the error found half the time halving as the sample doubles. A right relation buys the forecast almost nothing; a fifth wrong costs 13% of it at two hundred observations and 25% at eight hundred.
Named alongside it
The objects these essays reach for when they reach for this one.
Monte CarloError rateMultiple comparisonsSample sizeCritical valueReference distributionp-valueBonferroniClosed formCorrelationExperimental designFamilywise error rate