Normal Distribution
nothing is called 95% until it has been counted
Concept

Replication — where it appears

Running the same condition more than once, which is the only source of an estimate of pure error that does not assume the model. Without it, lack of fit and noise cannot be told apart, and a model can look adequate because nothing was measured twice.

Named by 4 essays across 4 fields — each of them below, with the objects they name alongside it.

Where a replication's estimate lands against a 95% interval, replication the same size. The chance that a 95% interval contains a replication's estimate is 95.00% when the original landed on the truth, 82.99% one standard error away and 48.40% two away. Averaged over where originals land it is 83.42%, and 5.00% of originals capture a replication less than half the time.

Five times in six

A 95% interval is read as a 95% chance that a replication's estimate will land inside it. With the spread known and a replication of the same size, the chance is 83.42% — five times in six — because both estimates are uncertain. An original that landed two standard errors from the truth captures a replication 48.40% of the time; a replication a tenth the size lands inside 44.54% of the time; and among significant originals from studies with 17% power, 66.94%.

alongside · Repetition Two factors that interact — where each design looks. The true response at the four corners is -10, 6, 4, 0. One factor at a time visits three of them, sees that raising either factor alone helps, and recommends raising both — a corner it never ran, and one that is worse than either single change. It picks the best corner 0.0% of the time against the factorial design's 99.4%, on the same number of runs.

One factor at a time

Changing one thing per experiment estimates each effect from two conditions; changing everything at once estimates each from every run. The ratio is (k+1)/2 and it is exact — and when two factors interact, the one-at-a-time design recommends a setting it never tried.

design · Factorial 12,000 studies of a real effect of 0.3, n = 16. Power is 21%. The studies that reached significance report a mean effect of 0.621 — 2.07 times the truth. Every one of them is honest; the selection did the inflating.

The winner's curse

Filter honest studies down to the ones that reached significance and the effects they report are systematically too large. At low power the inflation is a factor of two, nobody has done anything wrong, and the selection did all of it.

testing · Curse Where the bootstrap works and where it does not. Uniform data on [0, 1]. For the mean the percentile bootstrap covers 93.5%. For the maximum it covers 0.0%, because a resample can never contain a value larger than the largest one observed, so the interval cannot reach above it.

Where the bootstrap lies

Resampling is the most generally useful trick in the subject and it has a failure mode that is easy to state: it cannot see past the data. For a statistic that lives at the edge of the sample, coverage collapses from 95% to almost nothing.

intervals · Bootstrap

Named alongside it

The objects these essays reach for when they reach for this one.

CoverageStatistical powerThe winner's curseBootstrapCapture percentageConfidence intervalExtreme statisticFactorial designHidden replicationInteractionMain effectPrediction interval

All concepts