Normal Distribution
nothing is called 95% until it has been counted
Concept

Type i error — where it appears

Rejecting a null hypothesis that is true: declaring an effect where there is none. A test's type I error rate is the share of true nulls it rejects, which is the level it promises to hold, and every adaptation, look or choice made from the data is judged by whether it keeps that share where it was set.

Named by 2 essays across 2 fields — each of them below, with the objects they name alongside it.

Three charges, and only one of them is a test. What each of three thresholds does to the same decision, under AR(1) at 0.8, against the size of a genuine break in the mean at row 60. A chi-square on the 5 coefficients a split adds — 11.07 — declares a break on 73.6% of samples that have none: it is not a test at all. The break search's own 95% point, 69.6, carried into a rule that also chooses its window, fires on 0.0% of null samples and on 0.0% of samples with the largest break measured — the natural way of combining two published corrections does not lose a little power, it switches the test off. The calibrated charge, 27.2, holds 5.6% at no break and reaches 29.2% at the largest.

The charge that is not a sum

Charging two searches what each costs on its own is conservative, and conservative here means the test never fires. At the largest break measured it declares nothing, on every draw, while a calibrated threshold reaches 29%.

twice · Break point How much a look added at three quarters spends above 5%, by what the decision to add it reads. A two-look O'Brien–Fleming-type trial, interim at half the information, adds a look at three quarters with a stated chance under the null. Its error rate above 5%, in points, at chances of 0.1, 0.3, 0.5, 0.7, 0.9: triggered by a calendar or by the within-arm variance, 0.00000, 0.00000, 0.00000, 0.00000, 0.00000; by the blinded variance with 20 patients at the interim, 0.034, 0.064, 0.069, 0.054, 0.022; with 100, 0.014, 0.027, 0.030, 0.025, 0.012; with 1,000, 0.004, 0.008, 0.009, 0.008, 0.004; by the interim trend itself, 0.234, 0.163, 0.089, 0.042, 0.013. Exact integrals over the interim statistic.

The variance that contains the effect

A monitoring committee that adds an interim look because the variance came in higher than planned is reading a nuisance quantity, and a look added on a nuisance quantity should leave the spending function's 5% alone. It does, exactly, when the variance is the within-arm one, and when the trigger is the calendar. It does not when the variance is the one computed with the arm labels hidden: that variance is the within-arm sum of squares plus the interim statistic squared, so it is high more often when the trend is strong. The excess is 0.069 points at a twenty-patient interim and falls as one over the square root of the interim's size — a tenth of what a trigger on the trend itself spends, and never zero.

sequential · Stopping

Named alongside it

The objects these essays reach for when they reach for this one.

Statistical powerAlpha spendingBandwidthBlinded analysisCalibrationChi squaredCritical valueDependenceGroup sequentialInterim analysisLikelihood ratioMonte Carlo

All concepts