Normal Distribution
nothing is called 95% until it has been counted
Concept

Multiple testing — where it appears

Running several hypothesis tests and reporting a finding if any of them rejects. The chance that at least one rejects by luck is larger than each test's own level, so the critical values have to be set for the family as a whole, using how its tests are correlated.

Named by 2 essays across 2 fields — each of them below, with the objects they name alongside it.

The rate falls geometrically; the count does not fall at all. The share of equal splits of two hundred units that a tolerance of 1 coin-spreads admits, against the number of functions the tolerance is stated for, with the closed form (2Φ(1) − 1)^k drawn beside it. The rate falls by about two thirds with every constraint. The admissible count is that rate times C(200, 100), and it goes from 2^195 to 2^192 — it does not fall in any sense a trial cares about. What the falling rate costs is sampling: 9,878 draws to collect a thousand admissible ones at six constraints, against 1,465 at one.

What a reference distribution costs to sample

A randomisation test on a trial too large to enumerate has to sample its reference distribution, at 1/p attempts per draw and a p-value resolved to 1/(B + 1). Six constraints cost 9,878 attempts per thousand draws, and a thousand draws resolve p to 9.99·10⁻⁴ and not one digit finer.

product · Reference Power of four tests of two survival curves, in three shapes of treatment effect, at 50, 100 and 200 a side. 6,000 trials at each setting, dropout 0.1 a year, follow-up to three years. hazard halved throughout, 50 a side: log-rank 64.9%, best of eleven horizons 51.7%, restricted mean at 3 61.1%, one test for all twelve 56.6%; hazard halved throughout, 100 a side: log-rank 91.8%, best of eleven horizons 81.0%, restricted mean at 3 88.4%, one test for all twelve 86.0%; hazard halved throughout, 200 a side: log-rank 99.7%, best of eleven horizons 98.4%, restricted mean at 3 99.4%, one test for all twelve 99.3%; halved for a year, then nothing, 50 a side: log-rank 18.5%, best of eleven horizons 32.1%, restricted mean at 3 26.6%, one test for all twelve 30.8%; halved for a year, then nothing, 100 a side: log-rank 29.3%, best of eleven horizons 51.7%, restricted mean at 3 42.5%, one test for all twelve 50.2%; halved for a year, then nothing, 200 a side: log-rank 51.4%, best of eleven horizons 81.4%, restricted mean at 3 70.3%, one test for all twelve 80.2%; curves that cross, 50 a side: log-rank 5.1%, best of eleven horizons 45.7%, restricted mean at 3 10.4%, one test for all twelve 44.0%; curves that cross, 100 a side: log-rank 5.2%, best of eleven horizons 76.7%, restricted mean at 3 16.4%, one test for all twelve 75.5%; curves that cross, 200 a side: log-rank 5.2%, best of eleven horizons 96.7%, restricted mean at 3 26.5%, one test for all twelve 96.6%.

One test for a curve that might cross

The log-rank statistic and the differences in restricted mean survival at eleven horizons are one Gaussian vector under no effect, and its correlations are closed — from −0.461 at half a year to −0.949 at three, where the log-rank statistic is nearly the restricted mean itself. So one critical value, 2.374, prices the largest of all twelve. At 100 a side it keeps 93.6% of the log-rank test's power when the hazard is halved throughout and 98.5% of the best horizon's when the curves cross, where the log-rank test has none. Running both tests and keeping either rejection rejects 8.67% of trials with no effect.

survival · Censoring

Named alongside it

The objects these essays reach for when they reach for this one.

Critical valueMonte CarloAcceptance rateAllocationClosed formConfidence intervalCovariate balanceError rateGaussian processGranularityKaplan–MeierLog-rank test

All concepts