Normal Distribution
nothing is called 95% until it has been counted
Concept

Truncated normal — where it appears

A normal distribution cut off at a threshold and rescaled so its remaining part is a whole distribution. Its mean past a cut c is the inverse Mills ratio φ(c)/(1 − Φ(c)), which is how the size of a selected group's lead, and of its fall on a second reading, is computed.

Named by 5 essays across 3 fields — each of them below, with the objects they name alongside it.

The line is the sample, and the sample is the finding. 900 draws of two independent standard normal causes, with the 453 of them past a threshold of 0.00 marked and the 447 that fall short left pale. In the population the two are independent by construction. Inside the selected sample the correlation is -0.4669 in closed form and -0.5050 counted on these 453 rows, and the least-squares line through them has a slope of -0.545. The mechanism is visible in the picture rather than argued: the threshold removes one corner of the cloud, and a cloud with a corner missing is a cloud whose two coordinates carry information about each other.

The sample is a condition

Two independent standard normals, selected on their sum exceeding its median, read a correlation of exactly −1/(π − 1) = −0.4669 inside the sample. Nothing is measured badly and nothing is missing — and both halves of that split read it, in the same direction, while the population containing both reads zero.

collider · Conditioning A cohort screened once, the top tenth enrolled, and followed up with nothing given — correlation 0.6. 2000 people read once at screening and once at follow-up, with a test–retest correlation of 0.6 and no treatment. The 215 above a cut at the top ten per cent of one reading (1.282 standard deviations) are enrolled. Their mean screening reading is 1.744 and their mean follow-up reading 0.982, a fall of 0.762 ± 0.055 with nothing done to anyone. The closed form for the fall is (1 − ρ) times the truncated-normal mean, (1 − 0.6) × 1.755 = 0.702.

The measurement that got them enrolled

Enrol the top tenth of one screening reading and give them nothing, and they fall by 0.702 standard deviations at follow-up. Measured from a fresh reading taken after enrolment they fall by nothing. Averaging ten screening readings still leaves 0.101, and it takes twenty-one to get under 0.05.

paradox · Rtm Thirty winners of twenty analyses, effect 2 standard errors in one of them, with two intervals each. Each row is the largest of twenty statistics correlated at 0.6, reported because it cleared the family-wise threshold of 2.846. The thin grey bar is the ordinary interval, z ± 1.96; the heavy bar is the interval conditional on the statistic having won; the tick is the true mean of the analysis that won. Of these thirty the ordinary interval misses 1 and the conditional one 2; over 4,468 reported winners they cover 76.0% and 94.9%.

An interval for the winner

The largest of twenty correlated analyses, reported because it cleared the family-wise threshold, carries an ordinary 95% interval that covers its true mean 75.96% of the time at an effect of two standard errors and never at all when there is no effect. An interval built on the fact that it won covers 94.90% — and it is 2.48 times as wide, excludes zero for only 14.10% of the winners, and has no lower limit at all for 14.32% of them. It restores the promise by declining to say most of what the ordinary interval said.

paths · Forking How often a winner's interval covers when it is conditioned on the analyses admitted to rather than the twenty run, effect 2. The largest of twenty analyses reported when it clears the family-wise threshold for the k admitted. At k = 1, 2, 5, 10, 20: correlation 0.6, conditioned on the admitted family 95.3%, 94.9%, 94.9%, 95.1%, 95.2%; the ordinary interval 67.7%, 71.1%, 75.2%, 76.5%, 76.1%; correlation 0, conditioned on the admitted family 94.4%, 95.6%, 95.3%, 94.8%, 95.6%; the ordinary interval 47.2%, 52.0%, 58.9%, 62.4%, 64.7%.

An interval for the analyses admitted to

An analyst runs twenty correlated analyses, admits to one, and reports the winner with an interval conditioned on having cleared 1.96 — the selection as written down rather than as it happened. The interval covers 95.3% of the time, against 95.6% for one conditioned on all twenty, because a winner's truncation point is set by the threshold far more than by its unseen rivals: hiding nineteen analyses moves it by 0.061 on average. What the hiding changes is how often there is a winner at all. With no effect anywhere, a family of twenty produces a reported winner 35.5% of the time against 4.8% when all twenty are admitted, and one whose interval excludes zero 1.48% of the time against 0.13% — the interval keeps its promise and the report does not.

paths · Forking How often a diagnostically chosen winner's interval excludes zero, against the diagnostic's correlation with the statistic, no effect anywhere. Twenty analyses correlated 0.6; the one with the best diagnostic is reported if it clears 1.96. A winner is reported in 2.52%, 4.16%, 6.81%, 13.07% of families at γ = 0, 0.3, 0.6, 0.9. Its interval excludes zero: conditioned on clearing 1.96, 1.69%, 2.28%, 3.30%, 4.69%; conditioned on the diagnostic as well, 1.69%, 1.86%, 1.83%, 2.31%; rebuilt from the report's three numbers, 1.69%, 1.80%, 2.13%, 2.52%.

A winner chosen by its diagnostic

An analyst runs twenty correlated analyses and reports the one whose diagnostics look cleanest, if it clears 1.96 — and a diagnostic is rarely independent of the statistic it checks. With no effect anywhere, a diagnostic correlated 0.9 with its statistic lets a winner through in 13.07% of families against 2.52% for an unrelated one. The interval conditioned on clearing 1.96 still covers 93.27%, but its misses have moved: it excludes zero in 4.69% of reported winners, where a 95% interval should do so in 2.5%. Conditioning on the diagnostic as well repairs it, and so does something a reader can do alone — weight the reported statistic by the chance an analysis like it wins the diagnostic contest, from three numbers a report can state.

paths · Forking

Named alongside it

The objects these essays reach for when they reach for this one.

Confidence intervalCoverageThe garden of forking pathsSelective inferenceClosed formFamilywise error rateMonte CarloSelection biasSensitivity analysisThresholdThe winner's curseBerksons paradox

All concepts