Normal Distribution
nothing is called 95% until it has been counted
Concept

Conditional inference — where it appears

Inference that treats some feature of the data as fixed rather than random — the observed covariates, the observed margins, the assignment that was actually run. It changes what the reference set is, and therefore what a p-value or an interval is a statement about.

Named by 6 essays across 6 fields — each of them below, with the objects they name alongside it.

What each analysis does at a true null, by shape. Four analyses of the same trials — 500 of them at each shape, 120 units, assigned by the rule that reads the covariate. Every rejection is false. The unadjusted analysis is the one that moves: 1.60% against a linear outcome, where the design removed a great deal that the standard error still prices, and 5.20% against a quadratic, where it removed nothing and the standard error is right. Adjusting holds the level in all three columns, and so does the design's own reference distribution, which needs to be told the rule and nothing else.

The analysis and the shape

An unadjusted analysis after a rule that read the covariate is too cautious — by a third against a linear outcome, by nothing at all against a quadratic. And an adjustment for the wrong function recovers almost none of the precision the right one would.

shape · Randomisation A thin enough set is not one set. Every admissible set of 14 units this table can enumerate, by how much of the assignment space it admits and how many pieces it falls into under single swaps. A walk is uniform on the piece it starts in and never leaves it. The pieces are not fragments: at 522 admissible assignments the set splits into 3 halves of exactly 520 each, and every assignment's complement is in the other half — no sequence of admissible single swaps takes an assignment to its own mirror image. Two-swap proposals reconnect four of the six disconnected sets here — the two they do not are the thinnest, where a two-unit move rarely lands anywhere admissible either — which makes a bigger proposal a correctness repair rather than the speed dial it was measured as.

The walk that cannot cross

A thin enough admissible set is not one set. It splits into an assignment and its mirror image, no sequence of admissible single swaps joins them, and the walk that samples it is uniform on half the reference distribution for ever.

dict · Randomisation The price of insurance is noise, not coverage. What a robust standard error costs under a constant error variance — the case where the model-based one is exactly right — at five sample sizes over 20000 draws at the small end. It is not coverage: the leave-one-out interval read against a t on n − 2 covers 95.52% at 20 rows against the model-based 95.06%. It is a wider interval, by a factor of 1.0689 at 20 rows falling to 1.0049 at 250, and it is a variance estimate 2.57 times as variable at 20 rows and 1.85 times at 250 — against a denominator that is exactly V²·2/(n − 2), so only the numerator is counted. The uncorrected estimate is the cheaper of the two at small samples and the dearer at large: 1.31 against 1.76.

Right for the wrong reason

A robust standard error costs no coverage where the risk is absent — 95.52% against 95.06% at twenty rows. It costs a 6.89% wider interval and a variance estimate 2.572 times as variable, and the pre-test that would avoid paying recovers 15.9% of what the insurance is worth.

sandwich · Misspecification How often two orders of selection report a winner from twenty analyses, no effect anywhere. Twenty analyses correlated 0.6, each with a diagnostic correlated γ with its own statistic, over 20,000 families a point. Clean then significant — the best diagnostic, reported if it clears 1.96 — reports a winner in 2.55%, 4.25%, 6.91%, 13.15% of families at γ = 0, 0.3, 0.6, 0.9. Significant then clean — the best diagnostic among those that cleared 1.96 — reports one in 18.04%, 18.04%, 18.04%, 18.05%: in every family where any analysis cleared 1.96, whatever γ is.

The cleanest of the significant

An analyst with twenty correlated analyses can choose in two orders: take the one whose diagnostics look cleanest and report it if it clears 1.96, or take the analyses that cleared 1.96 and report the cleanest of them. The second is the one described as prudent, because it never reports an analysis whose diagnostics look poor — and with no effect anywhere it reports a significant analysis in about 18% of families whatever the diagnostic is, against 2.55% to 13.15% for the first. The diagnostic decides which significant analysis is shown and never whether one is. Each report's own interval can still be made exact, by conditioning on a selection set that is a union of pieces rather than a half-line; the family's claim count cannot.

paths · Forking The textbook (Wald) interval's coverage among studies whose count of 40 lands in the tails. Coverage of the nominal 95% textbook (Wald) interval at 40 trials, conditional on the count being at most k or at least 40 − k, computed exactly over the counts for every true proportion. With k = 2 its highest value anywhere is 88.96%; With k = 5 its highest value anywhere is 94.90%; With k = 8 its highest value anywhere is 95.57%. A set whose highest value stays below 95% is a set of counts on which the label is wrong whatever the proportion is.

A count that bets against its interval

A reader who sees only the count can bet against an interval's 95%, and win whatever the proportion is, exactly when some set of counts carries less than 95% at every proportion. Out of forty trials the textbook interval has one — a count of five or fewer, or thirty-five or more, covers at most 94.90% — and the score interval has none: a prior exists under which every one of its forty-one intervals is 96.46% credible, and that prior is the proof that no bet can be found.

intervals · Coverage The interval for a finding's effect given that it cleared its threshold, against where its estimate landed. Threshold c = 2.8. At z just past it the selective 95% interval runs from −0.58 to 0.95; at z = 3.8, from 0.18 to 5.73; at z = 6.3, from 4.31 to 8.26, against the ordinary z ± 1.96.

An interval that knows its finding was found

An interval for one Benjamini–Hochberg finding can be read from the distribution of its estimate given that it cleared the threshold it had to clear. It covers the finding's effect 94.8% of the time with real effects of two standard errors, against the ordinary interval's 87.0%, and it covers the most prominent finding 91.3% of the time where the ordinary one covers 72.4%. The price is width — a median 5.06 standard errors against 3.92 — and something larger: for three findings in four at that effect size the interval reaches back past zero, so a finding that cleared the line is, once the line is accounted for, not distinguishable from nothing.

multiplicity · Multiplicity

Named alongside it

The objects these essays reach for when they reach for this one.

Confidence intervalCoverageCovariate balanceExact enumerationInterval widthMultiple comparisonsRandomisation testReference distributionSelection effectAllocation ruleAssignment mechanismAuxiliary regression

All concepts