Normal Distribution
nothing is called 95% until it has been counted
Concept

Visual inference — where it appears

Judging a plot by hiding it among plots generated under the null and asking whether it can be picked out. It turns looking at a figure into a test with a stated error rate, and most apparent structure does not survive it.

Named by 3 essays across 3 fields — each of them below, with the objects they name alongside it.

Two 95% bands for a quantile plot of 40 points. The outer band is left by 5% of genuinely normal samples — which is what a reader is using a band for. The inner one holds each point separately at 95%, which is what software draws, and 45.0% of genuinely normal samples step outside it. The outer is the inner widened by a factor of 1.502.

The band the eye was standing in for

The confidence band software draws on a quantile plot holds each point at 95%, and a genuinely normal sample of forty has forty chances to leave it — so 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it.

lineup · Qq Twenty samples of 40, every one of them genuinely normal. Each panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 0.87 standard deviations off the line. Anything a reader would reject here would be a false alarm.

What normal actually looks like

A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there.

normal · Qq Twenty residual plots from data where the model is exactly right, n = 24. Every panel is a correctly specified linear model with normal errors. The apparent curvature, funnelling and outliers are all produced by noise, and the largest single residual across the twenty is 2.13 standard deviations of the error. This is the reference nobody has when judging a real residual plot.

Twenty residual plots

Judging whether a residual plot looks wrong requires knowing what a correct one looks like, and almost nobody has seen twenty of those. Here they are, from a model that is exactly right, at the sample size that matters.

regression · Qq

Named alongside it

The objects these essays reach for when they reach for this one.

NormalityQ–Q plotResidual plotSample sizeSampling variationConfidence bandError rateThe line-up testModel diagnosticsMultiple comparisonsOrder statisticp-value

All concepts