Normal Distribution
nothing is called 95% until it has been counted
Concept

Visual inference — where it appears

Judging a plot by hiding it among plots generated under the null and asking whether it can be picked out. It turns looking at a figure into a test with a stated error rate, and most apparent structure does not survive it.

Named by 7 essays across 3 fields — each of them below, with the objects they name alongside it.

Two 95% bands for a quantile plot of 40 points. The outer band is left by 5% of genuinely normal samples — which is what a reader is using a band for. The inner one holds each point separately at 95%, which is what software draws, and 45.0% of genuinely normal samples step outside it. The outer is the inner widened by a factor of 1.502.

The band the eye was standing in for

The confidence band software draws on a quantile plot holds each point at 95%, and a genuinely normal sample of forty has forty chances to leave it — so 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it.

lineup · Qq Twenty samples of 40, every one of them genuinely normal. Each panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 0.87 standard deviations off the line. Anything a reader would reject here would be a false alarm.

What normal actually looks like

A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there.

normal · Qq Twenty residual plots from data where the model is exactly right, n = 24. Every panel is a correctly specified linear model with normal errors. The apparent curvature, funnelling and outliers are all produced by noise, and the largest single residual across the twenty is 2.13 standard deviations of the error. This is the reference nobody has when judging a real residual plot.

Twenty residual plots

Judging whether a residual plot looks wrong requires knowing what a correct one looks like, and almost nobody has seen twenty of those. Here they are, from a model that is exactly right, at the sample size that matters.

regression · Qq How often each one-number summary of a quantile plot detects five departures from normality, forty observations. Power at 5% against five departures. The straightness statistic detects the skewed source 83.3% of the time and light tails 30.8%; kurtosis detects light tails 72.5%; skewness detects light tails 0.2%. Rejecting when any of the five exceeds its own 5% point rejects 13.9% of normal samples; correcting that search to 5% gives it 37.4% power against light tails.

A number for the shape

Five one-number summaries of a quantile plot's departure from its line, each a correct 5% test on forty normal observations, and no two agree about what matters. Straightness catches a skewed source 83% of the time and light tails 31%; kurtosis catches light tails 72% and skewness 0.2%. Reading all five and reporting whichever looks bad rejects 13.9% of genuinely normal samples. Correcting that search to 5% costs almost half the power against the departure it would have caught best — and a line-up of twenty panels gets the same correction for nothing.

lineup · Qq How often a quantile plot's shape test sees a skewed error distribution, on the errors and on residuals of fits of more columns. On the forty errors themselves the test flags 82.9%. On the residuals of fits with 2, 5, 10, 20 columns: 79.1%, 71.4%, 51.2%, 23.0%. On 40 − p errors, which is what losing the fitted columns' worth of observations alone would give: 80.6%, 76.1%, 69.1%, 49.5%.

Residuals that look too normal

The quantile plot most often read is of a regression's residuals, and residuals look more normal than the errors behind them. Each residual is a mix of all the errors, and the mix moves towards a bell as the fit takes more columns. On forty rows with skewed errors, the straightness test flags 82.9% of samples of the errors themselves, 79.1% of the residuals of a line, 71.4% with five columns, 51.2% with ten and 23.0% with twenty — far below the 49.5% that twenty errors would give, so the fit costs more than the observations it uses. Light tails vanish faster: 73.0% on the errors, 4.7% on twenty-column residuals. The band's level holds at every design, and one point with leverage 0.73 costs almost nothing; what costs is the number of columns.

lineup · Qq Whether residuals built to be independent see a skewed error better than ordinary ones. The straightness test, each version read at its own 5% point. On all forty errors: 82.9%. At 2, 5, 10, 20 columns: ordinary residuals 79.1%, 71.4%, 51.2%, 23.0%; BLUS residuals 76.4%, 64.9%, 38.3%, 16.2%; recursive residuals 64.4%, 49.5%, 25.9%, 9.1%; 40 − p errors 80.6%, 76.1%, 69.1%, 49.5%.

Residuals made independent

Recursive residuals — each row predicted from the fit to the rows before it — are exactly independent normals under the null for any design, so a quantile plot of them has an exact band. They see a skewed error worse than the ordinary residuals do, not better: 25.9% against 51.2% at ten columns on forty rows, 9.1% against 23.0% at twenty. BLUS residuals, the independent set closest to the errors, reach 38.3% and 16.2%. Every such set is a rotation of the ordinary residuals, so none holds information they lack, and independence is bought by mixing some residuals almost entirely out of other rows' errors — the least pure recursive residual at twenty columns is 0.4% its own.

lineup · Qq How often a quantile plot sees a skewed error when the fitted columns were chosen from the data. Forty rows, 3,000 samples a point, read against the band for forty independent normals. With the intercept alone the test flags 82.5%. With 1, 2, 5, 10 columns fixed in advance: 79.9%, 76.1%, 65.4%, 47.0%; with the same number picked by a forward search from twenty candidates unrelated to the outcome: 66.6%, 55.5%, 36.8%, 25.8%.

Residuals of a model the data chose

Five columns picked by a forward search from twenty candidates that have nothing to do with the outcome leave residuals that show a skewed error 36.8% of the time, where five columns fixed in advance show it 65.4% — the power of a fit of 13.4 fixed columns. Against light-tailed errors the chosen fit's plot is blind, 8.5% against 34.5%. Simulating the search on normal errors gives a band that holds 5%, which the ordinary band already nearly did; the level was never what broke. A reader told the final model has five columns is reading a plot with the power of thirteen.

lineup · Qq

Named alongside it

The objects these essays reach for when they reach for this one.

NormalityQ–Q plotModel diagnosticsStatistical powerResidualHat matrixLeverageThe line-up testMultiple comparisonsResidual plotSample sizeSampling variation

All concepts