Normal Distribution
nothing is called 95% until it has been counted
Series

Qq — the series

5 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. Twenty samples of 40, every one of them genuinely normal. Each panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 0.87 standard deviations off the line. Anything a reader would reject here would be a false alarm.

    What normal actually looks like

    A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there.

    part 2 · normal
  2. Twenty residual plots from data where the model is exactly right, n = 24. Every panel is a correctly specified linear model with normal errors. The apparent curvature, funnelling and outliers are all produced by noise, and the largest single residual across the twenty is 2.13 standard deviations of the error. This is the reference nobody has when judging a real residual plot.

    Twenty residual plots

    Judging whether a residual plot looks wrong requires knowing what a correct one looks like, and almost nobody has seen twenty of those. Here they are, from a model that is exactly right, at the sample size that matters.

    part 3 · regression
  3. Two 95% bands for a quantile plot of 40 points. The outer band is left by 5% of genuinely normal samples — which is what a reader is using a band for. The inner one holds each point separately at 95%, which is what software draws, and 45.0% of genuinely normal samples step outside it. The outer is the inner widened by a factor of 1.502.

    The band the eye was standing in for

    The confidence band software draws on a quantile plot holds each point at 95%, and a genuinely normal sample of forty has forty chances to leave it — so 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it.

    part 4 · lineup
  4. What the plot says, and what the interval does, at n = 40. For each source: how often a quantile plot of the data leaves its pointwise band, and how often the 95% t interval for the mean misses. The two-lump source leaves the band on 100% of samples and its interval covers 94.80%; the t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of the five.

    The plot is about the wrong quantity

    A t interval needs the sampling distribution of the mean to be normal, not the data. A two-lump source leaves its quantile band on 100% of samples of forty and its interval covers 94.80%; a t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of five sources.

    part 5 · lineup
  5. What a design does to the residuals of a correct model. Every residual has standard deviation sigma times the square root of one minus its leverage. On this design the leverages run from 0.045 to 0.663, so the residual spreads differ by a factor of 1.68 — and the model is exactly right. The high-leverage point's residual averages 0.46 of the fitted spread where a typical point's averages 0.79.

    Residuals are not the errors

    A residual's standard deviation is σ√(1 − hᵢᵢ), so a design whose leverages run from 0.045 to 0.663 produces residuals whose spreads differ by a factor of 1.68 with the model exactly right. On the samples where the high-leverage point really did have the largest error, a raw residual plot shows it as the largest on 0.0% of them.

    part 6 · lineup

All series