Normal Distribution
nothing is called 95% until it has been counted
Concept

Normality — where it appears

That a quantity follows the normal law, which several exact procedures here need and several do not. What usually matters is not whether the observations are normal but whether the statistic is, and that is a question about sums rather than about data.

Named by 14 essays across 7 fields — each of them below, with the objects they name alongside it.

Which side a 95% t interval misses on, exponential source. Both tails should be 2.5%. At 8 observations the interval falls short of the mean on 9.75% of samples and overshoots on 0.31%. At 500 they are 3.31% and 2.05%, and the total is 5.36% — which a coverage table reports as very nearly right.

Where the two tails disagree

A 95% t interval on an exponential source at 120 observations covers 94.81%, which reads as very nearly right. It misses below the mean on 4.08% of samples and above on 1.11% — one tail 63% too heavy and the other 56% too light, and the total is the statistic that hides it.

tails · Student Two 95% bands for a quantile plot of 40 points. The outer band is left by 5% of genuinely normal samples — which is what a reader is using a band for. The inner one holds each point separately at 95%, which is what software draws, and 45.0% of genuinely normal samples step outside it. The outer is the inner widened by a factor of 1.502.

The band the eye was standing in for

The confidence band software draws on a quantile plot holds each point at 95%, and a genuinely normal sample of forty has forty chances to leave it — so 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it.

lineup · Qq What the plot says, and what the interval does, at n = 40. For each source: how often a quantile plot of the data leaves its pointwise band, and how often the 95% t interval for the mean misses. The two-lump source leaves the band on 100% of samples and its interval covers 94.80%; the t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of the five.

The plot is about the wrong quantity

A t interval needs the sampling distribution of the mean to be normal, not the data. A two-lump source leaves its quantile band on 100% of samples of forty and its interval covers 94.80%; a t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of five sources.

lineup · Qq Twenty samples of 40, every one of them genuinely normal. Each panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 0.87 standard deviations off the line. Anything a reader would reject here would be a false alarm.

What normal actually looks like

A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there.

normal · Qq What a design does to the residuals of a correct model. Every residual has standard deviation sigma times the square root of one minus its leverage. On this design the leverages run from 0.045 to 0.663, so the residual spreads differ by a factor of 1.68 — and the model is exactly right. The high-leverage point's residual averages 0.46 of the fitted spread where a typical point's averages 0.79.

Residuals are not the errors

A residual's standard deviation is σ√(1 − hᵢᵢ), so a design whose leverages run from 0.045 to 0.663 produces residuals whose spreads differ by a factor of 1.68 with the model exactly right. On the samples where the high-leverage point really did have the largest error, a raw residual plot shows it as the largest on 0.0% of them.

lineup · Qq How many observations the smallest and largest of them need. The interval between the extremes of n draws holds at least 95% of the population with probability 1 - n p^(n-1) + (n-1) p^n, whatever the population is. Reaching 95% confidence takes 93 observations.

Ninety-three observations, and nothing assumed

The interval between the smallest and largest of a sample holds a share of the population whose distribution does not depend on the population — Beta(n − 1, 2), for anything continuous. Buying the 95/95 that normality buys at ten observations costs 93 of them, and that number is the exchange rate between an assumption and data.

estimated · Bands The normal density at sigma = 1.00. The bands hold 68.27%, 95.45%, 99.73% of the mass. Those figures are integrals of the curve drawn, not the memorised 68-95-99.7.

The shape, and where its mass is

68, 95, 99.7 is recited more often than any other set of numbers in the subject. They are integrals of a specific curve, they are worth computing rather than remembering, and the third one is the one people misuse.

normal · Bands How often each one-number summary of a quantile plot detects five departures from normality, forty observations. Power at 5% against five departures. The straightness statistic detects the skewed source 83.3% of the time and light tails 30.8%; kurtosis detects light tails 72.5%; skewness detects light tails 0.2%. Rejecting when any of the five exceeds its own 5% point rejects 13.9% of normal samples; correcting that search to 5% gives it 37.4% power against light tails.

A number for the shape

Five one-number summaries of a quantile plot's departure from its line, each a correct 5% test on forty normal observations, and no two agree about what matters. Straightness catches a skewed source 83% of the time and light tails 31%; kurtosis catches light tails 72% and skewness 0.2%. Reading all five and reporting whichever looks bad rejects 13.9% of genuinely normal samples. Correcting that search to 5% costs almost half the power against the departure it would have caught best — and a line-up of twenty panels gets the same correction for nothing.

lineup · Qq Student's t on 5 degrees of freedom, against the normal. The two-sided 95% critical value is 2.571 for t(5) and 1.960 for the normal — 31% wider. Using the normal at this sample size makes every interval too short by that much.

The correction for not knowing the spread

The t distribution exists because the standard deviation is estimated rather than known. At eight observations, using the normal instead makes every interval 12% too short — and the coverage that follows can be measured rather than argued about.

intervals · Student The top of a heavy-tailed population keeps its lead; the top of a light-tailed one gives it back. Select the top share on the first reading and read the group again: the share of its mean lead the second reading keeps, by integration over the true score (lines) and counted on 400,000 draws a parent in 20 batches (points, with two standard errors). The normal keeps exactly 0.6 at every selection. At the top half the Laplace keeps 0.541, the t 0.535 and the uniform 0.648 — the heavy tails keep LESS than the correlation. By the top one per cent the order has reversed: 0.761, 0.784 and 0.443. At one in ten thousand the t keeps 0.977 and the uniform 0.346.

A lead that a heavy tail keeps

Four populations whose readings all correlate at exactly 0.6, and whose least-squares slopes all read 0.6. Select the top one per cent on one reading and measure them again: they keep 60% of their lead if the true scores are normal, 76.1% if they are Laplace, 78.4% if they are a t on four degrees of freedom — and 44.3% if they are uniform. The correlation predicts the regression of the extremes for one shape of population only.

paradox · Rtm How often a quantile plot's shape test sees a skewed error distribution, on the errors and on residuals of fits of more columns. On the forty errors themselves the test flags 82.9%. On the residuals of fits with 2, 5, 10, 20 columns: 79.1%, 71.4%, 51.2%, 23.0%. On 40 − p errors, which is what losing the fitted columns' worth of observations alone would give: 80.6%, 76.1%, 69.1%, 49.5%.

Residuals that look too normal

The quantile plot most often read is of a regression's residuals, and residuals look more normal than the errors behind them. Each residual is a mix of all the errors, and the mix moves towards a bell as the fit takes more columns. On forty rows with skewed errors, the straightness test flags 82.9% of samples of the errors themselves, 79.1% of the residuals of a line, 71.4% with five columns, 51.2% with ten and 23.0% with twenty — far below the 49.5% that twenty errors would give, so the fit costs more than the observations it uses. Light tails vanish faster: 73.0% on the errors, 4.7% on twenty-column residuals. The band's level holds at every design, and one point with leverage 0.73 costs almost nothing; what costs is the number of columns.

lineup · Qq Whether residuals built to be independent see a skewed error better than ordinary ones. The straightness test, each version read at its own 5% point. On all forty errors: 82.9%. At 2, 5, 10, 20 columns: ordinary residuals 79.1%, 71.4%, 51.2%, 23.0%; BLUS residuals 76.4%, 64.9%, 38.3%, 16.2%; recursive residuals 64.4%, 49.5%, 25.9%, 9.1%; 40 − p errors 80.6%, 76.1%, 69.1%, 49.5%.

Residuals made independent

Recursive residuals — each row predicted from the fit to the rows before it — are exactly independent normals under the null for any design, so a quantile plot of them has an exact band. They see a skewed error worse than the ordinary residuals do, not better: 25.9% against 51.2% at ten columns on forty rows, 9.1% against 23.0% at twenty. BLUS residuals, the independent set closest to the errors, reach 38.3% and 16.2%. Every such set is a rotation of the ordinary residuals, so none holds information they lack, and independence is bought by mixing some residuals almost entirely out of other rows' errors — the least pure recursive residual at twenty columns is 0.4% its own.

lineup · Qq The excess harm of the thresholds validation studies recommend, when the diseased scores are no lower tail. A thousand studies of a hundred diseased and three hundred healthy people, prevalence 0.26, a miss costing a hundred false alarms. The best threshold flags 7.6% of healthy people. Excess harm per hundred people, and the share recommending that everyone be flagged: the study's own curve 16.26 (0.0%); a binormal fit 67.08 (89.7%); a two-normal mixture fit 22.71 (2.3%); binormal if normality passes 16.26 (0.0%).

A test whose tail is not normal

A validation study that fits a normal curve to each group saw the tail where a threshold jumps — when the diseased scores really were normal. Give the same published test a diseased group with no lower tail at all, and the binormal fit invents one: 89.7% of studies recommend flagging everyone, adding 67.08 false alarms' worth of harm per hundred people to a best threshold that costs 5.61. Give it a lump of mild cases instead, and the binormal fit misses the tail that is there and does worse than the study's own curve. A normality test on the hundred diseased scores sees the missing tail every time and the lump 60.4% of the time, and a study that fits the binormal model only when normality is not rejected is never the worst of the four.

screening · Baserate How often a quantile plot sees a skewed error when the fitted columns were chosen from the data. Forty rows, 3,000 samples a point, read against the band for forty independent normals. With the intercept alone the test flags 82.5%. With 1, 2, 5, 10 columns fixed in advance: 79.9%, 76.1%, 65.4%, 47.0%; with the same number picked by a forward search from twenty candidates unrelated to the outcome: 66.6%, 55.5%, 36.8%, 25.8%.

Residuals of a model the data chose

Five columns picked by a forward search from twenty candidates that have nothing to do with the outcome leave residuals that show a skewed error 36.8% of the time, where five columns fixed in advance show it 65.4% — the power of a fit of 13.4 fixed columns. Against light-tailed errors the chosen fit's plot is blind, 8.5% against 34.5%. Simulating the search on normal errors gives a band that holds 5%, which the ordinary band already nearly did; the level was never what broke. A reader told the final model has five columns is reading a plot with the power of thirteen.

lineup · Qq

Named alongside it

The objects these essays reach for when they reach for this one.

Q–Q plotVisual inferenceModel diagnosticsCoverageSample sizeSkewnessStatistical powerLeverageResidualSampling variationStudent's tCentral limit theorem

All concepts