Normal Distribution
nothing is called 95% until it has been counted
The distribution itself

What normal actually looks like

A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there.
11 min read 6 figures One run is an anecdoteCount it, do not claim it

“Does this data look normal?” is asked constantly and answered by eye. The answer by eye is unreliable for a reason that has nothing to do with skill: nobody has a calibrated sense of how much a genuinely normal sample wanders.

Twenty samples of 40, every one of them genuinely normalEach panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 1.52 standard deviations off the line. Anything a reader would reject here would be a false alarm.20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like
Fig. 1 Twenty quantile-quantile plots. Every one is forty draws from a normal distribution — there is no non-normal panel hidden among them.

The instrument

A quantile-quantile plot sorts the sample, plots each value against the value the theoretical distribution would put in that position, and draws the line the points would follow if the data really came from that distribution.

If the data is normal, the points fall near the line. If it is skewed, they curve. If it is heavy-tailed, the extremes pull away at both ends. It is a good instrument and the reading of it is the problem.

Twenty 95% intervals for a proportion that really is 0.350 of the twenty miss the true value. The 95% is a property of the procedure across repetitions — no single interval has a 95% chance of anything, because it either contains 0.35 or it does not.the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval
Fig. 2 The same argument for intervals: twenty of them, because one cannot be read.

The calibration nobody has

Look at the panels above. Several of them curve at one end. At least one has a point sitting well off the line — the worst departure across the twenty is more than half a standard deviation. Presented singly, with the question “is this normal?”, most people would say no about two or three of them.

All twenty are normal. The wandering is what forty points from a normal distribution do.

That is the whole content of the figure and it cannot be conveyed by a single panel, however carefully captioned. The only way to acquire the calibration is to see the distribution of pictures rather than one picture — which is the same argument as showing twenty intervals rather than one, applied to a different object.

8 exponential draws, standardised, against the normalThe source is one-sided and skewed. At n = 8 the standardised sum has skew 0.747, and the theory says 2/sqrt(n) = 0.707 — so the convergence is visible AND its rate is predicted.00.1000.2000.3000.400-2024standardised sumdensityskew 0.7472/√n = 0.70740,000 sums, one seed eachthe rate is predicted, not just the shape
Fig. 3 What the sampling distribution of a mean does as n grows, which is what most procedures actually depend on.

Sample size changes everything

Drag the panel size and the character changes completely.

At ten points the plots are almost uninformative. Nearly any distribution produces a plausible-looking line at that size, and nearly any normal sample produces something a suspicious reader could reject. A normality judgement from ten observations is essentially a coin toss.

At four hundred the panels are tight and a real departure would be obvious. But at four hundred observations, a test of normality will also reject for departures far too small to matter for any practical purpose — which is the opposite failure.

That combination is the awkward truth about normality checking: at small n a departure that matters cannot be detected, and at large n departures that do not matter are.

Coverage of a 95% interval for a mean, n = 8Measured over 20,000 samples. The t interval covers 95.0% and the z interval 91.3%. The difference is the price of pretending the standard deviation was known.t interval, 7 df95.0%± 0.3 at 2 s.e.z interval91.3%± 0.4 at 2 s.e.20,000 samples, one seed eachnominal 95%counted, not assumed
Fig. 4 A case where the normality assumption is doing real work, with the coverage counted.
Where the normal approximation converges, and where it does notRelative error against the exact binomial. At n = 1280 the error at the median is 0.96% and three sigma out it is 25.7% — a factor of 27. The tail is where the approximation is used.sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last
Fig. 5 And the reason a tail departure matters more than a departure in the body.

What to do instead

Three responses, in increasing order of usefulness.

Ask what the normality is needed for. Very often it is not needed at all. A t interval for a mean is robust to moderate non-normality at moderate n, because it relies on the sampling distribution of the mean rather than on the distribution of the data. A prediction interval for a single future observation relies on the distribution of the data directly, and is not robust at all. Those two get conflated constantly.

Look at the tails specifically, and say so. If the procedure reads a tail, then a tail departure matters and a centre departure may not. The tail is where approximations fail, and it is also where a QQ plot has the fewest points and the most noise — the two problems compound.

Show the reference panels. If a QQ plot is going into a report, put a handful of simulated normal panels beside it at the same sample size. The reader can then judge the real one against what the noise looks like, which is the judgement they were implicitly being asked to make anyway.

20,000 p-values from a true null, n = 12Flat, as it must be: under the null a p-value is uniform on (0,1). The Kolmogorov–Smirnov distance from uniform is 0.0165 (p = 0.13). That flatness is the check that catches an error a single rejection rate would miss.05001e+30.2000.4000.6000.800p-valuecountflatKS distance 0.0165a p-value that is not uniform is not a p-value
Fig. 6 Generating the null and looking at what it produces, in the setting where it is most standard.

The general form

This essay is a specific case of the rule this site applies everywhere: a single realisation of a random process cannot be interpreted without knowing the distribution of realisations.

That applies to a QQ plot, to a confidence interval, to a p-value, to a scatterplot that looks like it has a trend in it, and to a striking result in a small study. In every case the useful move is the same — generate the null and look at what it produces, then ask whether the observed result is unusual against that.

It is cheap, it needs no theory beyond the ability to simulate, and it is the single most reliable protection against reading structure into noise.

Reading the shape of a departure

The panels above are all normal, so nothing in them means anything. When a QQ plot does depart, the direction of the departure is informative, and it is worth having the four standard shapes to hand.

Both ends curving away from the line, above at the right and below at the left. Heavy tails: extreme values further out than a normal would produce. This is the departure that matters most for anything read in the tail, and it is the commonest in real data.

Both ends curving towards the line. Light tails, a distribution more compact than a normal. Rare in measurements, common in anything bounded.

One end curving and the other straight. Skew. The curving end is the long tail.

A step or a plateau. Discreteness or rounding — measurements recorded to the nearest unit produce visible horizontal runs, and a plot that looks like a staircase is usually reporting the recording rather than the process.

Each of those is a real diagnosis. What none of them supports is a verdict of “not normal” from wobble alone, which is what the twenty panels exist to inoculate against.

What a formal test adds, and takes away

The obvious response to unreliable eyeballing is to run a test — Shapiro–Wilk, Anderson–Darling, Kolmogorov–Smirnov. Each returns a p-value for the null that the data is normal.

That helps less than it appears, for a reason with the same shape as everything else on this site.

At small n the test has almost no power. It will fail to reject for almost any distribution, so a non-significant result says nothing — which is exactly the situation the power essay describes.

At large n the test has enormous power, and will reject for departures far too small to affect any procedure being contemplated. Real data is never exactly normal, so with enough of it the test always rejects, and the rejection carries no information about whether it matters.

The test is therefore informative in a narrow band of sample sizes and misleading either side of it. That is not an argument against running it; it is an argument for reporting the size of the departure alongside the p-value, which is the same recommendation the rest of this site keeps arriving at.

What actually depends on normality

The most useful move is to stop asking whether the data is normal and ask what the normality was needed for, because the answer varies enormously.

A t interval for a mean at moderate n: barely depends on it. The relevant distribution is that of the mean, which the central limit theorem makes approximately normal whatever the data looks like. This is the case people worry about most and should worry about least.

A t interval for a mean at n = 5: depends on it completely. There is no central limit theorem at five observations, and the interval’s guarantee is conditional on the assumption.

A prediction interval for one future observation: depends on it completely at any n. The interval is about the distribution of the data, not of its mean, and more data never makes the data more normal.

Anything reading a far tail: depends on it in the one place a sample can never check, since the observations that would test the tail are the ones a sample does not contain.

That list is more useful than any test, because it converts an unanswerable question about the data into an answerable one about the procedure.

The same problem in every other diagnostic plot

The calibration failure is not specific to quantile plots, and the response transfers without modification.

A residual plot is judged for structure by eye, and random residuals produce apparent structure routinely. Twenty simulated residual plots from a correctly specified model show what “no structure” looks like, and the real one can be judged against them.

A scatterplot is judged for a trend, and small samples produce visible trends from nothing. The same twenty-panel treatment applies.

A time series is judged for cycles, and noise produces apparent periodicity — this is the oldest version of the problem, and it has a name, the Slutsky–Yule effect.

The general procedure is called a line-up: put the real plot among nineteen null ones, in a random position, and see whether it can be identified. If it cannot, whatever was seen in it is not distinguishable from noise. If it can, the identification has a stated false-positive rate of one in twenty.

That turns visual inspection into a test with a calibrated error rate, which is a substantial improvement on judgement and requires nothing but the ability to simulate the null.

Why the eye is bad at this specifically

The failure is not general visual incompetence. It is a specific and well-documented bias, and knowing its shape helps.

People are very good at detecting systematic departures — a curve, a step, a cluster — and very poor at judging how much scatter is normal. The visual system is built to find structure, so it finds it, and there is no internal reference for what structureless looks like at a given sample size.

The QQ plot compounds this because it sorts the data. Sorted values are strongly correlated with their neighbours by construction, so the points do not wander independently: a single unusual value pulls a whole run of the plot away from the line, producing exactly the smooth systematic-looking departure the eye is tuned to detect.

So the plot presents noise in the form the visual system is least equipped to dismiss. That is why the calibration has to be supplied externally, by showing what the noise looks like, and why no amount of experience fully substitutes.

What twenty panels is, formally

The figure is doing something that has a name and a stated error rate, which is worth making explicit.

Placing the real plot at random among nineteen null ones and asking whether it can be picked out is a visual hypothesis test. If the data really comes from the null distribution, the chance of identifying the real panel is one in twenty, so a correct identification is significant at the 5% level.

That converts an impression into an inference with a calibrated false-positive rate, using nothing but simulation and the reader’s eye. It is one of the few places where informal judgement can be given a formal guarantee.

The figure here is the simpler version — twenty nulls with no real panel hidden, so its purpose is calibration rather than testing. Building the full line-up is a small extension and the right thing to do when a specific dataset is being judged.

Where this goes next

00.1000.2000.3000.400-2024standardised sumdensityskew 0.7472/√n = 0.70740,000 sums, one seed eachthe rate is predicted, not just the shape Sums of almost anything The theorem says sums converge on one shape whatever they are sums of, which is remarkable and true. Watching it happen from a one-sided skewed source, with the rate of convergence predicted in advance, is more convincing than watching the shape appear. the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval Twenty intervals and one expected miss The 95% belongs to the procedure, not to the interval in front of you. Twenty intervals from twenty samples make that visible in a way no definition does, and the one that misses is not a mistake. sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last The tail converges last The central limit theorem is usually shown as a shape arriving. What the demonstration leaves out is the rate — and the rate is wildly different in the middle and in the tail, which is where every approximation in the subject is actually read.