Normal Distribution
nothing is called 95% until it has been counted
The distribution itself

The tail converges last

The central limit theorem is usually shown as a shape arriving. What the demonstration leaves out is the rate — and the rate is wildly different in the middle and in the tail, which is where every approximation in the subject is actually read.
12 min read 6 figures The tail is where it is readTwo routes to a number

Every account of the central limit theorem shows the same thing: sums of one, two, five, thirty draws, and a bell emerging. It is true, it is the right picture, and it answers a question nobody has. The question anyone actually has is how close, at the sample size in hand, at the place it is going to be read — and the answer depends enormously on the second half of that.

Where the normal approximation converges, and where it does notRelative error against the exact binomial. At n = 1280 the error at the median is 0.96% and three sigma out it is 25.7% — a factor of 27. The tail is where the approximation is used.sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last
Fig. 1 Relative error of the normal approximation against the exact binomial, at the median and two and three standard deviations out, as the sample grows. Both fall. They do not fall together.
The normal approximation's error in the tail, n = 100, p = 0.05Approximate tail probability divided by the exact one. At one sigma the ratio is 0.98; at 4 sigma it is 0.104, so a rare event is understated by a factor of 10.1 sigma out0.982exact 1.3e-12 sigma out0.691exact 2.8e-23 sigma out0.335exact 4.3e-34 sigma out0.104exact 4.6e-41.0 would be exactapproximate ÷ exactunderstated where it matters most
Fig. 2 The approximation divided by the exact answer, at one sample size, as the threshold moves out.

Reading a distribution somewhere

Almost nothing in practice reads a distribution at its median. A p-value is the probability of a statistic at least as extreme — a tail. A control limit is a tail. A value at risk is a tail. A specification limit is a tail. The whole apparatus is built on statements about how much probability lies beyond a point that is a long way from the middle.

So the useful question about an approximation is not whether it is close, but whether it is close out there.

The measurement

Take a binomial with p = 0.1, where the exact answer is available by summation and needs no approximation at all. Compare it with its normal approximation at the median, at two standard deviations out, and at three. Do that as n grows from 10 to 1280.

At n = 1280 — a large sample by most standards — the approximation is off by

  • 0.96% at the median, and
  • 25.7% three standard deviations out.

A factor of twenty-seven between the two, at the same sample size, in the same distribution. The middle has converged and the tail has not.

Push further out and it gets worse. At n = 100, four standard deviations out, the normal approximation gives 0.104 times the true tail probability. It understates a rare event by a factor of ten.

8 exponential draws, standardised, against the normalThe source is one-sided and skewed. At n = 8 the standardised sum has skew 0.747, and the theory says 2/sqrt(n) = 0.707 — so the convergence is visible AND its rate is predicted.00.1000.2000.3000.400-2024standardised sumdensityskew 0.7472/√n = 0.70740,000 sums, one seed eachthe rate is predicted, not just the shape
Fig. 3 The shape arriving quickly, which is what the standard demonstration shows and why it misleads.

Why the tail is slower

The mechanism is not mysterious once stated.

The normal approximation is exact in the limit for the centre because the central limit theorem is a statement about the distribution converging at any fixed point. Fix a point, let n grow, and the error there goes to zero at a rate of roughly 1/√n — the Berry–Esseen bound.

But a tail probability is not evaluated at a fixed point. As n grows, “three standard deviations out” moves further into the tail in absolute terms, and the region being approximated is one where the true probability is itself shrinking rapidly. The relative error — which is what matters when the quantity is small — is governed by different mathematics, called large deviations, and it does not converge at 1/√n.

The practical statement: the absolute error shrinks quickly and the relative error does not. In the tail the probabilities are small, so relative error is the only kind worth reporting, and it is the kind that behaves badly.

The normal density at sigma = 1.00The bands hold 68.27%, 95.45%, 99.73% of the mass. Those figures are integrals of the curve drawn, not the memorised 68-95-99.7.00.1000.2000.3000.400-4-202standard deviations from the meandensity68.27%95.45%99.73%the bands are integrated, not recalledsigma = 1.00
Fig. 4 The density at the scale the usual picture is drawn at. The tails are visually indistinguishable from zero here.

What makes the usual demonstration misleading

Nothing in the standard picture is false. The problem is what it invites the reader to conclude.

It shows the shape, which converges quickly, and shows it at a scale where the tails are visually indistinguishable from zero. A reader watching the histogram settle onto the curve concludes that by n = 30 the approximation is good, because at the resolution of the picture it is.

At n = 30 the approximation is good in the middle and poor in the tail, and the picture cannot show that because a difference between 0.0011 and 0.0004 is invisible at any scale where the body of the distribution is also visible.

That is why the figure above is on a log scale and reports relative error. It is a less beautiful picture and it answers the question.

Coverage of four nominal 95% intervals, n = 30Computed exactly by summing over all 31 possible counts, not simulated. The Wald interval drops to 26.0% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted
Fig. 5 An interval failing for the same reason: it is a tail approximation, read in the tail.

Where this bites

Three ordinary situations where the tail is what is being read and the sample is not large.

A test of a proportion. The z-test for a proportion is exactly this approximation, evaluated in the tail. That is one of the reasons the standard interval fails — it is the same approximation, used the same way.

A rare adverse event. Two events in a hundred patients, and the question is whether that is consistent with a background rate. The relevant probability is a tail probability at small n and small p, which is the corner where the approximation is worst.

Anything with a stated sigma level. A process quoted at “six sigma” is making a claim about a probability of about two in a billion, derived from a normal assumption. No finite sample can validate a normal assumption that far out, and the departures that matter are invisible in any sample that could be collected.

Student's t on 5 degrees of freedom, against the normalThe two-sided 95% critical value is 2.571 for t(5) and 1.960 for the normal — 31% wider. Using the normal at this sample size makes every interval too short by that much.00.1000.2000.3000.400-4-2024standard errors from the meandensityt 2.57z 1.96solid: t · dashed: normal31% wider at 5 df
Fig. 6 A correction that is itself a tail statement, and whose size is largest where the sample is smallest.

The honest response

Not to abandon the approximation, which is useful and often the only thing available. The response is smaller and more specific.

State where it is being read. An approximation quoted without a location is not a quantified claim.

Use the exact answer when it exists. For a binomial it always does — the sum is a loop. This site’s entire tail comparison is possible only because the exact answer is available to compare against, and where it is available in practice there is no argument for using the approximation at all.

And treat a tail claim as weaker than a body claim from the same data. They are not the same evidence, and the difference is measurable, which is what the figure above is for.

Absolute error and relative error

The distinction doing all the work here, and worth separating cleanly because the two diverge exactly where it matters.

Absolute error is the difference between the approximate and exact probabilities. It is what Berry–Esseen bounds, it shrinks at 1/√n, and it is largest in the body of the distribution simply because that is where the probabilities are large.

Relative error is that difference divided by the true probability. In the tail the true probability is tiny, so a small absolute error becomes a large relative one.

Which matters depends on the use. If the quantity is a proportion of a population, absolute error is right. If it is a rate of a rare event — a false-positive rate, a failure probability, a p-value — then relative error is the only meaningful measure, because being wrong by a factor of ten is the whole story and the absolute difference is unremarkable.

Almost every use of the tail is the second kind, and almost every reassurance about the central limit theorem is about the first.

Large deviations, briefly

The mathematics governing the far tail is not the central limit theorem at all, and naming it explains why the behaviour is so different.

Tail probabilities beyond a fixed number of standard deviations are governed by large deviation theory, where the probability decays exponentially in n with a rate given by the source distribution’s own structure rather than by its variance.

The consequence is that the normal approximation, which is driven by the variance alone, gets the exponential rate wrong unless the source is normal. That error compounds as the threshold moves out, which is why the ratio in the figure falls away rather than settling.

So the two regimes are not the same theorem at different strengths. They are different theorems, and the crossover between them is what the middle-and-tail figure is tracing.

A continuity correction is not the fix

The obvious first response is that the approximation is being applied to a discrete distribution without care, and that the half-unit continuity correction will handle it.

It is applied in every figure here, and it helps in the body. It does not rescue the tail, because it corrects for the discreteness of the grid rather than for the shape of the decay. The residual error is a shape error and no offset repairs it.

Better approximations exist — saddlepoint methods, Edgeworth expansions with higher moments — and they do substantially better in the tail because they carry information about the source beyond its variance. They are also much less used, and the reason is that the normal approximation is the one everybody has memorised.

The practical rule

Three lines, since the essay is otherwise an argument.

Use the exact answer where it exists. For a binomial, a Poisson, or a hypergeometric it always does, and it is a loop. There is no defensible reason to approximate a binomial tail on a computer.

Where the exact answer does not exist, state the sample size and the threshold together. “The p-value is 0.002” and “the p-value is 0.002, from a normal approximation at n = 40, four standard deviations out” are different claims, and only the second can be checked.

And treat a far-tail number as a property of the model rather than of the data. No sample contains the observations that would test the approximation four sigma out — which is the same limit the empirical rule runs into, and the honest way to report it is as an assumption.

What a sample can and cannot check

The essay’s practical conclusion depends on a limit that is worth stating on its own, because it applies well beyond this figure.

To observe an event of probability one in a thousand takes, on average, a thousand observations. To estimate its probability to within a factor of two takes a few thousand. To say anything at all about a probability of one in a million takes millions.

So the far tail of a fitted distribution is never validated by the data the fit was made from. It is an extrapolation from the shape assumed in the middle, and its accuracy depends entirely on whether that shape continues.

Real distributions frequently do not continue. Mixtures, occasional different mechanisms, contamination and rounding all leave the body looking normal and change the tail. None of those is visible in a sample that contains no tail observations, which is every sample of realistic size.

This is why a six-sigma claim is a statement about a model. It is not pessimism about the model; it is arithmetic about what could ever have tested it.

The Edgeworth correction, and what it shows

There is a systematic way to do better than the normal approximation, and it is worth knowing because it identifies exactly what the normal is missing.

An Edgeworth expansion writes the true distribution as the normal plus correction terms involving the source’s higher moments — skewness first, then kurtosis. The leading correction is of order 1/√n and involves the skewness, which is why the skew of a standardised sum is the natural thing to measure.

Two things follow. The normal approximation is the first term of a series, so its error is governed by the next term, and a source with large skewness converges from further away. And the correction terms are largest in the tail, which is another route to the same conclusion this essay reaches by measurement.

Edgeworth expansions are not much used in practice, partly because the corrected density can go negative in the far tail — the expansion is asymptotic rather than convergent. That is itself informative: even the systematic improvement misbehaves precisely where the problem is worst.

Why exact answers are usually available and rarely used

The recommendation “use the exact answer” deserves a note on why it is not the default, because the reasons are historical rather than technical.

The normal approximation exists because the exact computation was infeasible by hand. Summing a binomial tail at n = 200 is a couple of hundred multiplications, which is an afternoon with a table of logarithms and a few seconds now.

The tables, the z-scores, the rules of thumb about np > 5 — all of that machinery is a workaround for an arithmetic constraint that has not existed for fifty years. It survives because it is what the textbooks contain, and the textbooks contain it because that is what their authors were taught.

The practical consequence is that a great deal of software still approximates by default. It is worth checking what a given function actually computes, because for the distributions in this essay an exact routine is usually available and usually not the default.

Where this goes next

00.1000.2000.3000.400-2024standardised sumdensityskew 0.7472/√n = 0.70740,000 sums, one seed eachthe rate is predicted, not just the shape Sums of almost anything The theorem says sums converge on one shape whatever they are sums of, which is remarkable and true. Watching it happen from a one-sided skewed source, with the rate of convergence predicted in advance, is more convincing than watching the shape appear. 0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted What the 95% refers to An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time. 0.8000.8500.9000.950255075100sample sizecoverage at a true proportion of 0.15n = 19n = 20WilsonWaldexact coverage at every n from 10 to 120more data is not automatically better here More data is not monotonically better Coverage of an interval for a proportion does not improve smoothly as the sample grows. It oscillates, and there are larger samples that cover materially worse than smaller ones — a sample of twenty covers twelve points worse than a sample of nineteen.