The tail converges last

Every account of the central limit theorem shows the same thing: sums of one, two, five, thirty draws, and a bell emerging. It is true, it is the right picture, and it answers a question nobody has. The question anyone actually has is how close, at the sample size in hand, at the place it is going to be read — and the answer depends enormously on the second half of that.

Reading a distribution somewhere #

Almost nothing in practice reads a distribution at its median. A p-value is the probability of a statistic at least as extreme — a tail. A control limit is a tail. A value at risk is a tail. A specification limit is a tail. The whole apparatus is built on statements about how much probability lies beyond a point that is a long way from the middle.

So the useful question about an approximation is not whether it is close, but whether it is close out there.

The measurement #

Take a binomial with p = 0.1, where the exact answer is available by summation and needs no approximation at all. Compare it with its normal approximation at the median, at two standard deviations out, and at three. Do that as n grows from 10 to 1280.

At n = 1280 — a large sample by most standards — the approximation is off by

• 0.96% at the median, and
• 25.7% three standard deviations out.

A factor of twenty-seven between the two, at the same sample size, in the same distribution. The middle has converged and the tail has not.

Push further out and it gets worse. At n = 100, four standard deviations out, the normal approximation gives 0.104 times the true tail probability. It understates a rare event by a factor of ten.

The two rates, and the sample sizes they imply #

The picture shows two curves falling at different speeds, and the speeds have a form worth stating because it prices the difference.

The absolute error of a normal approximation to a sum falls like $1/\sqrt{n}$, uniformly — that is the Berry–Esseen statement, and it is the same at every threshold.

The relative error in the tail does not. The leading correction carries a factor that grows like the cube of how many standard deviations out the reading is taken, so the relative error behaves roughly like $z^3/\sqrt{n}$.

A reading three standard deviations out therefore carries about twenty-seven times the relative error of one taken at a single standard deviation — and closing that gap by brute force takes $27^2$, or about seven hundred times the sample.

That is the whole of why the usual demonstration misleads. It shows the absolute error, which is what converges nicely and is what a picture of a bell displays; the quantity a tail probability is read for is the relative one, and no sample size a reader will meet makes it small at three standard deviations.

Why the tail is slower #

The mechanism is not mysterious once stated.

The normal approximation is exact in the limit for the centre because the central limit theorem is a statement about the distribution converging at any fixed point. Fix a point, let n grow, and the error there goes to zero at a rate of roughly 1/√n — the Berry–Esseen bound.

But a tail probability is not evaluated at a fixed point. As n grows, “three standard deviations out” moves further into the tail in absolute terms, and the region being approximated is one where the true probability is itself shrinking rapidly. The relative error — which is what matters when the quantity is small — is governed by different mathematics, called large deviations, and it does not converge at 1/√n.

The practical statement: the absolute error shrinks quickly and the relative error does not. In the tail the probabilities are small, so relative error is the only kind worth reporting, and it is the kind that behaves badly.

What makes the usual demonstration misleading #

Nothing in the standard picture is false. The problem is what it invites the reader to conclude.

It shows the shape, which converges quickly, and shows it at a scale where the tails are visually indistinguishable from zero. A reader watching the histogram settle onto the curve concludes that by n = 30 the approximation is good, because at the resolution of the picture it is.

At n = 30 the approximation is good in the middle and poor in the tail, and the picture cannot show that because a difference between 0.0011 and 0.0004 is invisible at any scale where the body of the distribution is also visible.

That is why the figure above is on a log scale and reports relative error. It is a less beautiful picture and it answers the question.

Where this bites #

Three ordinary situations where the tail is what is being read and the sample is not large.

A test of a proportion. The z-test for a proportion is exactly this approximation, evaluated in the tail. That is one of the reasons the standard interval fails — it is the same approximation, used the same way.

A rare adverse event. Two events in a hundred patients, and the question is whether that is consistent with a background rate. The relevant probability is a tail probability at small n and small p, which is the corner where the approximation is worst.

Anything with a stated sigma level. A process quoted at “six sigma” is making a claim about a probability of about two in a billion, derived from a normal assumption. No finite sample can validate a normal assumption that far out, and the departures that matter are invisible in any sample that could be collected.

The honest response #

Not to abandon the approximation, which is useful and often the only thing available. The response is smaller and more specific.

State where it is being read. An approximation quoted without a location is not a quantified claim.

Use the exact answer when it exists. For a binomial it always does — the sum is a loop. This site’s entire tail comparison is possible only because the exact answer is available to compare against, and where it is available in practice there is no argument for using the approximation at all.

And treat a tail claim as weaker than a body claim from the same data. They are not the same evidence, and the difference is measurable, which is what the figure above is for.

Absolute error and relative error #

The distinction doing all the work here, and worth separating cleanly because the two diverge exactly where it matters.

Absolute error is the difference between the approximate and exact probabilities. It is what Berry–Esseen bounds, it shrinks at 1/√n, and it is largest in the body of the distribution simply because that is where the probabilities are large.

Relative error is that difference divided by the true probability. In the tail the true probability is tiny, so a small absolute error becomes a large relative one.

Which matters depends on the use. If the quantity is a proportion of a population, absolute error is right. If it is a rate of a rare event — a false-positive rate, a failure probability, a p-value — then relative error is the only meaningful measure, because being wrong by a factor of ten is the whole story and the absolute difference is unremarkable.

Almost every use of the tail is the second kind, and almost every reassurance about the central limit theorem is about the first.

Large deviations, briefly #

The mathematics governing the far tail is not the central limit theorem at all, and naming it explains why the behaviour is so different.

Tail probabilities beyond a fixed number of standard deviations are governed by large deviation theory, where the probability decays exponentially in n with a rate given by the source distribution’s own structure rather than by its variance.

The consequence is that the normal approximation, which is driven by the variance alone, gets the exponential rate wrong unless the source is normal. That error compounds as the threshold moves out, which is why the ratio in the figure falls away rather than settling.

So the two regimes are not the same theorem at different strengths. They are different theorems, and the crossover between them is what the middle-and-tail figure is tracing.

A continuity correction is not the fix #

The obvious first response is that the approximation is being applied to a discrete distribution without care, and that the half-unit continuity correction will handle it.

It is applied in every figure here, and it helps in the body. It does not rescue the tail, because it corrects for the discreteness of the grid rather than for the shape of the decay. The residual error is a shape error and no offset repairs it.

Better approximations exist — saddlepoint methods, Edgeworth expansions with higher moments — and they do substantially better in the tail because they carry information about the source beyond its variance. They are also much less used, and the reason is that the normal approximation is the one everybody has memorised.

The practical rule #

Three lines, since the essay is otherwise an argument.

Use the exact answer where it exists. For a binomial, a Poisson, or a hypergeometric it always does, and it is a loop. There is no defensible reason to approximate a binomial tail on a computer.

Where the exact answer does not exist, state the sample size and the threshold together. “The p-value is 0.002” and “the p-value is 0.002, from a normal approximation at n = 40, four standard deviations out” are different claims, and only the second can be checked.

And treat a far-tail number as a property of the model rather than of the data. No sample contains the observations that would test the approximation four sigma out — which is the same limit the empirical rule runs into, and the honest way to report it is as an assumption.

What a sample can and cannot check #

The essay’s practical conclusion depends on a limit that is worth stating on its own, because it applies well beyond this figure.

To observe an event of probability one in a thousand takes, on average, a thousand observations. To estimate its probability to within a factor of two takes a few thousand. To say anything at all about a probability of one in a million takes millions.

So the far tail of a fitted distribution is never validated by the data the fit was made from. It is an extrapolation from the shape assumed in the middle, and its accuracy depends entirely on whether that shape continues.

Real distributions frequently do not continue. Mixtures, occasional different mechanisms, contamination and rounding all leave the body looking normal and change the tail. None of those is visible in a sample that contains no tail observations, which is every sample of realistic size.

This is why a six-sigma claim is a statement about a model. It is not pessimism about the model; it is arithmetic about what could ever have tested it.

The Edgeworth correction, and what it shows #

There is a systematic way to do better than the normal approximation, and it is worth knowing because it identifies exactly what the normal is missing.

An Edgeworth expansion writes the true distribution as the normal plus correction terms involving the source’s higher moments — skewness first, then kurtosis. The leading correction is of order 1/√n and involves the skewness, which is why the skew of a standardised sum is the natural thing to measure.

Two things follow. The normal approximation is the first term of a series, so its error is governed by the next term, and a source with large skewness converges from further away. And the correction terms are largest in the tail, which is another route to the same conclusion this essay reaches by measurement.

Edgeworth expansions are not much used in practice, partly because the corrected density can go negative in the far tail — the expansion is asymptotic rather than convergent. That is itself informative: even the systematic improvement misbehaves precisely where the problem is worst.

Why exact answers are usually available and rarely used #

The recommendation “use the exact answer” deserves a note on why it is not the default, because the reasons are historical rather than technical.

The normal approximation exists because the exact computation was infeasible by hand. Summing a binomial tail at n = 200 is a couple of hundred multiplications, which is an afternoon with a table of logarithms and a few seconds now.

The tables, the z-scores, the rules of thumb about np > 5 — all of that machinery is a workaround for an arithmetic constraint that has not existed for fifty years. It survives because it is what the textbooks contain, and the textbooks contain it because that is what their authors were taught.

The practical consequence is that a great deal of software still approximates by default. It is worth checking what a given function actually computes, because for the distributions in this essay an exact routine is usually available and usually not the default.

The same measurement at two more sample sizes #

The essay’s headline is a single comparison at one sample size, and a single comparison cannot separate “the approximation is bad here” from “the approximation is bad”. Widening it:

At a proportion of 0.05, the ratio of the normal approximation to the exact binomial tail:

• n = 100: 0.104 at four sigma
• n = 400: 0.472 at three sigma, 0.211 at four
• n = 1,600: 0.668 at three sigma, 0.418 at four

Two patterns, and they pull in opposite directions.

The approximation improves as n grows, at every distance out. That is the theorem doing what it promises, and it is why this is an approximation with a rate rather than a mistake.

It improves very slowly in the tail. Sixteen times the sample size takes the four-sigma ratio from 0.104 to 0.418 — better, and still understating the true tail by more than half. To get the four-sigma ratio to within ten per cent of one would take a sample size larger than most studies will ever have.

Set that against the middle of the distribution, where at n = 1,280 the approximation is already within 0.96% at the median while being 25.7% out three sigma away. The same approximation, the same data, and an error thirty times larger in the region that decides whether a rare event is worth worrying about.

That gap between the middle and the tail is the essay’s whole claim, and it is why quoting a single accuracy figure for a normal approximation is meaningless without saying where.

What to do instead, in order of preference #

The essay has been diagnostic; the prescription is short and worth stating explicitly.

Compute the exact answer. For a binomial tail this is a finite sum of n + 1 terms and takes microseconds. The reason it is not done is habit and the residue of an era when it was genuinely expensive, not any live constraint. Every number on this page that the approximation gets wrong was checked against a sum that any spreadsheet could carry out.

Where no exact form exists, use a better approximation than the plain normal. The Edgeworth correction adds a skewness term and recovers much of the tail error; a saddlepoint approximation does substantially better again and is not much harder to implement. Both are strictly better in the region where the plain normal fails, and neither is taught alongside the thing they fix.

Where the tail must be estimated from data, say how far out the data reaches. A sample of a thousand contains essentially no information about a one-in-a-million event, and an estimate of that event’s probability is an extrapolation from a model rather than a measurement. The honest report names the furthest point the data actually constrains, which is usually much closer in than the quantity being asked about.

And where a decision turns on a tail probability, check what the decision does if the probability is wrong by a factor of ten. That is roughly the error measured here at four sigma with a hundred observations, so it is not a pessimistic sensitivity analysis; it is the observed error of the method in common use at a common sample size.