Normal Distribution
nothing is called 95% until it has been counted
The distribution itself

Sums of almost anything

The theorem says sums converge on one shape whatever they are sums of, which is remarkable and true. Watching it happen from a one-sided skewed source, with the rate of convergence predicted in advance, is more convincing than watching the shape appear.
11 min read 6 figures Two routes to a numberOne run is an anecdote

The central limit theorem is the reason one distribution turns up everywhere, and it deserves its reputation. Add up enough independent draws from almost any distribution, standardise the total, and the result approaches the same bell — regardless of what was being added up.

8 exponential draws, standardised, against the normalThe source is one-sided and skewed. At n = 8 the standardised sum has skew 0.747, and the theory says 2/sqrt(n) = 0.707 — so the convergence is visible AND its rate is predicted.00.1000.2000.3000.400-2024standardised sumdensityskew 0.7472/√n = 0.70740,000 sums, one seed eachthe rate is predicted, not just the shape
Fig. 1 Sums of exponential draws, standardised. The source is one-sided and heavily skewed; the sum is neither. The number in the corner is the measured skew against what the theory says it should be.

Starting from something that is not a bell

The demonstration is only interesting if the source is genuinely unlike the answer. The exponential distribution is a good choice: it lives entirely on the positive half line, it has its mode at zero, and its skewness is 2 — pronounced, and in a fixed direction.

Sum n of them, subtract the mean, divide by the standard deviation, and plot. At n = 1 the picture is the exponential itself, which looks nothing like a bell. By n = 8 it is close. By n = 30 the difference is hard to see.

That much is the standard demonstration. What makes it a measurement rather than an illustration is the number beside it.

The normal density at sigma = 1.00The bands hold 68.27%, 95.45%, 99.73% of the mass. Those figures are integrals of the curve drawn, not the memorised 68-95-99.7.00.1000.2000.3000.400-4-202standard deviations from the meandensity68.27%95.45%99.73%the bands are integrated, not recalledsigma = 1.00
Fig. 2 The limit itself, with the mass in each band integrated rather than recalled.

Predicting the rate before drawing it

The skewness of a standardised sum of n independent copies of a distribution with skewness γ is exactly γ/√n. For the exponential, γ = 2, so the standardised sum of n draws should have skewness

2n\frac{2}{\sqrt{n}}

That is a prediction with no free parameters. At n = 8 it says 0.707; at n = 30 it says 0.365.

The figure computes the skew from the forty thousand simulated sums and prints both numbers. They agree. Every drag position is a fresh test of the same prediction, and the assertion behind the figure requires the agreement — so a frame in which the theory and the measurement disagreed would fail the build rather than being drawn.

This is what separates a demonstration from an argument. Anyone can show a histogram approaching a curve. Showing that the rate of approach is what the theory says is a claim the figure could fail.

Twenty samples of 40, every one of them genuinely normalEach panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 1.52 standard deviations off the line. Anything a reader would reject here would be a false alarm.20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like
Fig. 3 Twenty samples that really are normal. Judging convergence by eye is harder than it looks.

What the theorem requires

The theorem is often stated loosely enough to sound universal, and it is not. Three conditions, each of which fails somewhere in practice.

Finite variance. A Cauchy distribution has none, and sums of Cauchy draws are Cauchy however many are added — the standardised sum does not converge to anything. This is not a pathological curiosity: heavy-tailed distributions with infinite or barely finite variance turn up in finance, in network traffic and in city sizes.

Independence. Correlated draws sum differently, and the effective sample size is smaller than the count. This is the commonest failure in real data, and it usually shows up as an interval that is far too narrow.

Identical distribution, or a condition weak enough to replace it. Sums of things with wildly different scales are dominated by the largest, and the sum inherits its shape rather than the limit’s.

Each of those is a reason a real dataset might not behave as the theorem promises, and none of them is visible in a picture of a converging histogram.

Student's t on 5 degrees of freedom, against the normalThe two-sided 95% critical value is 2.571 for t(5) and 1.960 for the normal — 31% wider. Using the normal at this sample size makes every interval too short by that much.00.1000.2000.3000.400-4-2024standard errors from the meandensityt 2.57z 1.96solid: t · dashed: normal31% wider at 5 df
Fig. 4 A distribution that is not normal and is often treated as though it were, with the difference in critical value marked.

The part that gets over-read

The theorem says the distribution of the sum approaches normality. It says nothing about the distribution of the individual observations, and the inference in the other direction is not available.

Heights being roughly normal is not a consequence of the central limit theorem unless height is a sum of many small independent contributions of comparable size — which is an empirical claim about biology, not a mathematical one. Plenty of quantities that are sums are not usefully normal at any realistic n, and plenty of quantities that are roughly normal are not sums of anything.

The theorem licenses a statement about a sampling distribution. It is used, constantly, to license a statement about a population, and those are different claims.

Where the normal approximation converges, and where it does notRelative error against the exact binomial. At n = 1280 the error at the median is 0.96% and three sigma out it is 25.7% — a factor of 27. The tail is where the approximation is used.sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last
Fig. 5 The same convergence measured at the median and in the tail. They do not arrive together.
The normal approximation's error in the tail, n = 100, p = 0.05Approximate tail probability divided by the exact one. At one sigma the ratio is 0.98; at 4 sigma it is 0.104, so a rare event is understated by a factor of 10.1 sigma out0.982exact 1.3e-12 sigma out0.691exact 2.8e-23 sigma out0.335exact 4.3e-34 sigma out0.104exact 4.6e-41.0 would be exactapproximate ÷ exactunderstated where it matters most
Fig. 6 How far out the approximation is at one sample size, as a ratio to the exact answer.

Where the demonstration stops being enough

The convergence shown here is of the shape, judged by eye and by one summary number. That is the right thing to show first and it is not the thing that decides whether an approximation is usable.

Convergence in the tail is far slower than convergence in the middle, and the tail is where a p-value or a control limit is read. The histogram above has converged to the eye at n = 8 and is still substantially wrong three standard deviations out at n = 1280.

So the honest summary of this essay is: the theorem is true, its rate in the middle is predictable and confirmed here, and neither of those facts establishes that the approximation is safe for the use it is about to be put to.

The rate, and the constant in front of it

The skew check confirms one moment. The general statement about how fast the whole distribution converges is the Berry–Esseen theorem, and it is worth knowing because it puts a bound on the error rather than describing a tendency.

For independent identically distributed draws with finite third moment, the maximum difference between the standardised sum’s distribution function and the normal’s is at most

Cρσ3n\frac{C\rho}{\sigma^3\sqrt{n}}

where ρ is the third absolute moment and C is a universal constant, known to be below 0.47.

Two things follow. The convergence is 1/√n, which is slow: halving the error needs four times the data. And the constant involves the source distribution’s third moment, so a badly skewed source converges from further away — which is what the exponential demonstration is showing when the skew at n = 1 is 2 and at n = 30 is 0.365.

The bound is on the largest difference anywhere in the distribution. It says nothing about the relative error in the tail, which is a different quantity behaving much worse.

Sums of unequal things

The identical-distribution requirement can be weakened, and the weakened version is more useful because real sums are rarely of identical things.

Lindeberg’s condition is the standard replacement, and informally it says: no single term may dominate. If the variance of the sum is not concentrated in a few of its components, convergence still holds even when the components have different distributions.

That is the condition to check in practice, and it fails more often than the independence one is usually suspected of failing. A total that is mostly one large contribution plus many small ones inherits the shape of the large contribution, however many small ones there are. Income, city sizes, insurance losses and file sizes all have that character.

The failure is easy to test for and rarely tested: compute each component’s share of the total variance and look at the largest.

What breaks with dependence

Independence is the assumption most often violated and the least often noticed, so it is worth being concrete about the damage.

Positively correlated draws sum with a larger variance than n times the individual variance. The standardised sum still converges to a normal under conditions, but the standardisation constant is wrong if computed as though independent — so the practical failure is not a wrong shape, it is a wrong scale.

That produces intervals and tests that are too narrow, by a factor that depends on the correlation and can be large. Repeated measures on the same subject, observations close in time, and anything clustered by site or school or household all have it.

There is no correction that works without modelling the dependence, which is why the design of the study rather than the analysis is where this is usually decided.

Why the theorem gets misapplied

Two habits worth naming, since both are common.

Using it to justify treating the data as normal. The theorem is about the sampling distribution of a sum or mean, not about the population. Nothing in it says a single observation is normal, and procedures that depend on the observations rather than on their mean get no protection from it.

Treating n = 30 as sufficient. The number is folklore. How large n needs to be depends entirely on the source’s skew and on where the distribution is being read — the Berry–Esseen constant contains the third moment for exactly this reason. For a badly skewed source read in the tail, n = 30 is nowhere near enough, and the measurement of that gap is the next essay.

Why this is the most over-quoted theorem in the subject

A closing observation about how the theorem is used, since it is invoked far more often than it applies.

It is correct, it is remarkable, and it licenses one specific thing: treating the sampling distribution of a sum or a mean as approximately normal, at a sample size large enough for the source’s skew, when the draws are independent and no one of them dominates.

It is routinely used instead to license: treating the data as normal, treating a median or a variance or a ratio as normal, ignoring dependence, and reading a tail. None of those follows.

The mismatch matters because the theorem carries enormous authority — it is the one result everyone remembers — and that authority gets transferred to claims it does not support.

The defence is the same as everywhere here. Ask what the normality is needed for, ask at what sample size, and ask where in the distribution the answer will be read. The theorem answers one of those three, and the other two are measurements.

The other limit theorems

The central limit theorem is one of a family, and knowing the neighbours makes clearer what it does and does not say.

The law of large numbers says the sample mean converges to the true mean. That is about the location settling; the central limit theorem is about the shape of the remaining error. They are different statements and the second is much stronger.

The stable laws generalise it. When the variance is infinite, standardised sums still converge — not to a normal, but to another stable distribution with heavier tails. The normal is the special case where the variance is finite, and the family explains what happens when it is not.

Extreme value theory does for maxima what the central limit theorem does for sums: the standardised maximum of many draws converges to one of three distributions. That is the theorem to reach for when the quantity of interest is the largest value rather than the total — and it is why the bootstrap fails for a maximum while working for a mean. Different limit, different machinery.

Knowing which theorem applies to the statistic in hand is most of the work. Applying the central limit theorem to a maximum is a category error that produces a confident wrong answer.

Where the demonstration is honest about itself

A note on the figure’s own limits, since the site’s discipline applies to its own figures.

It shows forty thousand sums at each n, so the histogram is a good estimate of the density — but only where the density is appreciable. In the far tails of the plot there are few observations and the histogram is noisy, which is invisible at the scale drawn.

That is the same asymmetry the essay is about, appearing in the instrument rather than the subject: the body of the picture is well estimated and the tail is not, and no reasonable number of simulations changes that.

Which is why the tail comparison on this site is done against the exact binomial rather than against a simulation. A simulation cannot resolve a probability of 1e-5 without hundreds of millions of draws, and the exact answer is a loop.

Where this goes next

sample sizerelative error10204080160320640128010%1%at the mediantwo sigma outthree sigma outexact binomial against its normal approximationthe tail converges last The tail converges last The central limit theorem is usually shown as a shape arriving. What the demonstration leaves out is the rate — and the rate is wildly different in the middle and in the tail, which is where every approximation in the subject is actually read. 20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like What normal actually looks like A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there. 00.1000.2000.3000.400-4-202standard deviations from the meandensity68.27%95.45%99.73%the bands are integrated, not recalledsigma = 1.00 The shape, and where its mass is 68, 95, 99.7 is recited more often than any other set of numbers in the subject. They are integrals of a specific curve, they are worth computing rather than remembering, and the third one is the one people misuse.