Two standard deviations of what

The mass inside two standard deviations is 95.45%. That figure is an integral of the normal density between −2σ and +2σ, it is exact, and every quantity in it is given: the curve is centred at μ and scaled by σ, and the integration runs from one known point to another.

Nobody has μ or σ. What a reader has is a sample, a mean computed from it and a standard deviation computed from it, and the band they draw is

$$\bar{x} \pm 2s .$$

That is a different object. It is centred at the wrong place by a random amount and scaled by the wrong width by a random factor, and the share of the population it holds is therefore not a number at all — it is a random variable with a distribution.

The distribution, not the average #

At ten observations the band holds 91.1% of the population on average. Its fifth percentile is 74.7%, and it falls short of 95% on 59.9% of samples.

The first of those three numbers is the one that gets quoted and it is the least informative. A band that averages 91% sounds like a band that is slightly too narrow — a four-point correction, the sort of thing a footnote absorbs. What the distribution says instead is that the band is short of its label on three samples in five, and that on one sample in twenty it holds three quarters of the population rather than nineteen twentieths.

The shape is the reason. Content is bounded above by 1 and the distribution piles up against that bound, so the long tail runs the other way — towards bands that hold far too little. An average taken over a distribution with a long left tail and a hard right edge sits well above its own median and tells a reader nothing about the left tail, which is the only part that matters here. The median content at ten observations is 93.40%, and the mean is dragged up from it by the samples that happened to draw a large s.

Where the missing four points go #

Two things are estimated and they move the band in opposite directions.

The centre is wrong by x̄ − μ, which has standard deviation $\sigma/\sqrt{n}$. Shifting a symmetric band off the centre of a symmetric distribution can only reduce what it holds: the mass gained on the near side is smaller than the mass lost on the far side, because the density is falling faster where the band is leaving than where it is arriving. So a mis-centred band always holds less, whichever way it missed, and the loss is second order in the displacement.

The width is wrong by a factor s/σ, which is unbiased in the variance and therefore biased low in the standard deviation — s underestimates σ on average, by about 1/(4n) in relative terms at the sizes in question. A narrow band holds less and a wide one holds more, so this contribution is not one-sided. But it is not symmetric either: content is a concave function of width up here, since the density in the tail is small, so the gain from a band that came out wide is less than the loss from one that came out narrow by the same amount.

Both mechanisms push the same way, and the second is the larger at the sizes where this matters. Together they are the whole of the gap between 95.45% and 91.1%.

It does not go away at any size a reader has #

The natural expectation is that this is a small-sample problem and disappears with data.

It disappears slowly, and not in the way the average suggests. At a hundred observations the mean content is 95.1%, which looks like the target reached. The share of samples whose band is short of 95% is still 43.1%.

That is the fixed point the arithmetic is heading for and it is not the one anybody wants. As n grows, x̄ → μ and s → σ, so the content converges to 95.45% — but it converges to a point, and the point is a value the band exceeds about half the time and falls short of the other half. A band whose content is 95.45% ± nothing is short of 95% on roughly no samples; a band whose content is 95.45% on average, with any spread left at all, is short on a substantial share. The share falls from 60% at ten to 43% at a hundred to 37% at two hundred, and its limit is the share of samples on which the remaining noise happens to point down.

So “two standard deviations holds 95%” is not an approximation that improves. It is a claim about a quantity the reader does not have, applied to a quantity they do, and the error it makes is a different error at every sample size.

What the right multiplier is #

If the band is meant to hold 95% of the population — and to be right about having done so, with stated confidence — then the multiplier is not 2 and it is not 1.96. It is the tolerance factor, and it answers a question with two probabilities in it: find k such that

$$P\big(\,\text{content of } \bar{x} \pm k s \;\ge\; 0.95\,\big) \;=\; 0.95 .$$

Two probabilities, because two things are being promised. The band has to cover a stated share of the population, and the statement that it did has to be right a stated share of the time. Every other band in this collection promises one of those; this is the one that promises both, and the price is visible.

At ten observations the factor is 3.382. That is not a correction to 2; it is 69% wider, and it is wider than the 2.262 that Student’s t gives at nine degrees of freedom, which is the other correction people reach for and which is a correction to something else entirely.

At thirty it is 2.550. At a hundred, 2.233. At three hundred, 2.106. It reaches 1.96 only in the limit, because the confidence half of the promise never becomes free — there is always some chance that s came out small, and the factor has to be wide enough to survive it.

Why the two-sided factor has no closed form and the one-sided one does #

The factor above is Howe’s approximation, and saying so raises the obvious question of what it is approximating and why nobody simply computes that.

The exact two-sided factor solves

$$P_{\bar{x},s}\Big(\Phi(\bar{x} + ks) - \Phi(\bar{x} - ks) \ge p\Big) = 1 - \alpha ,$$

and the inner event is a statement about a difference of two normal integrals whose arguments both move with x̄ and both scale with s. There is no distribution in the standard catalogue for that difference. The equation can be solved numerically — which is what the second route on this page does, by finding each sample’s own k and taking a percentile of them — but it cannot be reduced to a quantile of anything, so the tables are printed from approximations and Howe’s is the one that won.

Change one thing and the difficulty vanishes. Ask for a band with the population on one side of it — an upper bound below which 95% of the population lies — and the event becomes x̄ + ks ≥ z_p, a single inequality. Rearranged, it says that a normal variable with mean z_p$\sqrt{n}$ divided by an independent $\chi^2$ root is below $k\sqrt{n}$, and that quantity has a name: it is a non-central t. So

$$k = \frac{t_{1-\alpha}\big(n-1,\ z_p\sqrt{n}\big)}{\sqrt{n}}$$

exactly, with no approximation anywhere. At ten observations it is 2.911, and that digit is not a table lookup — it is the same non-central t that [power without simulating it] is read off(/essays/how-many-subjects/) is read off, evaluated at a different non-centrality.

The two numbers are worth seeing together, because the relationship between them is not the factor of two that a reader expects from one tail against two. At ten observations the one-sided factor is 2.911 and the two-sided is 3.382; at a hundred they are 1.927 and 2.233. The two-sided band is about 16% wider in both cases, not twice as wide, because the two-sided promise is not two one-sided promises — it is a promise about the total content, and content lost at one end can be made up at the other.

The factor at the sizes people work at #

The confidence is a second knob, and it is the one nobody moves, so it is worth seeing what leaving it where it was found costs.

Two readings, and the second is the one that decides what to do.

The confidence is cheap at large n and ruinous at small. Going from 90% to 99% confidence costs 89% extra width at five observations and 5% at three hundred. That is the same shape as everything else in this collection that depends on an estimated variance: the cost of a promise about a procedure is the cost of surviving the worst s the procedure is likely to draw, and at five observations the worst likely s is very small indeed.

The sample size is what buys the band, and nothing else is. There is no analysis that recovers at ten observations what thirty observations give, because the width is not a modelling choice; it is the spread of the sample standard deviation, and the only thing that narrows that spread is more data. An author who finds 3.382 too wide to report is not looking at a conservative method. They are looking at what ten observations are worth.

The other correction, and why it is not this one #

The t distribution exists because the standard deviation is estimated rather than known, which is word for word the diagnosis above. It is a natural move to reach for it here, and it fixes a different band.

The t correction is about the mean. The quantity x̄ − μ has standard deviation $\sigma/\sqrt{n}$; replacing σ by s makes the standardised version a t rather than a normal, and widening by the t quantile restores the interval’s coverage of μ exactly. It is a complete repair, and its completeness is why it is taught: at eight observations a z interval covers 90.7% and a t interval covers 94.9%, and there is nothing left over.

The band on this page is about the population, and the difference is not a matter of degree. Covering μ is covering one point; holding 95% of the population is holding a region whose two edges are both wrong, in a distribution whose density at those edges is what decides the cost. The t quantile is the right width for the first and is not wide enough for the second, and no amount of t will make it so, because t is answering the question about the mean.

The two are confused often enough to be worth a name. The t interval is about where a parameter is; the tolerance interval is about where the data are. A laboratory reporting the range within which 95% of its measurements fall, a manufacturer stating a specification, an assay quoting a reference range — all three are asking the second question, and all three are routinely answered with the first.

What ten observations actually look like #

The factor at ten observations is 3.382 and it feels excessive until the sample is drawn.

The spread across those panels is the spread of s. On ten observations the sample standard deviation has a coefficient of variation of about 23%, so one sample in twenty produces an s that is more than a third below σ, and the band built on it is more than a third too narrow. The factor 3.382 is what it takes for that sample’s band still to hold 95%, and there is no cheaper way to make a promise that has to hold on nineteen samples out of twenty rather than on the average one.

This is the same reasoning that prices a prior or an interval’s coverage: the quantity being promised is a property of a procedure run many times, so it has to be measured across many runs, and a number computed on the average run is answering a question nobody asked.

Where the band is actually wanted #

The tolerance interval is not an exotic object. It is what three common reporting tasks are asking for, and all three are routinely answered with one of the other two bands.

A reference range. A clinical laboratory reports the interval containing the central 95% of values in a healthy population, and a result outside it is flagged. That is a statement about where the data are, made from a finite reference sample, and it is exactly a two-sided tolerance interval. Computed as x̄ ± 2s from a reference sample of forty, it holds 94.48% of the healthy population on average and less than 95% on exactly half of reference sets — so the flag rate is above its design value on one laboratory’s reference set in two, and nothing about the calculation says which.

A specification limit. A manufacturer states a bound that the product will meet. The population is on one side of it, the question is one-sided, and the exact non-central t factor above is the answer. Using z there — 1.645 for a 95% limit — puts the bound where the factor that is actually needed is 77% further out at ten observations and 17% at a hundred, in the direction that ships product outside specification.

A process capability index. C_p is a specification width divided by 6s, reported to two decimals as though s were σ. Every criticism above applies, and the sampling distribution is wider than the two decimals suggest: a process whose true capability is exactly the conventional 1.33 threshold, measured on twenty-five parts, reads between 1.08 and 1.75 ninety per cent of the time, and reads below its own threshold 46.2% of the time. Two hundred parts narrows that range only to 1.23–1.45. The index is not wrong; it is reported without the second number that would say how much of it is the sample.

The common thread is the one these essays keep arriving at from different directions. A summary computed from a sample is being read as though it were the population’s, and the summary does not carry the uncertainty that would say so. The repair is never a different summary; it is a wider band whose width is derived from how wrong the summary can be.

What is claimed here, and what is not #

Three statements, and the third is the one the page exists to refuse.

The content of x̄ ± k s is a random variable, and the distribution drawn is its distribution. It is counted across twenty thousand normal samples at each sample size. Two things have to hold at every setting for the counting to mean anything: the mean content must be below the population band’s 95.45%, and the share of short samples must be a fraction strictly between nothing and everything. Those are the two ways a miscounted answer would arrive as a number that could not be wrong.

The tolerance factor is computed twice, by arithmetic that shares nothing. Howe’s approximation is what the published tables are printed from. The second route counts it: for each sample, the k that would make that sample’s band hold exactly 95% is found by bisection, and the factor is the 95th percentile of those. At ten observations the two give 3.382 and 3.407, agreeing to eight parts in a thousand across forty thousand samples.

And the claim that does not survive is x̄ ± 2s read as a 95% band. The standard every coverage claim here is held to is that a 95% band holds 95%, and at ten observations this one holds 91.1%. A content that came back at 95.45% would mean either that the estimation had stopped being sampled or that the content was being computed against the sample rather than against the population — and both would make every other number on this page agree with its label, which is the shape of a measurement that is not measuring.

Still open: what the factor costs when the population is not normal #

Everything above is normal theory twice over. The content is an integral of a normal density, and the factor is derived from the fact that $(n - 1)s^2/\sigma^2$ is a $\chi^2$ independent of x̄ — a property only the normal has.

A tolerance factor computed under normality and applied to something else is not conservative in any direction anyone can state in advance. Heavier tails put more of the population outside a band of any given width, so the factor is too small; a shorter-tailed population makes it too large. Neither failure announces itself, because the band is drawn from the sample and the sample is small enough that its shape cannot be checked — which is precisely the condition under which a tolerance interval is wanted.

There is a way out, and it costs observations rather than assumptions. The interval between the smallest and the largest of a sample holds a share of the population whose distribution does not depend on the population at all, so a band can be built with no shape assumption whatsoever. What that costs, and why it is ninety-three observations rather than ten, is the question this leaves.