Ninety-three observations, and nothing assumed

Ten observations and an assumption of normality buy a band holding 95% of the population with 95% confidence: x̄ ± 3.382 s. The assumption is doing work that is easy to overlook, because it is not visible anywhere in the arithmetic once the factor has been looked up.

Here is what the same band costs without it.

Ninety-three observations, and the band is then the smallest and largest values in the sample. No parameters are estimated, no shape is assumed, and nothing is computed at all: the interval is read off the sorted data.

Why the population drops out #

Let the population have any continuous distribution function F. Apply F to each observation. The transformed values U_i = F(X_i) are uniform on (0, 1) — that is the probability integral transform, and it holds for every continuous F without exception.

F is increasing, so it preserves order: the smallest X becomes the smallest U, the largest X the largest U. And the content of the interval (X₍₁₎, X_(n)) — the share of the population inside it — is

$$C \;=\; F(X_{(n)}) - F(X_{(1)}) \;=\; U_{(n)} - U_{(1)},$$

which is the gap between the extremes of n uniforms. Whatever the population was, the content is that gap, and the gap’s distribution is Beta(n − 1, 2). So

$$P(C \ge p) \;=\; 1 - n\,p^{\,n-1} + (n-1)\,p^{\,n},$$

with no reference to F anywhere. The same argument prices any pair of order statistics: the content of the interval from the r-th smallest to the s-th largest is Beta(n + 1 − r − s, r + s).

The claim deserves more than a derivation, because it is the one claim in this collection a reader has no reason to believe on sight. Every other band here moves when the shape does — the empirical rule’s three numbers change under a t, the t interval’s own coverage moves under skew. This one does not move at all.

At twenty observations the closed form gives 60.8% for holding nine tenths of the population, and the four counts give 60.4%, 61.1%, 61.1% and 61.8% — agreement to within a point, on distributions that share nothing but continuity. One of them has no variance to speak of, one is one-sided, one has hard edges and one is a bell.

Two of those four are the same sample #

Look again at the exponential and the uniform: 61.1% and 61.1%, identical to every digit printed, and identical in the underlying doubles as well.

That is not a coincidence and it is not a mistake. Both draws consume exactly one uniform and transform it monotonically — the uniform draw u is returned as it stands, and the exponential draw is −log(1 − u). So the two samples are the same u values seen through two different increasing functions, and the theorem says the content is invariant to exactly that. The two rows are not independent evidence for the claim; they are the claim, executed.

The normal and the t rows are independent evidence, because the polar method that produces them consumes a variable number of uniforms per draw and rejects some of them, so those samples share no random numbers with the first two. Their agreement is a count rather than an identity, and it is the one that carries the weight.

This distinction is worth making because a table of four agreeing rows is the kind of evidence that looks stronger than it is. Two of these rows agree by construction and two agree by measurement, and reading all four alike would be reading a sample size of four where there is one of two.

The exchange rate #

The price is paid in observations, and the table is short.

Three things are legible in it.

The cost is roughly inverse in the tail probability. Holding 95% takes about twice what holding 90% takes, and holding 99% about twice what holding 98% takes. That is the right shape: the band’s edges have to be past the population’s own 2.5% points, and the chance that a sample of n contains an observation out there is what the arithmetic is really about.

Trimming is expensive. Using the second-smallest and second-largest — which is what an analyst does on reflex, because the extremes look like outliers — costs about 65% more observations at every row. The extremes are the whole of the method, and discarding them discards the band.

Against the normal-theory factor, the exchange rate is about nine to one at 95/95. Ten observations with normality assumed, ninety-three without. That is what the assumption is worth, in the only currency the question has.

One side is much cheaper than two #

Ask for a bound rather than a band and the arithmetic collapses to one line.

The largest observation exceeds the population’s p-quantile unless every observation fell below it, and that has probability pⁿ. So a one-sided distribution-free bound holding p of the population with confidence 1 − α needs

$$n \;=\; \frac{\log \alpha}{\log p},$$

which for 95/95 is 58.4 — fifty-nine observations, against ninety-three for the two-sided band. Holding 90% takes twenty-nine, and holding 99% takes two hundred and ninety-nine.

The saving is not the factor of two a reader expects from one tail against two, and the reason is the same one that made the two-sided normal factor only 16% wider than the one-sided one rather than twice as wide. A two-sided band fails if either end is inside the population’s tail, and the two ends are strongly negatively related across samples — a sample that reached far to the left has a large spread and probably reached far to the right as well. So the two failures overlap, and the cost of insuring against both is less than twice the cost of insuring against one.

That expression is also the exact relative of a formula used elsewhere here. Put p = 1 − x and α = 0.05, and since log(1 − x) ≈ −x the expression becomes nx = −log 0.05 = 2.996 — the rule of thumb that zero events in n trials is consistent with a rate up to 3/n, with the 3 revealed as a logarithm rounded. Both are the same statement: n observations all on one side of something bound how much mass can be on the other.

What the assumption costs when it is wrong #

Before pricing the assumption it is worth measuring what breaking it does, because the obvious guess is wrong twice.

The guess is that a heavier tail makes the band too narrow. A uniform population makes it conservative — the band holds 95% on 99.5% of samples at twenty observations, and essentially always at a hundred, because a distribution with hard edges has nothing out where the band’s edges are. And the population that breaks it worst is not the heaviest-tailed one: a Laplace, which has a peaked centre and an exponential tail, keeps the promise on 78.4% of samples at twenty, against the t on four degrees of freedom’s 82.8%.

The reading that matters is what happens as the sample grows.

The failure gets worse with more data. At ten observations the Laplace band keeps its promise four times in five; at a hundred, less than two times in three. That is the opposite of every other approximation error in this collection, and the mechanism is specific: the factor’s excess over 1.96 is an allowance for a badly estimated s, and that allowance shrinks as n grows. The error in the shape does not shrink at all. So the one term that was accidentally covering for the wrong shape is withdrawn, and what is left is a band built at very nearly 1.96 standard deviations around a population for which 1.96 is not the right multiple.

A guarantee that decays with sample size is not a small-sample caveat. It is a method whose stated property is false in the limit, and the only thing keeping it approximately true at small n is a correction that exists for an unrelated reason.

And the assumption cannot be checked at the size where it matters #

The obvious response is to test for normality, use the factor when the test comes back clean and the extremes when it does not. That does not work, and the reason is arithmetic rather than caution.

At the sample sizes where the normal factor is attractive — ten, twenty, thirty observations — a goodness-of-fit test has very little power against the departures in the table above. A Laplace and a normal with the same variance differ in the shape of the peak and in the weight of a tail that a sample of twenty barely reaches; twenty draws contain roughly one observation beyond two standard deviations under either, which is not enough to distinguish them.

At the sample sizes where such a test does have power — several hundred — the distribution-free band is already affordable, and the question has stopped being interesting. Worse, the table says the factor’s failure is largest at those sizes, so the regime where the test can detect the problem is the regime where the problem is most severe: the test becomes reliable at exactly the point where it has the least left to protect.

So the power is lowest where it is needed and highest where it is not. That is not a fixable defect of any particular test; it is the shape of the problem. The same trap catches the pre-test for equal variances and every other two-stage procedure that chooses a method from the data and then reports the chosen method’s error rate as though the choice had not happened.

What ninety-three observations actually give #

At n = 93 the extremes hold at least 95% of the population with probability 95.00%, and they hold 97.87% on average — the mean content of the extremes’ interval is (n − 1)/(n + 1) exactly.

That average is worth pausing on. The band is not a tight 95% band with a confidence attached; it is a band that usually holds rather more than 95%, and the confidence statement is about its worst behaviour rather than its typical one. A reader who wants a band holding about 95% of the population from ninety-three observations should use something narrower; a reader who wants a band that provably holds 95% has the right object, and the gap between those two wishes is the whole 2.87 percentage points.

That caption is the honest summary of the trade. At ninety-three observations the band a reader reaches for is short of its label on nearly half of samples and depends on a shape nobody checked. The normal factor at ninety-three is 2.245, which is a perfectly good band and is good conditional on that shape — and the table above says what the condition is worth when it fails. The extremes of the same sample carry a guarantee that does not have a condition attached to it at all.

The content is distribution-free and the width is not #

Nothing above says the band is narrow, and it is worth separating the two properties, because the guarantee is about one of them and a reader’s disappointment will be about the other.

The content of the extremes’ interval is distribution-free: Beta(n − 1, 2), whatever the population. The width of it is a property of the population and nothing else, and it is the widest thing a sample contains. At ninety-three normal observations the extremes span 4.97 standard deviations on average, against 4.49 for the normal-theory 95/95 band at the same sample size — so the assumption-free band is about 11% wider, which is the visible price on top of the sample size.

The invisible price is that it is much less stable. The extremes’ width has a coefficient of variation of 12.2% across samples, against 7.3% for the normal band, and its middle 90% runs from 4.05 to 6.03 standard deviations. Two laboratories following the same procedure on the same population with ninety-three observations each can report bands differing by half again in width, both correct, both carrying the same guarantee.

That is the structural fact behind it. The minimum and maximum are the two most variable statistics a sample has — every other order statistic has neighbours on both sides to pin it down — so a method built on them inherits their variability in full. The guarantee survives because the content is pinned by the theorem while the width wanders, and those are genuinely different quantities: the band can be much wider than usual and still hold exactly the share the theorem promises.

The same argument, pointed at a quantile #

The order statistics that give a distribution-free band also give a distribution-free interval for a quantile, and it is the same counting argument run the other way.

The population median lies below x_® only if fewer than r of the n observations fell below it, and the number that fell below it is Binomial(n, ½). So the interval from x_® to x_(n+1−r) covers the median with a probability that is a binomial tail sum, and nothing else enters.

At twenty observations the sixth and fifteenth values cover the median 95.9% of the time. The seventh and fourteenth cover it 88.5%, and the fifth and sixteenth 98.8%. There is no pair giving exactly 95%, and that is not an approximation failing — it is the discreteness that makes every exact interval for a discrete quantity land on a lattice of achievable levels, the same effect that makes an interval for a proportion oscillate rather than improve smoothly.

The two constructions look alike and answer opposite questions, which is worth keeping straight. A tolerance interval fixes the share of the population and asks what the confidence is. A quantile interval fixes the quantile and asks where it is. Both are built from order statistics, both are free of the population’s shape, and the first is about the data while the second is about a parameter — the same division that separates the three normal-theory bands.

What is claimed here, and what is not #

Two routes, as everywhere here, and they share no arithmetic.

The closed form evaluates 1 − n p^(n−1) + (n − 1)p^n and inverts it for the smallest n. It touches no random numbers.

The count draws eight thousand samples under each of four populations, sorts each, applies that population’s own distribution function to the extremes and subtracts. It touches no algebra beyond the subtraction.

The claim is that the widest disagreement between any population’s count and the closed form is under two percentage points, and it is made across the four at once rather than one at a time — because the claim is that the shape does not enter, and a claim about a family cannot be established inside a single member of it. Four separate per-shape statements would all hold and would say nothing.

The reading that does not survive is a count under one population compared with a closed form derived under another. If the probability integral transform were applied wrongly — if the distribution function used on the exponential’s extremes were the normal’s, say — three of the four rows would move and the comparison across the four would fail. It is deliberately arranged so that getting one shape wrong is visible, rather than averaging into a number that still looks about right.

Still open: the band that is neither the extremes nor the formula #

The two bands on this page sit at the ends of a range. One assumes a shape and needs ten observations; the other assumes nothing and needs ninety-three. Almost every real situation is between them, and nothing here says what to do there.

Still open: what symmetry alone is worth in observations #

The intermediate objects exist. A band can assume symmetry without assuming normality, and symmetry is a much weaker assumption that is often defensible on subject-matter grounds. It can assume unimodality, which buys a sharper version of Chebyshev’s bound — $1 - 4/(9k^2)$ rather than $1 - 1/k^2$. It can assume a parametric family with an extra shape parameter and pay for that parameter in width rather than in a leap of faith.

What none of those has here is a price. The exchange rate between normality and ninety-three observations is measured on this page; the exchange rate between symmetry and some number of observations is not, and it is the one a practitioner would actually spend. Whether assuming symmetry alone brings the ninety-three down to thirty or to eighty decides whether the assumption is worth making, and it is a question with a countable answer.