Normal Distribution
nothing is called 95% until it has been counted
Concept

Order statistic — where it appears

A sample's readings put in order, indexed by their rank. The largest is the object extreme-value theory is about, and its exact distribution is the parent's distribution function raised to the sample size.

Named by 11 essays across 5 fields — each of them below, with the objects they name alongside it.

The coverage is exact and it is not the nominal rate. ⌈(m+1)(1−α)⌉/(m+1) against m, the number of calibration points, at α = 0.05. It is a closed form and needs no data. It never falls below 95.0% and never reaches 1−α+1/(m+1), the two bounds the rank argument gives. It equals 95.0% exactly at 10 of the 182 sizes drawn — the sizes where (m+1)α is a whole number, which are 20 apart — and sits above it everywhere else, worst at 38 points where it is 97.4359%, or 2.4359% of coverage nobody asked for. Below 19 points there is no such order statistic and the interval is the whole line, which is where the curve starts.

Coverage from exchangeability alone

A conformal interval's coverage is a fact about the ranks of m+1 numbers, so it can be enumerated before any data arrive — all 40,320 orderings of eight values, agreeing with the closed form to machine precision. What that exactness delivers is not 95%.

conformal · Exchangeability Two 95% bands for a quantile plot of 40 points. The outer band is left by 5% of genuinely normal samples — which is what a reader is using a band for. The inner one holds each point separately at 95%, which is what software draws, and 45.0% of genuinely normal samples step outside it. The outer is the inner widened by a factor of 1.502.

The band the eye was standing in for

The confidence band software draws on a quantile plot holds each point at 95%, and a genuinely normal sample of forty has forty chances to leave it — so 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it.

lineup · Qq One tail arrives; the other is still on its way at a million. The Kolmogorov distance between the exact law of a normalised maximum and its Gumbel limit, at six block sizes, for two parents that both have that same limit. Both are closed form: the exact law of a maximum is F(x)^n and no simulation is involved. The exponential parent's distance falls from 0.0280 to 2.707e-7 — a factor of a hundred thousand, which is exactly one over n. The normal parent's falls from 0.0522 only to 0.0091, a factor of 5.74, because its rate is one over log n. At a million readings a block the two differ by a factor of 33556.3.

The maximum converges slowly

The rate at which a normalised maximum reaches its limit law is computable rather than simulable, because the exact law of a maximum is always available. For a normal parent the distance falls like one over the logarithm of the block and is still 0.0091 at a million readings; for an exponential parent, with the same limit, it is 2.707×10⁻⁷.

extreme · Extremes The split decides the width. The width of the interval against the share of 200 observations spent on fitting rather than on calibrating, over 3000 draws. Spending more on the fit shrinks the residuals; spending more on calibration builds the interval at a less extreme order statistic. The two meet at 0.5, where the width is 4.0416 against 4.1603 at 0.1 and 4.3820 at 0.9. Full conformal, which spends the same 200 points on both jobs, is 3.9865 — so the whole cost of splitting is 1.38%.

What the split costs

Splitting a sample between fitting and calibrating looks like a trade against the guarantee, and it is not: coverage moves 0.63 points across nine splits and every reading sits on its own promise. The whole cost is 1.38% of width — and at sixty observations the width falls, rises and falls again.

conformal · Exchangeability What a longer list actually changes. How often the fifteen candidates choose different tuning parameters, at a true null where every one of them contains the truth, so a disagreement is manufactured rather than discovered. The order's list is an interval of integers, and thinning it moves the rate smoothly from 0% at two values to 39% at thirteen. The window's is not an interval — it runs 0, 1, 2, 4, 8, 12, 20, 30 — so a thinned window list jumps depending on whether it happens to keep the width the criterion wants, between 0% and 42% with no order to it. So "the same length" was never quite the same thing for the two rules, and it is a smaller effect than the field it was invoked to explain.

A list is not a rule

How often fifteen candidates disagree about a tuning parameter runs from nothing at two values on the list to two draws in five at thirteen. What the disagreement costs does not move at all.

lists · Order selection One promise, kept on average and inside neither group. What each calibration scheme covers inside each of two equally common groups whose noise scales are 1 and 3, over 6000 draws with 200 calibration points. One interval for everybody covers 100.00% of the quiet group and 90.66% of the noisy one, averaging to 95.28% — and the closed form for that population says 99.9999% and 90.0001% at a half-width of 4.9346, from two normal cdfs and no simulation. Dividing by an estimated per-group scale gives 95.29% and 95.48%; calibrating separately inside each group gives 95.49% and 96.01% against a closed-form expectation of 95.4645%. Only the last of those is a guarantee rather than a repair, because the rank argument runs inside each group.

Marginal is not conditional

One exactly valid interval covers 100.00% of a quiet group and 90.66% of a noisy one, and the floor is arithmetic rather than a measurement — a group of share π is guaranteed only 1 − α/π, which is zero when the group is as rare as the miss rate.

conformal · Exchangeability How many observations the smallest and largest of them need. The interval between the extremes of n draws holds at least 95% of the population with probability 1 - n p^(n-1) + (n-1) p^n, whatever the population is. Reaching 95% confidence takes 93 observations.

Ninety-three observations, and nothing assumed

The interval between the smallest and largest of a sample holds a share of the population whose distribution does not depend on the population — Beta(n − 1, 2), for anything continuous. Buying the 95/95 that normality buys at ten observations costs 93 of them, and that number is the exchange rate between an assumption and data.

estimated · Bands The record stops here, and the curve does not. The level exceeded once in T blocks, against T, for a normal parent at 365 readings a block. The truth is closed form — the block maximum's own distribution function is Φ(x) raised to the 365, so the T-block level is Φ⁻¹ of the 365-th root of (1 − 1/T), with nothing fitted in it — and the fitted mean over 800 records of 50 blocks sits on top of it, 4.0186 against 4.0330 at 100 blocks. What moves is not the level but its error, which grows from 0.0956 at 10 blocks to 0.6383 at a thousand while the level itself moves only from 3.4421 to 4.5454. The rule marks the largest reading an average record contains, 4.0062: everything to the right of where it crosses is read from a fit rather than from data.

A level with no data in it

The largest of fifty block maxima is a 51-block event by its own plotting position, so a hundred-block level is read 1.96 times past the longest event the record contains — and it lands above the largest reading on 52.4% of records. The estimate stays nearly unbiased out there; what grows is its error, sixfold from ten blocks to a thousand.

extreme · Extremes How many observations a band from a sample's extremes needs to let at most r of the next 100 outside, 95% of the time, for any population. From the sample's smallest and largest: none of the next 100 outside needs 3,850 observations, at most one 637, at most five 107 and at most ten 50. From the second smallest and second largest, 7,750, 1,204, 187 and 85.

A range allowed a few misses

Without assuming a shape, the band a sample can offer is its own smallest and largest values, and the number of future draws falling outside it is beta-binomial for every continuous population. A promise that at most one of the next hundred lands outside, with 95% probability, then needs 637 observations; none of the hundred, 3,850; at most five, 107. The normal-theory band keeps the one-in-a-hundred promise from ten. On normal data, at the sizes the range needs, it is only 6% wider than the normal band at the same size — the price of dropping the assumption is paid in observations, not in width — and its misses cluster far less than a band from ten: a second follows a first 19.7% of the time, against 45.4%.

estimated · Bands How many starting observations a range needs to flag at most r of the next 100, fixed at the start or learning as the lot is used. 95% of the time, for any continuous population. At most 0: fixed 3,850, learning 3,850; At most 1: fixed 637, learning 512; At most 2: fixed 300, learning 197; At most 3: fixed 190, learning 101; At most 5: fixed 107, learning 36; At most 7: fixed 74, learning 15; At most 10: fixed 50, learning 4. With no flag allowed the two are the same, because a lot with no flag never moves the learning range.

A range that grows with the lot

A band set from the range of everything seen so far flags a unit only when it sets a new record, and for any continuous population those flags are independent, with the k-th unit flagged with probability 2/(n + k). That makes a few-misses promise far cheaper than the fixed range's — at most five of the next hundred from 36 starting observations rather than 107 — and exactly as expensive when no miss is allowed, because a lot with no flag never moves the band. The saving is bought by believing every flag. After a two-standard-deviation step in the mean, the fixed range flags 25.03 of the eighty units that follow and the learning range 3.62; by the last twenty units it sees a unit outside in 25.5% of lots, against 18.4% with no step at all.

estimated · Bands How many units a range needs to let at most r of the next hundred outside, fixed, learning or forgetting. 95% of the time, for any continuous population. At most 1: fixed 637, window 564, learning 512; At most 2: fixed 300, window 236, learning 197; At most 5: fixed 107, window 64, learning 36; At most 10: fixed 50, window 27, learning 4. The window counts are from 20,000 lots of uniform draws; the other two are exact.

A range that forgets

A band set from the range of the last sixty-four units flags each unit of a stable lot with probability exactly 2/65, for ever, and its flags arrive as if independent — so a promise of at most five outside in the next hundred costs sixty-four units, between the learning range's thirty-six and the fixed range's hundred and seven. What it was supposed to buy, a flag that lasts as long as a step and then stops, it cannot give: it absorbs a two-sigma step as fast as the learning range, 1.89 flags in ten units against 1.93, because it learns from what it flags. Refusing to learn from a flag makes it a fixed range that costs 141.

estimated · Bands

Named alongside it

The objects these essays reach for when they reach for this one.

Closed formDistribution-freeExchangeabilityCoverageTolerance intervalCalibration setConformal predictionOvercoverageQuality controlSample sizeBlock maximaBlock size

All concepts