Normal Distribution
nothing is called 95% until it has been counted
Concept

Distribution-free — where it appears

Holding for every distribution in a very large class rather than for a named one. Such a guarantee is bought by making the claim about ranks or about the design instead of about the data, and what it costs is a statement weaker than the one a correct model would have given.

Named by 9 essays across 2 fields — each of them below, with the objects they name alongside it.

The coverage is exact and it is not the nominal rate. ⌈(m+1)(1−α)⌉/(m+1) against m, the number of calibration points, at α = 0.05. It is a closed form and needs no data. It never falls below 95.0% and never reaches 1−α+1/(m+1), the two bounds the rank argument gives. It equals 95.0% exactly at 10 of the 182 sizes drawn — the sizes where (m+1)α is a whole number, which are 20 apart — and sits above it everywhere else, worst at 38 points where it is 97.4359%, or 2.4359% of coverage nobody asked for. Below 19 points there is no such order statistic and the interval is the whole line, which is where the curve starts.

Coverage from exchangeability alone

A conformal interval's coverage is a fact about the ranks of m+1 numbers, so it can be enumerated before any data arrive — all 40,320 orderings of eight values, agreeing with the closed form to machine precision. What that exactness delivers is not 95%.

conformal · Exchangeability One promise, kept on average and inside neither group. What each calibration scheme covers inside each of two equally common groups whose noise scales are 1 and 3, over 6000 draws with 200 calibration points. One interval for everybody covers 100.00% of the quiet group and 90.66% of the noisy one, averaging to 95.28% — and the closed form for that population says 99.9999% and 90.0001% at a half-width of 4.9346, from two normal cdfs and no simulation. Dividing by an estimated per-group scale gives 95.29% and 95.48%; calibrating separately inside each group gives 95.49% and 96.01% against a closed-form expectation of 95.4645%. Only the last of those is a guarantee rather than a repair, because the rank argument runs inside each group.

Marginal is not conditional

One exactly valid interval covers 100.00% of a quiet group and 90.66% of a noisy one, and the floor is arithmetic rather than a measurement — a group of share π is guaranteed only 1 − α/π, which is zero when the group is as rare as the miss rate.

conformal · Exchangeability How many observations the smallest and largest of them need. The interval between the extremes of n draws holds at least 95% of the population with probability 1 - n p^(n-1) + (n-1) p^n, whatever the population is. Reaching 95% confidence takes 93 observations.

Ninety-three observations, and nothing assumed

The interval between the smallest and largest of a sample holds a share of the population whose distribution does not depend on the population — Beta(n − 1, 2), for anything continuous. Buying the 95/95 that normality buys at ten observations costs 93 of them, and that number is the exchange rate between an assumption and data.

estimated · Bands Six scores, one coverage. What each nonconformity score's interval covers, over 2000 draws with 200 calibration points, against the 95.0249% the rank argument promises. The column runs from 94.30% to 94.90%, a spread of 0.60% against a standard error of a difference of 0.69% — one number, six times. That includes a score aimed five units off the fit and a score that never reads the response at all, because the rank argument does not read the score either: it needs the scores exchangeable and nothing else. Every decision a modeller makes has to show up somewhere else, and the next two readings are where.

The score is the modelling

Six nonconformity scores on the same draws cover within 0.60 points of each other, against a standard error of a difference of 0.69 — one number six times. Their widths run over a factor of 2.361 and their adaptivity over a factor of 8.377.

conformal · Exchangeability What a scale that grows across the sample costs. What the interval covers when the noise scale grows across the sample, against how far the departure has gone, over 1500 draws at each setting. The coverage runs from 94.47% at no departure to 83.93% at the end of the sweep, a loss of 11.07%. The rank argument needs the 200 calibration scores and the test score to be exchangeable, and this is one of the three ways that fails. A test built for it reaches 80% power at 4.054, where the coverage is 85.13% — so 9.87% of the loss is inside the region such a test would have missed.

When the order matters

Three ways of breaking exchangeability cost 4.93, 11.07 and 1.07 points of coverage, and the ordering by cost is the reverse of the ordering by how soon a test would have caught them. The departure practitioners check for is the cheapest one.

conformal · Exchangeability How many observations a band from a sample's extremes needs to let at most r of the next 100 outside, 95% of the time, for any population. From the sample's smallest and largest: none of the next 100 outside needs 3,850 observations, at most one 637, at most five 107 and at most ten 50. From the second smallest and second largest, 7,750, 1,204, 187 and 85.

A range allowed a few misses

Without assuming a shape, the band a sample can offer is its own smallest and largest values, and the number of future draws falling outside it is beta-binomial for every continuous population. A promise that at most one of the next hundred lands outside, with 95% probability, then needs 637 observations; none of the hundred, 3,850; at most five, 107. The normal-theory band keeps the one-in-a-hundred promise from ten. On normal data, at the sizes the range needs, it is only 6% wider than the normal band at the same size — the price of dropping the assumption is paid in observations, not in width — and its misses cluster far less than a band from ten: a second follows a first 19.7% of the time, against 45.4%.

estimated · Bands Three detectors for one departure, all at 5%. How often each of three checks on the calibration scores fires, against the size of the drift, with every critical value simulated under no drift so that all three sit at 5.0% exactly. The incumbent — a rank comparison of the first half of the scores against the second — reaches four-in-five power at a growth factor of 4.31. Reading each score's rank against its position reaches it at 2.65, and the largest running departure of the scores from their mean at 2.12. The ordering of the three is the ordering by how much of the sample's arrangement each one uses.

A detector built for the ordering

The best of three checks for a drifting scale fires at half the growth factor the standard one needs — 2.12 against 4.31 — and still leaves 6.50 points of coverage gone before it does, against 0.51 for serial correlation. The reversal was not a property of the test.

conformal · Exchangeability How many starting observations a range needs to flag at most r of the next 100, fixed at the start or learning as the lot is used. 95% of the time, for any continuous population. At most 0: fixed 3,850, learning 3,850; At most 1: fixed 637, learning 512; At most 2: fixed 300, learning 197; At most 3: fixed 190, learning 101; At most 5: fixed 107, learning 36; At most 7: fixed 74, learning 15; At most 10: fixed 50, learning 4. With no flag allowed the two are the same, because a lot with no flag never moves the learning range.

A range that grows with the lot

A band set from the range of everything seen so far flags a unit only when it sets a new record, and for any continuous population those flags are independent, with the k-th unit flagged with probability 2/(n + k). That makes a few-misses promise far cheaper than the fixed range's — at most five of the next hundred from 36 starting observations rather than 107 — and exactly as expensive when no miss is allowed, because a lot with no flag never moves the band. The saving is bought by believing every flag. After a two-standard-deviation step in the mean, the fixed range flags 25.03 of the eighty units that follow and the learning range 3.62; by the last twenty units it sees a unit outside in 25.5% of lots, against 18.4% with no step at all.

estimated · Bands How many units a range needs to let at most r of the next hundred outside, fixed, learning or forgetting. 95% of the time, for any continuous population. At most 1: fixed 637, window 564, learning 512; At most 2: fixed 300, window 236, learning 197; At most 5: fixed 107, window 64, learning 36; At most 10: fixed 50, window 27, learning 4. The window counts are from 20,000 lots of uniform draws; the other two are exact.

A range that forgets

A band set from the range of the last sixty-four units flags each unit of a stable lot with probability exactly 2/65, for ever, and its flags arrive as if independent — so a promise of at most five outside in the next hundred costs sixty-four units, between the learning range's thirty-six and the fixed range's hundred and seven. What it was supposed to buy, a flag that lasts as long as a step and then stops, it cannot give: it absorbs a two-sigma step as fast as the learning range, 1.89 flags in ten units against 1.93, because it learns from what it flags. Refusing to learn from a flag makes it a fixed range that costs 141.

estimated · Bands

Named alongside it

The objects these essays reach for when they reach for this one.

ExchangeabilityOrder statisticCalibration setConformal predictionCoverageNonconformity scoreMarginal coverageTolerance intervalHeteroskedasticityQuality controlSample sizeAutocorrelation

All concepts