Normal Distribution
nothing is called 95% until it has been counted
Concept

Exchangeability — where it appears

That the joint distribution of a set of observations is unchanged by reordering them. It is weaker than independence and is what most pooling arguments actually need. It is also what makes a permutation reference distribution valid without any distributional assumption at all.

Named by 20 essays across 7 fields — each of them below, with the objects they name alongside it.

The coverage is exact and it is not the nominal rate. ⌈(m+1)(1−α)⌉/(m+1) against m, the number of calibration points, at α = 0.05. It is a closed form and needs no data. It never falls below 95.0% and never reaches 1−α+1/(m+1), the two bounds the rank argument gives. It equals 95.0% exactly at 10 of the 182 sizes drawn — the sizes where (m+1)α is a whole number, which are 20 apart — and sits above it everywhere else, worst at 38 points where it is 97.4359%, or 2.4359% of coverage nobody asked for. Below 19 points there is no such order statistic and the interval is the whole line, which is where the curve starts.

Coverage from exchangeability alone

A conformal interval's coverage is a fact about the ranks of m+1 numbers, so it can be enumerated before any data arrive — all 40,320 orderings of eight values, agreeing with the closed form to machine precision. What that exactness delivers is not 95%.

conformal · Exchangeability The weight on the population, σ = 3. Each curve is one population spread τ. A group's estimate moves B = se²/(se² + τ²) of the way to the population mean, where se = σ/√n is what the group's own mean does not know. At τ = 1 a group of 9 observations sits halfway.

The weight that decides

B = se²/(se² + τ²) is not a compromise between two answers. It is exactly the posterior mean's weight, it agrees with a numerical integration to ten digits, and an argument that mentions no population at all arrives at almost the same estimator.

hierarchical · Shrinkage The split decides the width. The width of the interval against the share of 200 observations spent on fitting rather than on calibrating, over 3000 draws. Spending more on the fit shrinks the residuals; spending more on calibration builds the interval at a less extreme order statistic. The two meet at 0.5, where the width is 4.0416 against 4.1603 at 0.1 and 4.3820 at 0.9. Full conformal, which spends the same 200 points on both jobs, is 3.9865 — so the whole cost of splitting is 1.38%.

What the split costs

Splitting a sample between fitting and calibrating looks like a trade against the guarantee, and it is not: coverage moves 0.63 points across nine splits and every reading sits on its own promise. The whole cost is 1.38% of width — and at sixty observations the width falls, rises and falls again.

conformal · Exchangeability One promise, kept on average and inside neither group. What each calibration scheme covers inside each of two equally common groups whose noise scales are 1 and 3, over 6000 draws with 200 calibration points. One interval for everybody covers 100.00% of the quiet group and 90.66% of the noisy one, averaging to 95.28% — and the closed form for that population says 99.9999% and 90.0001% at a half-width of 4.9346, from two normal cdfs and no simulation. Dividing by an estimated per-group scale gives 95.29% and 95.48%; calibrating separately inside each group gives 95.49% and 96.01% against a closed-form expectation of 95.4645%. Only the last of those is a guarantee rather than a repair, because the rank argument runs inside each group.

Marginal is not conditional

One exactly valid interval covers 100.00% of a quiet group and 90.66% of a noisy one, and the floor is arithmetic rather than a measurement — a group of share π is guaranteed only 1 − α/π, which is zero when the group is as rare as the miss rate.

conformal · Exchangeability A thin enough set is not one set. Every admissible set of 14 units this table can enumerate, by how much of the assignment space it admits and how many pieces it falls into under single swaps. A walk is uniform on the piece it starts in and never leaves it. The pieces are not fragments: at 522 admissible assignments the set splits into 3 halves of exactly 520 each, and every assignment's complement is in the other half — no sequence of admissible single swaps takes an assignment to its own mirror image. Two-swap proposals reconnect four of the six disconnected sets here — the two they do not are the thinnest, where a two-unit move rarely lands anywhere admissible either — which makes a bigger proposal a correctness repair rather than the speed dial it was measured as.

The walk that cannot cross

A thin enough admissible set is not one set. It splits into an assignment and its mirror image, no sequence of admissible single swaps joins them, and the walk that samples it is uniform on half the reference distribution for ever.

dict · Randomisation One group 6 population widths from the rest. Squared error for each group under partial pooling, with each group's own mean beside it. Seven of the eight are estimated better by pooling. The eighth, which was never from the population, is estimated 6.1 times worse — 13.9 against 2.3.

When borrowing goes wrong

Partial pooling wins on the total and can lose badly on one group. Placed six population widths out, the group that was never from the population is estimated six times worse than by its own mean — and nothing in the output says so.

hierarchical · Pooling Six scores, one coverage. What each nonconformity score's interval covers, over 2000 draws with 200 calibration points, against the 95.0249% the rank argument promises. The column runs from 94.30% to 94.90%, a spread of 0.60% against a standard error of a difference of 0.69% — one number, six times. That includes a score aimed five units off the fit and a score that never reads the response at all, because the rank argument does not read the score either: it needs the scores exchangeable and nothing else. Every decision a modeller makes has to show up somewhere else, and the next two readings are where.

The score is the modelling

Six nonconformity scores on the same draws cover within 0.60 points of each other, against a standard error of a difference of 0.69 — one number six times. Their widths run over a factor of 2.361 and their adaptivity over a factor of 8.377.

conformal · Exchangeability Coverage across the proportion of the randomised interval with the arrangement of the successes as its coin, 20 trials. Between proportions of 0.2 and 0.8 the arrangement coin's coverage stays within 0.121% of 95%, where mid-p's strays by 2.76%. Its worst coverage is 92.60%, at a proportion of 0.140, near the ends of the range where the count leaves few arrangements; mid-p's is 92.94%. Averaged over the proportion, 95.63% against mid-p's 96.11% and Clopper–Pearson's 97.71%.

The coin that is already there

The randomised interval covers exactly 95% at every proportion because it adds a drawn coin to the count, and a drawn coin is why nobody reports it. A sequence of trials already holds one: given the count, the order the successes arrived in is equally likely to be any of its arrangements whatever the proportion is. Used as the coin, it keeps twenty trials' coverage within 0.12 points of 95% from proportions of 0.2 to 0.8, where mid-p strays by 2.8, and it gives the same interval to every analyst. It fails where the count has few arrangements — a count of none has one — and it holds only while the order is read as recorded: an analyst choosing among sixteen orders lifts a 5% test on sixteen trials to 7.68%.

discrete · Oscillation What a scale that grows across the sample costs. What the interval covers when the noise scale grows across the sample, against how far the departure has gone, over 1500 draws at each setting. The coverage runs from 94.47% at no departure to 83.93% at the end of the sweep, a loss of 11.07%. The rank argument needs the 200 calibration scores and the test score to be exchangeable, and this is one of the three ways that fails. A test built for it reaches 80% power at 4.054, where the coverage is 85.13% — so 9.87% of the loss is inside the region such a test would have missed.

When the order matters

Three ways of breaking exchangeability cost 4.93, 11.07 and 1.07 points of coverage, and the ordering by cost is the reverse of the ordering by how soon a test would have caught them. The departure practitioners check for is the cheapest one.

conformal · Exchangeability Coverage against the expected count of the interval whose coin is read from the arrival times. The randomised interval for a Poisson mean, with its coin read from where in the window the events fell, covers exactly 97.5% at every expected count between 0.025 and ln 40 = 3.689 and exactly 95% past it, when a count of zero is given the exact interval's [0, ln 40]; its worst is 95.00%. Given mid-p's [0, ln 20] instead, it falls to 92.50% just past 3.00. Mid-p itself reaches 91.66% at 3.03; the exact interval never falls below 95.62% and averages well above 95%.

The coin a clock supplies

A count observed over a window comes with the times its events arrived, and given the count those times are uniform whatever the rate is — so they are a coin with a continuum of faces at every count from one up. Used to randomise the interval for a Poisson mean, they leave exactly one count without a coin, the count of zero, and what that count is given decides everything: mid-p's limit of ln 20 leaves a hole at 92.5%, and the exact interval's ln 40 is the shortest limit that leaves none, after which the interval covers exactly 97.5% below an expected count of 3.69 and exactly 95% above it.

discrete · Oscillation Too small breaks it and too large does not. Coverage of the weighted interval against the factor the true likelihood ratio is multiplied by, at a test population 80.0% drawn from the noisier group and 200 calibration points. The exact weight is the factor of 1 and covers 95.70%. Overstating it costs nothing: 96.13% at sixteen times too large. Understating it costs, and costs steeply below about a half — 94.93% at half, 88.37% at an eighth and 67.90% at a thirtieth. The question this answers was whether a wrong weight degrades smoothly or falls off a cliff, and the answer is that it does neither symmetrically: the curve is smooth and one-sided.

The weight that has to be estimated

A likelihood ratio sixteen times too large costs 5.5% of interval width and no coverage at all; one a thirtieth of the right size covers 67.90%. The estimate from a batch of five unlabelled covariates covers 95.10% against an exact repair's 95.30%, and the binomial says why.

conformal · Exchangeability How many observations a band from a sample's extremes needs to let at most r of the next 100 outside, 95% of the time, for any population. From the sample's smallest and largest: none of the next 100 outside needs 3,850 observations, at most one 637, at most five 107 and at most ten 50. From the second smallest and second largest, 7,750, 1,204, 187 and 85.

A range allowed a few misses

Without assuming a shape, the band a sample can offer is its own smallest and largest values, and the number of future draws falling outside it is beta-binomial for every continuous population. A promise that at most one of the next hundred lands outside, with 95% probability, then needs 637 observations; none of the hundred, 3,850; at most five, 107. The normal-theory band keeps the one-in-a-hundred promise from ten. On normal data, at the sizes the range needs, it is only 6% wider than the normal band at the same size — the price of dropping the assumption is paid in observations, not in width — and its misses cluster far less than a band from ten: a second follows a first 19.7% of the time, against 45.4%.

estimated · Bands Coverage of an interval for a ratio of two rates against the expected total. The unconditional coverage of three intervals for the ratio of two Poisson rates, each built conditionally on the total and each reporting the whole of (0, ∞) when the total is zero, at the conditional proportion 0.5, against the expected total count. With the arrival times in both windows as the coin the interval covers exactly 95% plus a twentieth of e raised to minus the expected total: 96.84% at an expected total of one, 95.25% at three, 95.00% from about ten. The exact conditional interval covers 100.00%, 99.79%, 97.82% and 96.55% at forty; mid-p 99.11% at three and 95.89% at ten.

Two windows and a ratio

Two counts from two windows give an interval for the ratio of their rates through the conditional binomial: given the total, the first window's count is binomial with a proportion that depends only on the ratio. The arrival times in both windows supply that binomial with a continuous coin at every total from one up, and the interval then covers exactly 95% plus a twentieth of the chance of no events at all. A total of zero can only report the whole of (0, ∞). And the exact conditional test's familiar conservatism is not the zero total's: the zero is the larger cause only while the expected total is below ln 2, and at an expected total of ten the exact interval still covers 97.8% for reasons that have nothing to do with it.

discrete · Oscillation Seven products of three splits. The correlation of every product of three centred median splits with every other, at a Spearman rank correlation of 0.4 between each pair of covariates. Under a Gaussian copula a main effect correlates with the other mains and with the triple (0.273), and with no pair at all, because the third moment is 0.000. Under a Frank copula, whose pairs are radially symmetric, the third moment is 0.078 and every other cell fills in: a main correlates 0.082 with the pair it is not in. In both panels a main is exactly uncorrelated with a pair it belongs to, which is the known zero.

The zero that needed two

A rule balancing two median splits removes nothing of their interaction, on every copula, because a centred split squares to a constant. Add a third covariate and the same arithmetic stops protecting the thing it looked as if it protected. The product of three splits is not orthogonal to the mains: at a rank correlation of 0.4 a rule on the mains removes 14.47% of it under a Gaussian copula. And the symmetry that decides the rest belongs to the triple rather than the pair, so Frank's copula, whose pairs are radially symmetric, leaks where Clayton's, whose pairs are not, barely does.

compound · Adjustment Which side the misses fall on. Misses of a nominal 95% forecast interval 1 step ahead on 60 observations of a first-order autoregression at 0.7, split by side. normal shocks: the interval as printed 3.1% below and 3.1% above, shocks from the residuals 3.8% below and 4.2% above, the same, coefficient re-estimated 3.3% below and 4.4% above. Student's t on five: the interval as printed 3.3% below and 3.0% above, shocks from the residuals 3.9% below and 3.0% above, the same, coefficient re-estimated 3.6% below and 3.1% above. skewed shocks: the interval as printed 0.0% below and 6.5% above, shocks from the residuals 4.3% below and 4.1% above, the same, coefficient re-estimated 1.8% below and 4.3% above.

A tail sixty residuals have barely seen

A forecast interval that reads a normal quantile has every one of its misses on one side when the shocks are skewed, and the natural repair is to draw future shocks from the model's own residuals. It works on the side: 0.0% below and 6.5% above becomes 4.3% and 4.1%. It pays for that with coverage, 93.5% to 91.6%, and on sixty observations it does this under normal shocks too. And a nominal 99% interval has a ceiling nothing in the resampling can lift, because one new shock beats all of sixty old ones in either direction with probability 3.3%.

forecast · Forecast Twelve groups from two clusters, τ = 1. Every group's truth is in one of two clusters, and the population's total spread is exactly τ = 1, so the analysis recovers τ̂ = 1.515 and every shrinkage weight is what it would be for a single normal population. The estimates are pulled towards the grand mean, which is the middle of the gap — a place 0 of the 12 truths are and 6 of the estimates end up.

One population, or two

Group effects from two clusters rather than one bell, with the same total spread. The analysis recovers the same population spread, uses the same weight for every group, and reports nothing unusual — while 46% of its estimates land in a region holding 6.6% of the truths.

hierarchical · Pooling Three detectors for one departure, all at 5%. How often each of three checks on the calibration scores fires, against the size of the drift, with every critical value simulated under no drift so that all three sit at 5.0% exactly. The incumbent — a rank comparison of the first half of the scores against the second — reaches four-in-five power at a growth factor of 4.31. Reading each score's rank against its position reaches it at 2.65, and the largest running departure of the scores from their mean at 2.12. The ordering of the three is the ordering by how much of the sample's arrangement each one uses.

A detector built for the ordering

The best of three checks for a drifting scale fires at half the growth factor the standard one needs — 2.12 against 4.31 — and still leaves 6.50 points of coverage gone before it does, against 0.51 for serial correlation. The reversal was not a property of the test.

conformal · Exchangeability How many starting observations a range needs to flag at most r of the next 100, fixed at the start or learning as the lot is used. 95% of the time, for any continuous population. At most 0: fixed 3,850, learning 3,850; At most 1: fixed 637, learning 512; At most 2: fixed 300, learning 197; At most 3: fixed 190, learning 101; At most 5: fixed 107, learning 36; At most 7: fixed 74, learning 15; At most 10: fixed 50, learning 4. With no flag allowed the two are the same, because a lot with no flag never moves the learning range.

A range that grows with the lot

A band set from the range of everything seen so far flags a unit only when it sets a new record, and for any continuous population those flags are independent, with the k-th unit flagged with probability 2/(n + k). That makes a few-misses promise far cheaper than the fixed range's — at most five of the next hundred from 36 starting observations rather than 107 — and exactly as expensive when no miss is allowed, because a lot with no flag never moves the band. The saving is bought by believing every flag. After a two-standard-deviation step in the mean, the fixed range flags 25.03 of the eighty units that follow and the learning range 3.62; by the last twenty units it sees a unit outside in 25.5% of lots, against 18.4% with no step at all.

estimated · Bands Coverage under a drifting scale, with the quantile taken over fewer and more recent scores. The noise scale grows linearly across the sample to 1 + v times its starting value; 4,000 draws at each v. Worst coverage over the sweep: all 199 scores 85.63%, the last 99 90.88%, the last 39 93.42%, the last 19 93.53%. At v = 2.12, where the best of three detectors first fires 80% of the time, they cover 88.11%, 91.86%, 94.02%, 94.45%. Every window covers 95% when nothing drifts.

A quantile over the recent scores

A noise scale that grows across the sample takes a split conformal interval from 95% to 85.63%, and the best detector cannot see most of that loss. Take the quantile over the last thirty-nine calibration scores instead of all 199 and the worst coverage across the whole sweep is 93.42%, for 3.7% more width when nothing has drifted. The same window covers 91.30% where all 199 cover 93.53% under strongly correlated errors — and switching to it only when the detector fires is worse than either, because the detector fires on correlation too.

conformal · Exchangeability How many units a range needs to let at most r of the next hundred outside, fixed, learning or forgetting. 95% of the time, for any continuous population. At most 1: fixed 637, window 564, learning 512; At most 2: fixed 300, window 236, learning 197; At most 5: fixed 107, window 64, learning 36; At most 10: fixed 50, window 27, learning 4. The window counts are from 20,000 lots of uniform draws; the other two are exact.

A range that forgets

A band set from the range of the last sixty-four units flags each unit of a stable lot with probability exactly 2/65, for ever, and its flags arrive as if independent — so a promise of at most five outside in the next hundred costs sixty-four units, between the learning range's thirty-six and the fixed range's hundred and seven. What it was supposed to buy, a flag that lasts as long as a step and then stops, it cannot give: it absorbs a two-sigma step as fast as the learning range, 1.89 flags in ten units against 1.93, because it learns from what it flags. Refusing to learn from a flag makes it a fixed range that costs 141.

estimated · Bands

Named alongside it

The objects these essays reach for when they reach for this one.

CoverageCalibration setConformal predictionDistribution-freeInterval widthNonconformity scoreOrder statisticClosed formMarginal coverageMonte CarloModel misspecificationOvercoverage

All concepts