Normal Distribution
nothing is called 95% until it has been counted
Concept

Credible interval — where it appears

A range containing a stated share of the posterior distribution, which is a statement about belief given the data rather than about repetition. Its frequentist coverage is a separate question with a separate answer, and the two agree only under conditions worth naming.

Named by 12 essays across 5 fields — each of them below, with the objects they name alongside it.

Coverage of four nominal 95% intervals, n = 20. Computed exactly by summing over all 21 possible counts, not simulated. The Wald interval drops to 18.2% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.

What the 95% refers to

An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time.

intervals · Coverage What a 95% credible interval covers, n = 20. Computed by summing over all 21 possible counts rather than by simulating them. Jeffreys' prior covers close to 95% across the range; a confident prior centred in the wrong place covers almost nothing where the truth is far from it.

What a credible interval covers

A credible interval makes the statement everyone wants and does not claim to have a coverage. It has one anyway, it can be summed over the sample space exactly, and on a reasonable prior it beats the interval taught first.

bayes · Credible A normal mean with a flat prior — one interval, two readings. Both intervals are [1.878, 3.446]. The frequentist reading is that the procedure captures the truth 95% of the time; the Bayesian reading is that the parameter is in this interval with probability 0.95. The endpoints are identical to machine precision.

Where the two schools agree

With a flat prior on a normal mean, the credible interval and the confidence interval are the same interval, endpoint for endpoint. Knowing exactly when that stops being true is more useful than either camp's general argument.

bayes · Credible Counted coverage of two 95% intervals, over 2,000 datasets. Each point is one of the eight groups, at its own standard error. The integrated interval covers 95.2% overall against its stated 95%; the plug-in covers 78.8%, and its shortfall grows with the group's standard error — from 86.1% at se 0.5 to 77.0% at se 2.1. The mean widths are 3.48 and 2.66.

The interval that integrates

A credible interval for one group in a hierarchy has to average over every value the population spread might take. That averaging is what makes it cover — 95.2% against the plug-in's 78.8% — and it costs 31% more width, a heavier tail, and a mixture rather than a normal.

fullbayes · Credible Two 95% intervals for 2 of 20, under Jeffreys — Beta(½, ½). The equal-tailed interval runs from 0.0214 to 0.2839 and is 0.2625 wide; the shortest interval runs from 0.0093 to 0.2540 and is 0.2447 wide. Both hold 95% of the posterior, and the shorter one buys its 6.8% by moving its lower endpoint towards the denser side.

The shortest interval, and the one that does not move

Two 95% intervals come out of every posterior and they are not the same set. The shorter one is shorter by 4.86% on average and 22.41% at its best, it covers 86.72% where the other covers 95.68%, and it is not even the shortest once the parameter is written a different way.

bayes · Credible Four 95% intervals for the odds after 2 of 20. credible, transformed: 0.0218 to 0.3964. Wald, transformed: -0.0305 to 0.3012. delta method on the odds: -0.0512 to 0.2734. delta method on the log-odds: 0.0258 to 0.4789. The first two are the same intervals for the proportion with their endpoints put through the odds; the last two are fresh approximations made on the new scale.

An interval for something else

An interval for the odds is free — put the endpoints through the odds and the coverage does not move, exactly, for any interval at all. The method everyone uses instead computes a new standard error on the new scale, and at twenty trials that costs four points of coverage, produces negative odds, and has no value at all when nothing was observed.

bayes · Credible How often seven 95% intervals cover the small groups of a league table whose size relation is steepening below the median size. Groups of twenty patients or fewer, 600 tables of a hundred groups. Coverage: one centre, plugged in 58.0% (average width 2.81); one centre, its uncertainty added 58.5% (average width 2.83); a fitted line, plugged in 83.1% (average width 1.58); a fitted line, its uncertainty added 88.4% (average width 1.77); a fitted quadratic, plugged in 85.5% (average width 1.53); a fitted quadratic, its uncertainty added 93.7% (average width 1.92); the true relation and spread 94.8% (average width 1.55).

The interval a small group is given

A league table's 95% interval for a small group, pooled towards a size relation fitted from the table, covers 92.8% of the time when the relation is straight and 83.1% when it steepens below the median — and 77.4% for the groups of eight patients or fewer. Adding the fitted centre's own uncertainty, weighted by how much the group borrows from it, brings the line to 88.4% and a fitted quadratic to 93.7%, and the quadratic's interval covers 94.6% of the smallest groups at every shape tried. One centre, plugged in, covers the smallest groups 35.2% of the time when the relation steepens, and no added variance repairs a centre in the wrong place.

borrowed · Shrinkage A prior worth 35 observations, moved across the range — truth 0.1, n = 20. The same prior weight centred at each of 33 places. Its interval covers 100.0% where the centre is near the truth and 0.0% at its worst, while the mean width where it covers least is 0.221 against a flat prior's 0.263 on the same data.

When the prior is confident and wrong

A prior worth thirty-five observations, centred in the wrong place, produces a 95% interval that covers nothing at all — and reports a width 5% narrower than an honest one. It takes seventeen thousand observations to repair, not thirty-five, and the worst study to run is the one whose sample size equals the prior's weight, exactly.

bayes · Credible Every group's 90% rank interval in one league table whose size relation is steepening below the median size, with its true rank. A hundred groups from four to four hundred patients, ranked from their posterior under a fitted quadratic size relation whose uncertainty is drawn once per table. The groups of twenty patients or fewer have rank intervals 30.1 places wide on average, and 34 of their 36 true ranks fall inside; the groups of over a hundred, 32.1 places, with 27 of 30 inside.

A ranking with its uncertainty

A league table's reader wants a rank, and a rank interval inherits a fitted size relation's errors differently from an effect interval. When the relation steepens below the median size, a fitted line's 95% interval for a small group's effect covers 83.1% of the time; its 90% interval for the same group's rank covers 95.6%, because the line's error lifts the small groups into the crowded middle of the table, where rank intervals are wide. The error surfaces instead in the large groups' ranks, 87.6%, and in width: 36.9 places against the 27.3 the true relation needs. A quadratic's uncertainty has to be drawn once per table, since it moves every small group together — added group by group it over-covers at 92.8%.

borrowed · Shrinkage The textbook (Wald) interval's coverage among studies whose count of 40 lands in the tails. Coverage of the nominal 95% textbook (Wald) interval at 40 trials, conditional on the count being at most k or at least 40 − k, computed exactly over the counts for every true proportion. With k = 2 its highest value anywhere is 88.96%; With k = 5 its highest value anywhere is 94.90%; With k = 8 its highest value anywhere is 95.57%. A set whose highest value stays below 95% is a set of counts on which the label is wrong whatever the proportion is.

A count that bets against its interval

A reader who sees only the count can bet against an interval's 95%, and win whatever the proportion is, exactly when some set of counts carries less than 95% at every proportion. Out of forty trials the textbook interval has one — a count of five or fewer, or thirty-five or more, covers at most 94.90% — and the score interval has none: a prior exists under which every one of its forty-one intervals is 96.46% credible, and that prior is the proof that no bet can be found.

intervals · Coverage A prior centred on the truth's population and too confident about it: coverage across the population, and how often the check flags it. Truths drawn from a population of proportions centred at 0.2 and worth ten observations; twenty trials per study; the analyst's prior centred at 0.2 with the weight on the axis. Coverage of the 95% credible interval, exact over the population: 94.7% at 2, 96.0% at 5, 95.0% at 10, 88.1% at 20, 75.9% at 35, 61.3% at 60, 48.2% at 100, 34.0% at 200, 23.9% at 400. Share of studies whose count the prior-predictive check puts in its 5% tail: 0.1%, 1.8%, 3.2%, 14.2%, 17.3%, 17.3%, 21.9%, 21.9%, 21.9%.

A prior wrong about its own spread

A prior can be centred exactly where the truths lie and still be wrong about how far they spread. Ten times too confident — worth a hundred observations about proportions whose population is worth ten — its 95% credible intervals cover 48.2% of the population's studies at twenty trials, and nothing in the interval shows it: it is 0.456 of a flat prior's width, against 0.436 for a prior whose hundred observations' confidence is earned. A prior with tails too thin misses the one study in ten that comes from somewhere unusual a third of the time, and a flat component of two per cent brings that study's coverage from 65.6% to 84.0%.

bayes · Credible Coverage of the population mean from two units, by the true spread between units. 4,000 two-unit studies at each true spread, ten observations a unit, within-unit standard deviation 1.2. exact, on one degree of freedom: 94.97%, 95.50%, 94.95%, 94.83%, 95.15%, 95.20%, 94.67%. spread supplied as 1: 100.00%, 100.00%, 99.92%, 95.53%, 81.85%, 70.03%, 51.55%. half-normal, scale 1: 100.00%, 100.00%, 99.78%, 94.63%, 86.25%, 79.70%, 68.38%. half-Cauchy, scale 1: 100.00%, 100.00%, 100.00%, 98.98%, 95.50%, 92.38%, 88.00%. inverse-gamma(0.001, 0.001) on the variance: 100.00%, 100.00%, 100.00%, 99.15%, 97.05%, 95.28%, 92.65% — at spreads 0, 0.25, 0.5, 1, 1.5, 2, 3. The exact interval holds 95% at every spread; every integrated one over-covers when the spread is small and under-covers when it is large, and where it crosses 95% is set by the prior's tail.

A prior on the spread between two sites

Integrate the between-site spread of a two-site study under a half-normal prior of scale 1 and the interval for the overall mean is 3.01 wide and covers 94.63% when the true spread is 1 — the supplied interval again, under another name. At a true spread of 3 it covers 68.38%. Every prior tried covers 95% averaged over its own draws and none covers it at every spread; each holds up to about its own scale, because two sites move the posterior median of the spread by less than a third of what the truth moves.

multilevel · Levels

Named alongside it

The objects these essays reach for when they reach for this one.

CoveragePriorPosteriorFlat priorHierarchical modelBinomial proportionConfidence intervalInterval widthJeffreys' priorPrior sensitivitySample sizeShrinkage

All concepts