Normal Distribution
nothing is called 95% until it has been counted
Field

The prior, doing visible work

A credible interval says the thing everyone wants a confidence interval to say, and it needs a prior to do it. So the prior is treated as a component with a measurable effect: it is worth a stated number of observations, and the interval it produces has a coverage that can be summed over the sample space like any other.
weakly informative — Beta(2, 2), updated by 5 of 20. The prior is worth 4 observations. With 20 observations the posterior mean is 0.292, against a data proportion of 0.250 and a prior mean of 0.500.

What a prior is worth

A prior is not a philosophical position, it is a component with a stated size. For a proportion it is worth exactly a + b observations, which turns "how much does the prior matter" from an argument into a subtraction.

 What a 95% credible interval covers, n = 20. Computed by summing over all 21 possible counts rather than by simulating them. Jeffreys' prior covers close to 95% across the range; a confident prior centred in the wrong place covers almost nothing where the truth is far from it.

What a credible interval covers

A credible interval makes the statement everyone wants and does not claim to have a coverage. It has one anyway, it can be summed over the sample space exactly, and on a reasonable prior it beats the interval taught first.

 A normal mean with a flat prior — one interval, two readings. Both intervals are [1.878, 3.446]. The frequentist reading is that the procedure captures the truth 95% of the time; the Bayesian reading is that the parameter is in this interval with probability 0.95. The endpoints are identical to machine precision.

Where the two schools agree

With a flat prior on a normal mean, the credible interval and the confidence interval are the same interval, endpoint for endpoint. Knowing exactly when that stops being true is more useful than either camp's general argument.

 What a positive test means, sensitivity 90%, specificity 95%. At a prevalence of one in a thousand, 98 of every hundred positives are false. At one in 10, 33 are. The test has not changed.

The base rate was always Bayes

The screening arithmetic everybody finds counter-intuitive is a posterior update with a prior of one in a thousand. Naming it that way turns a famous puzzle into an instance of a rule, and makes the sequential version obvious.

 Two 95% intervals for 2 of 20, under Jeffreys — Beta(½, ½). The equal-tailed interval runs from 0.0214 to 0.2839 and is 0.2625 wide; the shortest interval runs from 0.0093 to 0.2540 and is 0.2447 wide. Both hold 95% of the posterior, and the shorter one buys its 6.8% by moving its lower endpoint towards the denser side.

The shortest interval, and the one that does not move

Two 95% intervals come out of every posterior and they are not the same set. The shorter one is shorter by 4.86% on average and 22.41% at its best, it covers 86.72% where the other covers 95.68%, and it is not even the shortest once the parameter is written a different way.

 Four 95% intervals for the odds after 2 of 20. credible, transformed: 0.0218 to 0.3964. Wald, transformed: -0.0305 to 0.3012. delta method on the odds: -0.0512 to 0.2734. delta method on the log-odds: 0.0258 to 0.4789. The first two are the same intervals for the proportion with their endpoints put through the odds; the last two are fresh approximations made on the new scale.

An interval for something else

An interval for the odds is free — put the endpoints through the odds and the coverage does not move, exactly, for any interval at all. The method everyone uses instead computes a new standard error on the new scale, and at twenty trials that costs four points of coverage, produces negative odds, and has no value at all when nothing was observed.

 A prior worth 35 observations, moved across the range — truth 0.1, n = 20. The same prior weight centred at each of 33 places. Its interval covers 100.0% where the centre is near the truth and 0.0% at its worst, while the mean width where it covers least is 0.221 against a flat prior's 0.263 on the same data.

When the prior is confident and wrong

A prior worth thirty-five observations, centred in the wrong place, produces a 95% interval that covers nothing at all — and reports a width 5% narrower than an honest one. It takes seventeen thousand observations to repair, not thirty-five, and the worst study to run is the one whose sample size equals the prior's weight, exactly.

 A prior centred on the truth's population and too confident about it: coverage across the population, and how often the check flags it. Truths drawn from a population of proportions centred at 0.2 and worth ten observations; twenty trials per study; the analyst's prior centred at 0.2 with the weight on the axis. Coverage of the 95% credible interval, exact over the population: 94.7% at 2, 96.0% at 5, 95.0% at 10, 88.1% at 20, 75.9% at 35, 61.3% at 60, 48.2% at 100, 34.0% at 200, 23.9% at 400. Share of studies whose count the prior-predictive check puts in its 5% tail: 0.1%, 1.8%, 3.2%, 14.2%, 17.3%, 17.3%, 21.9%, 21.9%, 21.9%.

A prior wrong about its own spread

A prior can be centred exactly where the truths lie and still be wrong about how far they spread. Ten times too confident — worth a hundred observations about proportions whose population is worth ten — its 95% credible intervals cover 48.2% of the population's studies at twenty trials, and nothing in the interval shows it: it is 0.456 of a flat prior's width, against 0.436 for a prior whose hundred observations' confidence is earned. A prior with tails too thin misses the one study in ten that comes from somewhere unusual a third of the time, and a flat component of two per cent brings that study's coverage from 65.6% to 84.0%.

All essays