Normal Distribution
nothing is called 95% until it has been counted
Concept

Prior sensitivity — where it appears

How much a conclusion moves when the prior is changed, which is the check that says whether a Bayesian answer is about the data or about the assumption. It matters most where the data is thin, which is exactly where a prior is most tempting to reach for.

Named by 11 essays across 5 fields — each of them below, with the objects they name alongside it.

Three priors on the spread, at a true τ of 0.5. The posterior for τ under a flat prior (mean 1.66), a half-Cauchy of scale 1 (1.32) and one of scale 0.25 (1.15). The three answers differ by 30% of the widest. The prior does visible work when eight groups cannot separate a small spread from none, and almost none when they can.

A prior on the spread

Integrating over the population spread means putting a prior on it, which sounds like the objection rather than the repair. The prior's effect is measurable, it is invisible where the groups are clearly different, and the reflex choice for a scale parameter turns out not to have a posterior at all.

fullbayes · Prior What a 95% credible interval covers, n = 20. Computed by summing over all 21 possible counts rather than by simulating them. Jeffreys' prior covers close to 95% across the range; a confident prior centred in the wrong place covers almost nothing where the truth is far from it.

What a credible interval covers

A credible interval makes the statement everyone wants and does not claim to have a coverage. It has one anyway, it can be summed over the sample space exactly, and on a reasonable prior it beats the interval taught first.

bayes · Credible A design that is right once, against one that is never wrong by much. Two designs for the same two-parameter model, scored at every true value of K across a range of 16-fold. The local design is the two-point optimum for a guess of K = 1: it reaches 100% there and 66.7% at the worst point of the range. The hedged design maximises the average of log|M| over a prior spanning a factor of 4 either side, uses 3 settings rather than two, and is never below 75.4%. What it costs is 10.4 points at the one value the local design was built for — which is the whole trade, and it is only available to somebody willing to say how wrong the guess might be.

The design that hedges

A locally optimal design is right at one value of the unknown and 23.9% efficient at the edge of a sixteenfold range. Averaging the criterion over a prior instead buys the worst case back to 56.3% — and buys it by adding support points, at spreads the arithmetic decides rather than the experimenter — a third setting at a factor of 3.36 and a fourth at 8.86.

criteria · Local design How often a trial that borrows its control from an earlier one declares a treatment with no effect a success, by the earlier control's drift. Fifty patients an arm, an earlier control of two hundred, a one-sided 2.5% threshold. Without borrowing the rate is 2.5% at every drift. Pooling outright reaches 73.1% at a drift of half a standard deviation. A half-normal prior of scale 0.05 on the between-trial spread peaks at 35.2% and is still at 31.8% at a drift of 1.5; a half-Cauchy of the same scale peaks at 11.0% near a drift of 0.4 and falls back to 3.4%.

A control borrowed from the last trial

A trial of fifty patients an arm can borrow its control group from an earlier trial of two hundred by treating the two control means as draws from one population with a spread between trials. With two trials there is one difference to learn that spread from, so the prior on it decides how much is borrowed — and its tail decides whether the borrowing stops when the two trials disagree. A half-normal prior of scale 0.05 buys 17.4 points of power and, at a drift of 1.5 standard deviations, still declares a treatment with no effect a success 31.8% of the time. A half-Cauchy of the same scale buys 13.6 points, peaks at 11.0% and falls back to 3.4%.

fullbayes · Prior How often a trial borrowing from 16 earlier trials that agree declares a treatment with no effect a success, by its own control's drift. Fifty patients an arm, each earlier control of two hundred, a one-sided 2.5% threshold. With 16 earlier trials, the half-normal prior peaks at 88.1%, the half-Cauchy at 80.7% near a drift of 0.6, and the half-Cauchy with a fifth of the prior on a vague component at 16.9%. At a drift of 1.5 the three read 73.3%, 25.5% and 3.7%.

A history that agrees with itself

Borrowing a control from sixteen earlier trials that agree with each other should be safer than borrowing from one, and it is the opposite. Sixteen agreeing trials estimate the spread between trials as small, and a small spread estimated confidently is a licence to pool: under a half-Cauchy prior that let go of a single disagreeing trial, a current control 0.6 standard deviations from a sixteen-trial history turns a treatment with no effect into a success 80.7% of the time, against 10.5% at worst with one earlier trial. Moving a fifth of the prior onto a vague component brings the worst case to 16.9% and keeps power at 91.0%, against 70.5% without borrowing.

fullbayes · Prior How far four estimates of the chosen arm sit from its true effect, with one arm ahead by 0.5. Eight arms of sixty, the best carried forward with thirty more, 20,000 trials. Average error: both stages, as published +0.006; shrunk under a normal prior −0.103; shrunk under a t prior, 3 df −0.101; normal prior with a fifth vague −0.073. Root-mean-square error: 0.151, 0.189, 0.192, 0.176.

A prior that lets one arm stand out

Shrinking the chosen arm of eight under a normal prior understates a genuine winner by 0.103. A t prior on three degrees of freedom with the same spread understates it by 0.101 — the tail is too far out to reach a lead of three and a half standard errors. A prior with a fifth of its weight on a vague component understates it by 0.073, costs 0.010 of bias when the arms are alike, and still has more error than the unshrunk estimate. With sixteen arms the same mixture recovers three fifths of the loss, because a bell of fifteen shows its outlier; with eight, the arms put more than half their posterior weight on 'one stands out' in 0.4% of trials where one does.

adaptive · Curse A prior worth 35 observations, moved across the range — truth 0.1, n = 20. The same prior weight centred at each of 33 places. Its interval covers 100.0% where the centre is near the truth and 0.0% at its worst, while the mean width where it covers least is 0.221 against a flat prior's 0.263 on the same data.

When the prior is confident and wrong

A prior worth thirty-five observations, centred in the wrong place, produces a 95% interval that covers nothing at all — and reports a width 5% narrower than an honest one. It takes seventeen thousand observations to repair, not thirty-five, and the worst study to run is the one whose sample size equals the prior's weight, exactly.

bayes · Credible Each interval covers one question and not the other. Coverage of each interval for the overall mean, scored against both estimands, over 20,000 two-site studies of 10 observations apiece. The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and over-covers the two sites in hand at 98.25%. Both are correct; they are answers to different questions printed in the same place.

What a two-unit study should report

The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and is 11.6 times wider.

multilevel · Levels How often a trial borrowing from 8 earlier trials on a trend declares a treatment with no effect a success. Fifty patients an arm, each earlier control of two hundred, the current trial one step after the last. At slopes 0, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1: exchangeable 2.7%, 4.4%, 6.6%, 8.3%, 9.9%, 8.3%, 6.7%; with a fifth of the prior vague 2.4%, 3.6%, 5.5%, 6.9%, 7.6%, 6.5%, 5.7%; with a line in time 2.9% at every slope.

A history that has been moving

Earlier control arms borrowed as exchangeable draws are pooled towards the history's middle, and when the control mean has been rising the current trial sits at its end. A drift of 0.02 standard deviations a trial — a step between neighbouring trials of under a third of either one's standard error — turns a treatment with no effect into a success 9.9% of the time from sixteen earlier trials, and a drift of 0.05 overstates the effect by 0.130 standard deviations from eight. A line in time removes the bias exactly, at every slope, and pays for it in borrowing: from eight earlier trials it borrows 175 control patients where the exchangeable model borrows 439.

fullbayes · Prior The bias each model puts into the treatment effect when sixteen earlier controls stepped once. Sixteen earlier trials of two hundred, the first eight a step below the last eight, which sit at the current control's level; the current trial has fifty patients an arm and no treatment effect; 1,500 trials at each step. Exchangeable trials: -0.003, 0.020, 0.041, 0.056, 0.063, 0.065, 0.054. A line in time: -0.003, -0.016, -0.029, -0.041, -0.052, -0.067, -0.075. A change point: -0.003, 0.003, 0.003, -0.001, -0.002, -0.003, -0.003 — at steps of 0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5.

A history that stepped

When sixteen earlier control arms stepped once — the first eight a fifth of a standard deviation below the last — borrowing them as exchangeable flatters a treatment with no effect by 0.063 and stops borrowing, and fitting a line through them penalises it by 0.052. A model with a change point is unbiased at every step size, borrows more the larger the step, 512 patients at a fifth, and puts two thirds of its belief on the right place for the change. Against a steady drift it is the one that fails, so each of the three is safe only against the shape of history it assumes.

fullbayes · Prior A prior centred on the truth's population and too confident about it: coverage across the population, and how often the check flags it. Truths drawn from a population of proportions centred at 0.2 and worth ten observations; twenty trials per study; the analyst's prior centred at 0.2 with the weight on the axis. Coverage of the 95% credible interval, exact over the population: 94.7% at 2, 96.0% at 5, 95.0% at 10, 88.1% at 20, 75.9% at 35, 61.3% at 60, 48.2% at 100, 34.0% at 200, 23.9% at 400. Share of studies whose count the prior-predictive check puts in its 5% tail: 0.1%, 1.8%, 3.2%, 14.2%, 17.3%, 17.3%, 21.9%, 21.9%, 21.9%.

A prior wrong about its own spread

A prior can be centred exactly where the truths lie and still be wrong about how far they spread. Ten times too confident — worth a hundred observations about proportions whose population is worth ten — its 95% credible intervals cover 48.2% of the population's studies at twenty trials, and nothing in the interval shows it: it is 0.456 of a flat prior's width, against 0.436 for a prior whose hundred observations' confidence is earned. A prior with tails too thin misses the one study in ten that comes from somewhere unusual a third of the time, and a flat component of two per cent brings that study's coverage from 65.6% to 84.0%.

bayes · Credible

Named alongside it

The objects these essays reach for when they reach for this one.

PriorHierarchical modelPartial poolingThe half-Cauchy priorCoverageError rateStatistical powerVariance componentsCredible intervalFlat priorInterval widthPosterior

All concepts