Base rate — where it appears
Named by 13 essays across 4 fields — each of them below, with the objects they name alongside it.
An identity in three terms
Reliability minus resolution plus uncertainty is quoted as a rewriting of a probability score. It is an identity to 2.6·10⁻¹⁵ on the one grouping where reliability is the whole score and resolution exactly cancels uncertainty, and it is out by 0.004125 on the coarsest grouping anybody would actually draw.
The second test that is not a second opinion
Two positives from a 90/95 test on a one-in-a-thousand condition give a 24.49% chance of disease if the tests are independent. At a correlation of 0.1 between their errors it is 10.16%, and at 0.5 it is 3.16% — barely more than the 1.77% one positive was worth.
What a positive test is worth
A test that is 90% sensitive and 95% specific sounds accurate. For a condition affecting one person in a thousand, 98% of its positive results are wrong, and a worse test on a commoner condition beats a better test on a rare one.
A curve that is a binning
A forecaster with no miscalibration in it at all reads 0.001429 at five bins and 0.014100 at fifty, on the same five hundred forecasts. The closed form is K/n times the forecaster's own irreducible score, and subtracting it returns zero.
The test is a point somebody chose
A test reported as 90% sensitive and 95% specific is not two properties of a test. It is one property read at a threshold, and the threshold that minimises harm runs from 3.05 standard deviations of the score at a prevalence of one in ten thousand to −0.12 at one in two — 45% of cases detected at one end and 99.9% at the other.
Calibrated and useless
Six forecasters that are calibrated to 2·10⁻³³ run from resolution exactly 0 to 0.092758, and three forecasters with reliabilities from 0 to 0.013025 have areas under the ROC curve identical to every bit a double carries. Each measure is exactly blind to what the other one sees.
The prevalence the test has to estimate
Every predictive value takes a prevalence as given, and the prevalence is usually estimated from the same test's positive rate. At a true prevalence of one in a thousand that rate reads 5.09% — fifty times the truth — and the correction that inverts it is unbiased, 18% more variable, and negative on 48.6% of samples of a thousand.
The base rate was always Bayes
The screening arithmetic everybody finds counter-intuitive is a posterior update with a prior of one in a thousand. Naming it that way turns a famous puzzle into an instance of a rule, and makes the sequential version obvious.
A score that rewards lying
An absolute-error score pays a forecaster exactly ⅛ of a point to replace a true quarter with a zero, and over two hundred records a liar beats a truthful forecaster on 200 of 200. A skill score against the forecaster's own average buys 0.012633 of reported skill for 0.002035 of real score.
A threshold that jumps
When a test's scores are equally spread in the healthy and the diseased, the harm-minimising threshold slides smoothly as the prevalence changes. Make the diseased scores twice as spread — the ordinary shape of a group that mixes mild and severe cases — and keep the published 90/95 pair, and the best threshold flags 81.1% of healthy people at a prevalence just under 25.7% and every healthy person just above it. At three times the spread it jumps from 45.7% to everyone at 13.0%. A threshold that jumps cannot be given as a formula; it has to be given with the second-best point beside it.
The liar with two answers
The forecaster an absolute-error score pays for says only 0 or 1, and on the ROC square it is a single point: its area is (TPR + TNR)/2 = 0.7684, against the honest forecaster's 0.8683, and it falls below the honest one on 200 of 200 counted records. No relabelling of its two answers returns what it threw away — the best recovers a Brier score short of the honest one by exactly the 0.022154 of resolution lost — and below a signal correlation of 0.7332 the same score prefers saying no every time to an honest forecast.
The threshold a validation study can see
When diseased scores are twice as spread as healthy ones, the harm-minimising threshold jumps near a prevalence of 26% from an interior cut to flagging everyone, because the true operating curve keeps climbing in a tail where few diseased people score. A validation study of a hundred diseased and three hundred healthy cannot see that tail: its own curve reaches 100% sensitivity at its lowest diseased score and goes flat. Just past the jump, only 7.7% of such studies recommend flagging everyone, and their recommended thresholds cause 15.4% more harm than the best on average. Fitting a normal curve to each group first sees the tail: 47.4% of studies then recommend flagging everyone and the excess harm falls to 2.5%. More healthy people in the study change nothing; more diseased people do.
A threshold reported with its interval
Resample a validation study of a hundred diseased and three hundred healthy people, refit the binormal model each time, and read off the threshold each resample recommends. Far from the prevalence at which the best threshold jumps to flagging everyone, the interval is tight — flagging 12% to 24% of healthy people at a prevalence of 5%. Near the jump, the median study's interval runs from flagging under half of the healthy to flagging everyone, and 91% of intervals cross the jump. The two ends differ by fifteen false alarms' worth of harm per hundred people, and no rule for choosing inside the interval beats the study's own choice at every prevalence.
Named alongside it
The objects these essays reach for when they reach for this one.
Sensitivity and specificityROC curveScreeningBrier scorePredictive valueProbability forecastDecision thresholdExpected lossForecast calibrationPrevalenceReliability termClimatological forecast