The prevalence the test has to estimate

Every predictive value needs a prevalence. The predictive value curve is drawn against it, the odds form multiplies it, and the best threshold is decided by it.

Where does it come from? Usually from testing people and counting the positives — which means the prior a test needs is estimated by the test that needs it.

The apparent rate is not the prevalence #

The share of a tested population that comes back positive is

$$\text{apparent} = p\,\text{sens} + (1-p)(1-\text{spec}),$$

which at p = 0.001, 90% sensitivity and 95% specificity is 5.09%.

The true prevalence is 0.1%. The measured rate is fifty times it, and essentially all of it is the test’s own false-positive rate: 5% of the 99.9% who are healthy is 4.995%, against 0.09% from the people who actually have the condition.

That is not a subtlety. At any prevalence well below the test’s false-positive rate, the apparent rate measures the test rather than the population, and the population’s contribution is a rounding error on the instrument.

The same arithmetic run at 2% true prevalence gives an apparent 6.70%, and at 10% gives 13.50%. The raw rate is always an overestimate when the false-positive rate exceeds the false-negative contribution, which is almost always.

The correction, which is one line #

Invert the expression:

$$\hat{p} = \frac{\text{apparent} - (1 - \text{spec})}{\text{sens} + \text{spec} - 1}.$$

That is the Rogan–Gladen estimator, it is unbiased whenever the sensitivity and specificity are known, and the counted mean confirms it: 0.100% against a true 0.100% over twenty thousand samples of a thousand people.

The denominator is worth naming. sens + spec − 1 is Youden’s index, it is the test’s ability to distinguish the two groups on one scale, and it is 0 for a useless test and 1 for a perfect one. Here it is 0.85.

What the denominator costs #

Dividing by 0.85 multiplies everything, including the noise.

The corrected estimate’s standard deviation is the apparent rate’s divided by Youden’s index, so the inflation factor is 1/0.85 = 1.176, and the count agrees: 1.178 measured over twenty thousand samples.

That sounds mild. The number that matters is the estimate’s spread against the thing being estimated:

At a true prevalence of 0.1% the estimate’s standard deviation is 0.82 percentage points — eight times the quantity it is estimating. An unbiased estimator whose noise is eight times its target is unbiased and useless, and the first column is the only one anybody quotes.

How many people it would take #

The table’s first row is the situation a screening programme is in, so the useful question is what sample size would make it work.

The corrected estimate’s standard deviation is $\sqrt{a(1 - a)/n}$ divided by Youden’s index, with a the apparent rate. At a true prevalence of 0.1% the apparent rate is 5.09% and that gives

$$\operatorname{sd}(\hat{p}) = \frac{1}{0.85}\sqrt{\frac{0.0509 \times 0.9491}{n}} .$$

Asking for a standard deviation of a tenth of the true prevalence — 0.01 percentage points, which is what it would take for the estimate to be worth reporting to one significant figure — needs

$$n \approx 6.7 \times 10^{6}.$$

Six and a half million people, to measure a prevalence of one in a thousand with a test that is 95% specific.

The reason is visible in the expression: the numerator is the apparent rate’s variance, and the apparent rate is fifty times the prevalence, so the noise is set by a quantity that has almost nothing to do with what is being measured. A perfect test — specificity 1, Youden 0.9 — would face an apparent rate of 0.09% and need about 111,000 people for the same precision, a factor of sixty fewer.

That factor is the honest price of the test’s false positives, and it is not a correction that can be applied afterwards. Once the sample is collected, the information is not there.

And it goes negative #

The correction subtracts (1 − spec) from the apparent rate, so it returns a negative number whenever the apparent rate falls below the test’s own false-positive rate.

At a true prevalence of 0.1% with a thousand people tested, that happens on 48.6% of samples. Nearly half the time, the honest answer to “what is the prevalence” is a negative number.

At 0.5% it is 30.4%. At 1%, 13.8%. At 5% and above it stops happening.

Two responses are available and they are not equivalent.

Truncate at zero. Report max(0, p̂). That is what software does and it destroys the unbiasedness — truncating a symmetric estimator at a boundary it crosses half the time produces something biased upwards, and the bias is largest exactly where the estimate was already useless.

Report it as it is. A negative prevalence estimate is not an error; it is the data saying the observed positive rate is consistent with nobody having the condition, and it is the correct summary of that evidence. It reads badly and it is honest, which is the usual trade that ends up being the right one — the estimate is a point on a line and the interval around it is what should have been reported anyway.

The circularity, stated properly #

Putting the three pieces together gives a loop worth naming.

The predictive value needs the prevalence. The prevalence is estimated from the test’s positive rate. The correction needs the sensitivity and specificity. Those come from a validation study against a reference standard — and the reference standard is another test, whose own characteristics are estimated against something else.

At the bottom of the chain there is either a genuinely definitive test (an autopsy, a culture, a direct observation) or there is not. When there is, the chain terminates and everything above it is sound. When there is not, the whole structure is estimated against itself, and the literature has a name for what happens: imperfect gold standard bias, which makes every downstream characteristic wrong in a direction that depends on the reference’s own errors.

The practical form of the problem is short. Sensitivity and specificity are treated as known constants in every calculation above, and they are estimates with their own standard errors. A validation study of two hundred known positives estimates a 90% sensitivity to within about ±4 points, and propagating that uncertainty through the correction above widens everything again — by an amount nobody on this page has computed and which is not small when Youden’s index is in the denominator.

What the uncertainty in the test’s own characteristics adds #

Treating sensitivity and specificity as known is the assumption that makes every line above exact, and it is worth seeing what relaxing it does, because the direction is not neutral.

The correction divides by Youden’s index. Youden is estimated, so the divisor is a random variable near 0.85, and dividing by a random variable near 0.85 is not the same as dividing by 0.85 — the result has a heavier upper tail, because the occasional small divisor produces a very large estimate.

The effect is worse for a worse test. At Youden 0.85 the divisor is comfortably away from zero and the extra spread is modest. At Youden 0.3 — a 65% sensitive, 65% specific test, which exists — a validation study’s sampling error can put the estimated Youden near 0.1, and the corrected prevalence then jumps by a factor of three for reasons that are entirely about the validation study.

That is the same structure as a ratio whose denominator can be small, and it carries the same consequence: the estimator’s distribution is not approximately normal, so an interval built from a standard error is the wrong shape, and the standard error itself may not exist.

The practical rule that follows is a threshold on the test rather than on the sample. A prevalence correction is trustworthy when Youden’s index is comfortably away from zero relative to its own standard error, and unreliable otherwise — which is a condition on the validation study’s size as well as on the test’s quality, and neither is usually reported beside a corrected prevalence.

What actually gets done #

Three practices, in the order of how defensible they are.

Use a definitive test on a subsample. Test everybody with the cheap assay, and a random sample of them with the definitive one. The subsample estimates the test’s characteristics in this population — which is not the same as in the validation population — and the correction then uses the right numbers. It is the design that closes the loop, and it costs a subsample.

Use an external prevalence. Registry data, a previous survey, a different population. That removes the circularity and introduces a transportability assumption, which is a different problem and often a smaller one — and it is exactly a prior brought in from outside, with the same obligation to say how much of the answer it is supplying.

Report the apparent rate and call it a prevalence. Common, and it is the practice this page exists to name. At the prevalences screening programmes operate at, the apparent rate is mostly the test’s false-positive rate, so what gets reported as a population characteristic is an instrument characteristic.

The last one has a signature that is easy to spot: a reported prevalence very close to a test’s known false-positive rate, across populations that ought to differ. Several early serosurvey estimates had exactly that shape, and the correction is a line of arithmetic away.

There is a second signature and it is the stronger one. A reported prevalence that does not fall when the test is improved is a reported prevalence that was measuring the old test. Switching to a more specific assay lowers the apparent rate by exactly the difference in false-positive rates, and a field that reads that fall as a real decline in the condition has confused an instrument change with a finding — which is the same confusion between a study’s own properties and its subject’s that keeps arriving here from different directions.

The same correction, pointed at a comparison #

A prevalence is rarely wanted for its own sake. What is usually wanted is a comparison — between regions, between years, between exposed and unexposed — and misclassification does something to a comparison that it does not do to a single estimate.

Both groups’ apparent rates are inflated by the same (1 − spec), so the difference of apparent rates is

$$a_1 - a_0 = (p_1 - p_0)(\text{sens} + \text{spec} - 1),$$

with the false-positive term cancelling exactly. So the raw difference is the true difference multiplied by Youden’s index — attenuated by a factor of 0.85 here — and the correction for a difference is a single multiplication that needs no specificity at all, only Youden.

That is a much better situation than the single estimate’s, and it has a sting. The attenuation is towards the null, so a comparison made on apparent rates understates every difference, and a null result is exactly what a misclassified comparison produces when the true difference is small. A study finding no difference between two populations has found either no difference or a difference divided by 0.85, and at a worse test the divisor is larger.

The ratio behaves worse than the difference. Apparent rates of 5.09% and 5.43% have a ratio of 1.07 where the true prevalences are 0.1% and 0.5%, a ratio of 5. Misclassification attenuates a ratio far more violently than a difference, because the shared additive term sits in both the numerator and the denominator and dominates both, and the field that reports prevalence ratios is the field this hurts most.

What is claimed here, and what is not #

Two statements, both counted rather than derived.

The corrected estimate averages the true prevalence, to within a fraction of the truth over twenty thousand samples, at every prevalence the slider takes. Unbiasedness is the estimator’s only claim, and counting it is what separates this from an adjustment chosen to look reasonable.

It is more variable than the raw positive rate, by the predicted factor of one over Youden’s index. That is the cost, and a version that was not more variable would be a version that had not divided by anything — the plausible slip, and the one that would produce an estimator looking better on every summary.

The reading that does not survive is the apparent rate taken as a prevalence. The standard an estimate is held to is that it averages the thing it estimates, and at a true prevalence of one in a thousand the raw positive rate averages 5.09%. A factor of fifty is not a small bias in a reasonable estimator; it is a different quantity.

The positive companion is worth stating beside it. The corrected estimate’s mean over twenty thousand samples is 0.100% against a true 0.100%, which is four significant figures of agreement on a quantity the raw rate misses by a factor of fifty. Unbiasedness is cheap to establish and it is the only property the estimator has; everything else on this page is about what it costs.

Where the apparent rate is the right answer #

Everything above treats the apparent rate as a mistake, and there is one setting where it is exactly what is wanted, which is worth naming so the rule does not get over-applied.

When the decision is about the tested population rather than about the disease. A laboratory planning how many confirmatory tests it will have to run needs the apparent rate: 5.09% of everyone screened will come back positive and will need a second test, and the true prevalence is irrelevant to that budget.

When the test defines the condition. Some conditions are operationally defined by a measurement — a diagnostic threshold on a continuous marker — and then there is no misclassification to correct, because there is no separate truth. The apparent rate is the prevalence, by definition, and applying a correction to it would be correcting towards a quantity that does not exist.

When the same test is used throughout a trend. A series of apparent rates measured with one unchanged assay carries the trend faithfully, attenuated by Youden as the previous section says but otherwise intact. Correcting each point changes the level and the slope by the same factor, and if only the direction is wanted, the correction adds noise and no information.

The distinction across all three is what the number is going to be used for, which is the question a summary’s own units keep raising in an unrelated field. A rate is not right or wrong on its own; it is right for some questions.

Still open: what an interval for a prevalence should look like #

The estimate is a point and the page has argued twice that the point is not the reading. What an interval for it should be is genuinely unsettled, and for a specific reason.

The obvious construction — the corrected estimate plus or minus two of its standard errors — inherits both problems. It can lie entirely below zero, and it is symmetric about an estimate whose sampling distribution is not, because the binomial count behind the apparent rate is skewed near its own boundary.

The pieces of a better answer exist elsewhere in this collection. An exact interval for a proportion handles the binomial’s discreteness, and mapping such an interval through the Rogan–Gladen transformation is a monotone map that preserves coverage exactly. That would give an interval for the prevalence with the right coverage and a lower limit that can still be negative.

Whether the right move is then to truncate the interval at zero — which is defensible in a way that truncating the estimate is not, since the parameter really is non-negative — or to report the untruncated interval and let the reader see that the data are consistent with nothing, is a question about reporting rather than about arithmetic. Both have been done and neither is standard, and the choice changes what a reader concludes from the same evidence.

The third option is to stop reporting a point at all. A prevalence whose estimate is negative half the time is a quantity the data barely constrain, and the honest output is the constraint: an upper bound. “The prevalence is below 0.3% with 95% confidence” is a true, useful, single-sided statement that survives everything on this page, costs nothing to compute, and is what most early surveys of a rare condition should have said.