Two groups, one treatment, and both readings of the same numbers

The treatment wins in group A (93.0% against 87.0%) and in group B (73.0% against 69.0%), and loses overall (78.1% against 82.6%). Nothing here is a trick; the allocation differs between the groups.

Four hundred studies a prevalence, two hundred resamples each. Median interval and true best threshold, as false-positive rates: 0.05: 12.3% to 24.4%, truth 18.5%, holding it in 94.0% of studies, crossing the jump in 0.0%; 0.1: 21.6% to 47.6%, truth 33.9%, holding it in 92.3% of studies, crossing the jump in 15.3%; 0.2: 36.7% to 100.0%, truth 63.2%, holding it in 91.5% of studies, crossing the jump in 80.0%; 0.26: 45.3% to 100.0%, truth 100.0%, holding it in 92.5% of studies, crossing the jump in 91.0%; 0.35: 57.0% to 100.0%, truth 100.0%, holding it in 98.3% of studies, crossing the jump in 91.3%.

Average excess harm per hundred people, misses costing a hundred false alarms. 0.05: lower end 1.91, upper end 1.31, the study's own choice 0.32; 0.1: lower end 4.52, upper end 7.29, the study's own choice 0.94; 0.2: lower end 10.77, upper end 5.34, the study's own choice 2.76; 0.26: lower end 15.29, upper end 0.11, the study's own choice 1.97; 0.35: lower end 24.38, upper end 0.13, the study's own choice 3.40.

400 studies a prevalence. the study's own choice: 0.32, 0.94, 2.76, 1.97, 3.40; harm averaged over resamples: 0.33, 0.94, 3.00, 1.96, 3.01; worst excess over resamples: 0.39, 2.20, 2.66, 1.30, 3.53 at prevalences 0.05, 0.1, 0.2, 0.26, 0.35.

Trials of eighty whose treatment helps every patient by six points, twenty baseline variables tracking the risk with correlations from 0.95 to 0.10; 4,000 trials. Searching the strongest 1, 2, 5, 10, 20: every cut, 23.4%, 30.7%, 41.0%, 46.9%, 49.0%; the deciles, 13.2%, 19.4%, 27.9%, 33.3%, 36.4%; the median, 4.4%, 6.9%, 11.3%, 14.4%, 16.2%.

One seeded trial of eighty; the whole trial's difference is −2.5 points. 8 of the twenty variables reverse at some cut: variable 1 (correlation 0.95 with the risk) at 11 cuts, variable 2 (correlation 0.91 with the risk) at 2 cuts, variable 7 (correlation 0.68 with the risk) at 1 cut, variable 8 (correlation 0.64 with the risk) at 2 cuts, variable 10 (correlation 0.55 with the risk) at 6 cuts, variable 12 (correlation 0.46 with the risk) at 1 cut, variable 13 (correlation 0.41 with the risk) at 1 cut, variable 15 (correlation 0.32 with the risk) at 6 cuts.

4,000 trials of eighty. Scanned over every cut, from the strongest variable to the weakest: 23.4%, 22.1%, 20.0%, 19.9%, 17.3%, 17.7%, 14.6%, 14.2%, 12.8%, 11.9%, 10.3%, 8.6%, 8.6%, 7.0%, 6.1%, 5.7%, 4.6%, 4.2%, 4.3%, 3.4%; at the median: 4.4%, 3.5%, 3.5%, 2.6%, 1.7%, 1.8%, 1.5%, 1.6%, 1.4%, 0.9%, 1.0%, 0.9%, 0.6%, 0.5%, 0.4%, 0.4%, 0.3%, 0.1%, 0.1%, 0.2%.

Every cut in the middle 80% of the strongest 1, 2, 5, 10, 20 variables. Eighty patients: 23.4%, worth 9.5 tables; 30.7%, worth 13.6 tables; 41.0%, worth 23.2 tables; 46.9%, worth 34.8 tables; 49.0%, worth 59.5 tables. 640 patients: 11.8%, worth 7.1; 14.5%, worth 8.8; 17.9%, worth 13.4; 20.5%, worth 21.5; 20.9%, worth 37.4.

Root mean squared error, over 400 studies of a thousand readings at each retest size, of the share of the top one per cent's lead a second reading keeps: Tweedie's correction with a kernel log-slope and the error variance estimated from the retested people (solid), and the linear rule at the reliability those people estimate (dashed). Laplace parent: 0.121, 0.108, 0.084, 0.086, 0.084, 0.078 against 0.343, 0.259, 0.201, 0.183, 0.171, 0.164. T, four degrees parent: 0.133, 0.103, 0.086, 0.084, 0.080, 0.078 against 0.358, 0.278, 0.235, 0.212, 0.205, 0.187. Uniform parent: 0.247, 0.192, 0.156, 0.140, 0.118, 0.117 against 0.260, 0.208, 0.185, 0.166, 0.160, 0.157 — at 10, 20, 50, 100, 200, 1000 retested.

Share of 400 studies on which Tweedie's estimated correction is closer to the share kept than the linear rule at the estimated reliability. Normal errors (solid): Laplace 67%, 73%, 81%, 85%, 88%, 94%; t, four degrees 72%, 77%, 86%, 89%, 93%, 96%; uniform 59%, 62%, 71%, 70%, 78%, 81%. Laplace errors (dashed): Laplace 42%, 31%, 24%, 15%, 12%, 6%; t, four degrees 43%, 37%, 30%, 22%, 18%, 13%; uniform 33%, 20%, 18%, 18%, 13%, 13% — at 10, 20, 50, 100, 200, 1000 retested.

For each parent and error law, at a thousand readings with the whole sample retested: the share of the top one per cent's lead a second reading keeps, Tweedie's correction with the true error variance averaged over 400 studies, and the linear rule's estimated reliability. Laplace parent, normal errors: kept 0.761, Tweedie 0.782, rule 0.599; Laplace parent, Laplace errors: kept 0.643, Tweedie 0.807, rule 0.600; t, four degrees parent, normal errors: kept 0.784, Tweedie 0.802, rule 0.600; t, four degrees parent, Laplace errors: kept 0.666, Tweedie 0.830, rule 0.599; uniform parent, normal errors: kept 0.443, Tweedie 0.526, rule 0.599; uniform parent, Laplace errors: kept 0.328, Tweedie 0.702, rule 0.599.

The share of studies whose retest differences have an excess kurtosis above its 95th percentile under normal errors, when the errors are Laplace: 11.8% at 10, 19.8% at 20, 34.9% at 50, 57.4% at 100, 80.3% at 200, 100.0% at 1000, over 4,000 studies at each size.

The treatment wins in group A (93.0% against 87.0%) and in group B (73.0% against 69.0%), and loses overall (78.1% against 82.6%). Nothing here is a trick; the allocation differs between the groups.

The per-group success rates are held fixed; only the split of each group between treatment and control changes. 32% of the allocations reverse, and the worst reverses by 13.1 percentage points.

At a prevalence of one in a thousand, 98 of every hundred positives are false. At one in 10, 33 are. The test has not changed.

9 people have it and test positive. 50 do not have it and test positive anyway. So of the 59 positive results, 15% are right — and that is with a test most people would call accurate.

Pick the worst 15% on the first measurement and their average rises by 0.78 on the second. Pick the best and theirs falls by 0.48. No treatment was given to anybody.

600 units in two pre-existing groups whose true means are 1.00 apart, read once at baseline and once at follow-up, with no change for anybody. The two groups' mean changes are −0.075 and −0.032, so the change-score analysis reports a group difference of 0.043. The regression of follow-up on baseline and group reports 0.409, against a closed form of (1 − λ) × 1.00 = 0.400: at any one baseline reading the two groups' lines sit that far apart, because each group's units regress towards their own group's mean. The pooled slope in this sample is 0.614, the baseline's reliability.

2000 people read once at screening and once at follow-up, with a test–retest correlation of 0.6 and no treatment. The 215 above a cut at the top ten per cent of one reading (1.282 standard deviations) are enrolled. Their mean screening reading is 1.744 and their mean follow-up reading 0.982, a fall of 0.762 ± 0.055 with nothing done to anyone. The closed form for the fall is (1 − ρ) times the truncated-normal mean, (1 − 0.6) × 1.755 = 0.702.

The expected second reading divided by the first, for readings from 0.5 to 4 standard deviations, where the true scores come from a normal, a Laplace, a t with four degrees of freedom or a uniform parent, every one with variance 0.6, and every reading adds normal error of variance 0.4. All four put the two readings at a correlation of exactly 0.6. For the normal parent the share kept is 0.6 at every reading. At a reading of 2 the Laplace parent keeps 0.644, the t parent 0.604 and the uniform 0.507; at a reading of 4 they keep 0.817, 0.866 and 0.301. The curves are posterior means by integration, checked against Tweedie's formula to 7e-8 and against closed forms for three of the parents to 9e-10.

Each thin line is one study of 1000 readings from the t, four degrees parent: Tweedie's formula with the log-density's slope estimated by a degree-5 log-spline, drawn up to that study's largest reading. The thick line is the share kept by integration over the true score, and the dashed line the correlation, 0.6. The top ten readings of the median study begin at 2.45. Over 400 studies the corrected share kept by the top one per cent averages 0.8190, with a spread of 0.0806, against 0.7842 by integration; a Gaussian kernel averages 0.7914 with a spread of 0.0853.

Simple randomisation against randomisation stratified by group, 4,000 trials at each size. The simple design reverses on 3.40% of trials at its worst size and 0.50% at 1280 units; the stratified design reverses on none of them, at any size.

Two events on the same trials. The overall difference sitting outside the range of the two group differences happens on about 33% of trials and does not fall with the trial's size. The reversal falls from 3.23% to 0.50%. Only the second is evidence of anything.

Five strata with baseline risks from 5% to 85%, a treatment allocated by a coin in each, and a conditional odds ratio of exactly 2.5 throughout. The marginal odds ratio is 1.789. Nothing is confounded; the odds ratio is simply not a weighted average of odds ratios.

Five strata centred on a 45% baseline risk, spread by the half-width on the axis, with a conditional odds ratio of 2.5 in each. The marginal odds ratio falls from 2.500 to 1.832 — 45% of the way to no effect at all, with no confounding introduced at any point.

The direct effect stays at -0.109 throughout, because it is defined with the mediator held fixed. The total effect runs from -0.218 to 0.027 and crosses zero at 1.5. On 15% of the sweep the two have opposite signs, and both are correct answers.

A 90% sensitive, 95% specific test. One positive gives 1.77%. Two independent positives give 24.49%, which is what multiplying the likelihood ratios says. At a correlation of 0.1 between the tests' errors it is 10.16%, and at 0.5 it is 3.16%.

The published pair — 90% sensitive, 95% specific — is the open mark. With a miss costing 100 times a false alarm and a prevalence of 0.10%, the threshold that minimises expected cost sits at 74.7% sensitivity and 98.81% specificity, with a predictive value of 5.9%.

One test, one pair of costs, and a threshold that runs from 3.05 to -0.12 standard deviations of the score. At one in ten thousand the best test detects 45% of cases; at one in two it detects 99.9%. The published 90/95 is right for one population.

The positive rate reads 5.09%, which is 50.9 times the truth. The Rogan-Gladen correction averages 0.100% — unbiased — with a standard deviation of 0.820 points against the positive rate's 0.695, and it comes out negative on 48.6% of samples.

Each row is one survey. The thin bar is the corrected estimate ± 1.96 standard errors; the heavy bar is the exact interval for the positive rate carried through the correction; the tick is the one-sided 95% upper limit. 30 of the thirty printed intervals reach below zero and 0 lie entirely below it.

The printed interval covers within two points of 95% at every prevalence from 0.05% to 20%. At a true prevalence of 0.1% it reaches below zero in 98.0% of surveys and lies entirely below zero in 3.4%; the typical one-sided upper limit is 1.64%, 16.4 times the truth.

With a 90%-sensitive test, bounding a prevalence of 0.1% below 0.2% takes 184,521 people at 95% specificity, 39,957 at 99%, 9,110 at 99.9% and 5,083 with no false positives at all. A prevalence of 1% needs 2,362 at 95%.

A miss costs a hundred false alarms. With diseased scores as spread as healthy ones the best threshold flags more healthy people smoothly as the prevalence rises. With them twice as spread it flags 81.1% just below a prevalence of 25.7% and everyone just above it; three times as spread, 45.7% and then everyone at 13.0%.

At a prevalence of 25.69% the harm curve's interior minimum, at a cut of −0.88, costs exactly what flagging everyone costs, which is the curve's value at the far left. A little below that prevalence the interior cut is better; a little above, flagging everyone is.

At equal spreads the likelihood ratio at the threshold falls all the way as the threshold is lowered. With the diseased scores twice as spread it bottoms out where 92.0% of healthy people are flagged and rises beyond; three times as spread, at 75.4%. A cut on the rising stretch flags more healthy people for a better ratio, which no harm-minimising single threshold would do.

Every patient's hazard ratio is 0.5. With two risk groups — half at 0.1 a year, half at 1 — the ratio of the arms' hazards rises from 0.5 to 0.929 at 3.5 years and falls back towards 0.5. With a gamma frailty of variance 1 it rises steadily, to 0.857 at ten years.

Both arms start half high-risk. The control arm loses its high-risk patients faster, so at one year 28.9% of its survivors are high-risk against 38.9% in the treated arm, and at three years 6.3% against 20.6%. The arms compared at any moment after the start are no longer alike.

The same trial analysed at different follow-ups. Unadjusted, with two risk groups, the Cox hazard ratio is 0.524 at six months, 0.635 at three years and 0.673 at ten; with a gamma frailty, 0.527, 0.598 and 0.655. Stratified by risk group it is 0.500 throughout.

The treatment helps every patient equally. On one nominated, strongly prognostic variable the reversal appears in 4.1% of trials. Tabulating 2, 5, 10, 20, 50, 100 variables of mixed prognostic value, some variable reverses in 4.1%, 6.5%, 10.8%, 16.1%, 26.1%, 34.1% of trials; stratifying the randomisation on the most prognostic one gives 0.2%, 2.9%, 7.7%, 12.2%, 23.4%, 31.9%.

A variable reverses only if it separates patients of different risk, so the rate climbs with its correlation with the risk: 0.13% at 0.10 and 4.07% at 0.95. Summed over the twenty the rates come to 26.7%; the chance that at least one reverses is 16.1%.

With twenty variables tabulated, some variable reverses in 17.5%, 16.1%, 15.4%, 12.6%, 6.2% of trials of 40, 80, 160, 320, 640 patients; the nominated, most prognostic variable in 4.0%, 4.1%, 4.6%, 3.5%, 1.6%.

The best threshold flags 100.0% of healthy people. Reading the study's own operating curve, 7.7% of studies recommend flagging nearly everyone, and the recommended thresholds cost 15.4% more than the best on average; fitting a normal curve to each group's scores first, 47.4% do, at an average excess of 2.5%.

The true curve still climbs from 97.60% at 40% of healthy people flagged to 100% only as everyone is flagged. The study's curve reaches 100% as soon as its lowest diseased score is passed, at 67.3% of its healthy people flagged, and is flat beyond: it cannot see what flagging the rest would buy.

From the study's own curve with three hundred healthy people the excess harm is 81.0% at 25 diseased, 15.4% at 100 and 2.2% at 1,000; with fifteen hundred healthy, 15.1% at 100. From a fitted binormal, 9.3%, 2.5% and 0.5%.

The treatment helps every patient equally. Cut at its median, the variable's table reverses in 4.4% of trials; trying 2, 3, 5, 9 cuts, in 6.4%, 8.7%, 10.8%, 13.2%. Independent chances would give 33.3% at nine.

Cut at the median the table reverses in 4.4% of trials; at the thirtieth and seventieth percentiles 3.2% and 4.0%; at the first and ninth deciles 0.7% and 0.4%.

With twenty continuous variables cut at their medians, some table reverses in 16.2% of trials; cut at five points each, in 32.2%. The hundred tables of the second amount to about 31.2 independent chances at the average single table's rate.

The treatment helps every patient by the same six points. At 40, 80, 160, 320, 640 patients: some cut in the middle 80% reverses in 18.8%, 23.4%, 24.3%, 19.2%, 10.6% of trials; some decile in 12.2%, 13.2%, 14.1%, 11.0%, 6.2%; the median in 3.9%, 4.4%, 4.2%, 3.1%, 1.7%.

The whole trial's difference is -0.079. As the cut slides from the 10th to the 90th percentile, the subgroups below and above it both show the treatment helping at 2 cut positions, between 28.7% and 30.0% of patients below; at no decile do they.

Over the 23.4% of trials whose scan reverses somewhere, the reversing cut positions by where they fall. In 5.3% of those trials every reversing cut lies outside the middle 60% of the variable; on average the reversing cuts form 2.62 separate runs.

At the average decile's own reversal rate. The scan of the middle 80% is worth 9.0, 9.5, 8.5, 7.5, 7.2 independent cuts at 40, 80, 160, 320, 640 patients, while trying 33, 65, 129, 257, 513 positions; the nine deciles are worth 5.6, 5.1, 4.7, 4.1, 4.1.

The variable's correlation with the underlying risk at 0.95, 0.8, 0.6, 0.4, 0.2: some cut in the middle 80% reverses in 23.4%, 18.3%, 14.0%, 8.5%, 4.5% of trials; some decile in 13.2%, 10.1%, 7.2%, 3.8%, 1.7%; the median in 4.4%, 2.6%, 1.3%, 0.7%, 0.3%.

A thousand studies of a hundred diseased and three hundred healthy people, prevalence 0.26, a miss costing a hundred false alarms. The best threshold flags 7.6% of healthy people. Excess harm per hundred people, and the share recommending that everyone be flagged: the study's own curve 16.26 (0.0%); a binormal fit 67.08 (89.7%); a two-normal mixture fit 22.71 (2.3%); binormal if normality passes 16.26 (0.0%).

Healthy scores are standard normal. Diseased scores are normal with mean 4.21 and standard deviation 2; or a mixture of nine in ten normal with mean 3.54 and one in ten normal with mean 1, each of standard deviation 1; or 3.43 plus twice an exponential draw less one, which has no score below 1.43. All three are 90% sensitive at the cut 1.645 where healthy scores are 95% specific.

A thousand studies at each prevalence. the study's own curve: 4.86, 12.24, 21.88; a binormal fit: 2.10, 15.55, 24.04; a two-normal mixture fit: 1.54, 7.38, 8.58; binormal if normality passes: 4.16, 12.24, 18.46 at prevalences 0.1, 0.26, 0.4.

A straightness test on the diseased scores at 5%, 2,000 studies at each size. At 25, 50, 100, 200 diseased people: binormal, twice the spread 5.5%, 5.8%, 5.7%, 5.1%; one in ten mild cases 19.2%, 34.6%, 60.4%, 87.6%; no lower tail 90.3%, 99.7%, 100.0%, 100.0%.