Every essay — page 24
Two tests, a threshold, and the rate they are read against
A predictive value is computed from a sensitivity, a specificity and a prevalence, and each of the three is softer than it is quoted as being. Two positives from a 90/95 test on a one-in-a-thousand condition are worth 24.49% if the tests' errors are independent, 10.16% at a correlation of 0.1, and 3.16% at 0.5 — barely more than the 1.77% one positive was worth. The sensitivity and specificity are not two properties of a test but one property read at a threshold somebody chose, and the threshold that minimises harm moves by three standard deviations of the score across the prevalences such a test is used at. The prevalence itself is usually estimated from the same test's positive rate, which at one in a thousand reads 5.09%.
The second test that is not a second opinion
Two positives from a 90/95 test on a one-in-a-thousand condition give a 24.49% chance of disease if the tests are independent. At a correlation of 0.1 between their errors it is 10.16%, and at 0.5 it is 3.16% — barely more than the 1.77% one positive was worth.
The test is a point somebody chose
A test reported as 90% sensitive and 95% specific is not two properties of a test. It is one property read at a threshold, and the threshold that minimises harm runs from 3.05 standard deviations of the score at a prevalence of one in ten thousand to −0.12 at one in two — 45% of cases detected at one end and 99.9% at the other.
The prevalence the test has to estimate
Every predictive value takes a prevalence as given, and the prevalence is usually estimated from the same test's positive rate. At a true prevalence of one in a thousand that rate reads 5.09% — fifty times the truth — and the correction that inverts it is unbiased, 18% more variable, and negative on 48.6% of samples of a thousand.
An upper limit is the finding
A survey of a thousand people with a 90%-sensitive, 95%-specific test, estimating a prevalence of one in a thousand, prints a corrected 95% interval that covers the truth 95.17% of the time — and reaches below zero in 97.95% of surveys and lies entirely below zero in 3.35%. Coverage is not the problem. The problem is what the interval can say, and the honest summary of a typical survey is one number: the prevalence is below 1.64%. Bringing that limit down to twice the truth takes 184,521 people at this specificity and 5,083 with a test that never gives a false positive.
A threshold that jumps
When a test's scores are equally spread in the healthy and the diseased, the harm-minimising threshold slides smoothly as the prevalence changes. Make the diseased scores twice as spread — the ordinary shape of a group that mixes mild and severe cases — and keep the published 90/95 pair, and the best threshold flags 81.1% of healthy people at a prevalence just under 25.7% and every healthy person just above it. At three times the spread it jumps from 45.7% to everyone at 13.0%. A threshold that jumps cannot be given as a formula; it has to be given with the second-best point beside it.
The threshold a validation study can see
When diseased scores are twice as spread as healthy ones, the harm-minimising threshold jumps near a prevalence of 26% from an interior cut to flagging everyone, because the true operating curve keeps climbing in a tail where few diseased people score. A validation study of a hundred diseased and three hundred healthy cannot see that tail: its own curve reaches 100% sensitivity at its lowest diseased score and goes flat. Just past the jump, only 7.7% of such studies recommend flagging everyone, and their recommended thresholds cause 15.4% more harm than the best on average. Fitting a normal curve to each group first sees the tail: 47.4% of studies then recommend flagging everyone and the excess harm falls to 2.5%. More healthy people in the study change nothing; more diseased people do.
A test whose tail is not normal
A validation study that fits a normal curve to each group saw the tail where a threshold jumps — when the diseased scores really were normal. Give the same published test a diseased group with no lower tail at all, and the binormal fit invents one: 89.7% of studies recommend flagging everyone, adding 67.08 false alarms' worth of harm per hundred people to a best threshold that costs 5.61. Give it a lump of mild cases instead, and the binormal fit misses the tail that is there and does worse than the study's own curve. A normality test on the hundred diseased scores sees the missing tail every time and the lump 60.4% of the time, and a study that fits the binormal model only when normality is not rejected is never the worst of the four.
A threshold reported with its interval
Resample a validation study of a hundred diseased and three hundred healthy people, refit the binormal model each time, and read off the threshold each resample recommends. Far from the prevalence at which the best threshold jumps to flagging everyone, the interval is tight — flagging 12% to 24% of healthy people at a prevalence of 5%. Near the jump, the median study's interval runs from flagging under half of the healthy to flagging everyone, and 91% of intervals cross the jump. The two ends differ by fifteen false alarms' worth of harm per hundred people, and no rule for choosing inside the interval beats the study's own choice at every prevalence.
What a diagnostic plot is showing
A quantile plot is read by eye against a band drawn for one point at a time, and a genuinely normal sample of forty has forty chances to leave it: 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it. The quantity is wrong as well as the width — a t interval needs the sampling distribution of the mean to be normal rather than the data, and the source that leaves the band on every sample of forty covers 94.80% while one that leaves it on 57% covers 95.73%. And what is plotted are not the errors: a design whose leverages run from 0.045 to 0.663 gives residuals whose spreads differ by a factor of 1.68 with the model exactly right.
The band the eye was standing in for
The confidence band software draws on a quantile plot holds each point at 95%, and a genuinely normal sample of forty has forty chances to leave it — so 45.0% of them do. The band a reader is actually using is that one widened by a factor of 1.502, and nothing draws it.
The plot is about the wrong quantity
A t interval needs the sampling distribution of the mean to be normal, not the data. A two-lump source leaves its quantile band on 100% of samples of forty and its interval covers 94.80%; a t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of five sources.
Residuals are not the errors
A residual's standard deviation is σ√(1 − hᵢᵢ), so a design whose leverages run from 0.045 to 0.663 produces residuals whose spreads differ by a factor of 1.68 with the model exactly right. On the samples where the high-leverage point really did have the largest error, a raw residual plot shows it as the largest on 0.0% of them.
Three runs at the end of the line
With one run at the top of a regression's range, a bad observation there and a line that bends there produce data with exactly the same distribution, so no residual, influence measure or test can say which happened. Move six of twelve runs to the ends, three at each, and a discrepancy of four σ is named correctly as a bad run 84.1% of the time and as a bend 91.6% — while the slope's standard error falls from 0.0836 to 0.0709. The design, not the diagnostic, decides whether the question has an answer.
A number for the shape
Five one-number summaries of a quantile plot's departure from its line, each a correct 5% test on forty normal observations, and no two agree about what matters. Straightness catches a skewed source 83% of the time and light tails 31%; kurtosis catches light tails 72% and skewness 0.2%. Reading all five and reporting whichever looks bad rejects 13.9% of genuinely normal samples. Correcting that search to 5% costs almost half the power against the departure it would have caught best — and a line-up of twenty panels gets the same correction for nothing.
Residuals that look too normal
The quantile plot most often read is of a regression's residuals, and residuals look more normal than the errors behind them. Each residual is a mix of all the errors, and the mix moves towards a bell as the fit takes more columns. On forty rows with skewed errors, the straightness test flags 82.9% of samples of the errors themselves, 79.1% of the residuals of a line, 71.4% with five columns, 51.2% with ten and 23.0% with twenty — far below the 49.5% that twenty errors would give, so the fit costs more than the observations it uses. Light tails vanish faster: 73.0% on the errors, 4.7% on twenty-column residuals. The band's level holds at every design, and one point with leverage 0.73 costs almost nothing; what costs is the number of columns.
Residuals made independent
Recursive residuals — each row predicted from the fit to the rows before it — are exactly independent normals under the null for any design, so a quantile plot of them has an exact band. They see a skewed error worse than the ordinary residuals do, not better: 25.9% against 51.2% at ten columns on forty rows, 9.1% against 23.0% at twenty. BLUS residuals, the independent set closest to the errors, reach 38.3% and 16.2%. Every such set is a rotation of the ordinary residuals, so none holds information they lack, and independence is bought by mixing some residuals almost entirely out of other rows' errors — the least pure recursive residual at twenty columns is 0.4% its own.
Residuals of a model the data chose
Five columns picked by a forward search from twenty candidates that have nothing to do with the outcome leave residuals that show a skewed error 36.8% of the time, where five columns fixed in advance show it 65.4% — the power of a fit of 13.4 fixed columns. Against light-tailed errors the chosen fit's plot is blind, 8.5% against 34.5%. Simulating the search on normal errors gives a band that holds 5%, which the ordinary band already nearly did; the level was never what broke. A reader told the final model has five columns is reading a plot with the power of thirteen.
Past the first term of the normal approximation
The Berry–Esseen bound guarantees how far a standardised sum can be from the normal, and on an exponential source it is 8.62 times the real worst error at every sample size — a constant set by a fair coin, whose error is a half-step, and at a hundred draws larger than the 2.5% tail it would be asked to certify. One Edgeworth term takes the normal's error at two standard deviations from 38% to 8% on ten draws, and turns negative in the short tail at every sample size, beyond a threshold that recedes only as the sixth root of n. The saddlepoint approximation is built at the threshold rather than at the mean, and reads a tail six standard deviations out to within 0.19% on five draws and within 2.2% on one.
A bound written for a coin
The Berry–Esseen theorem guarantees how far a standardised sum can be from the normal, and the guarantee is true. On an exponential source it is 8.62 times the real worst error at every sample size, the worst error sits at the centre rather than in a tail, and at a hundred draws the bound is larger than the 2.5% tail it would be asked to vouch for.
A correction that goes below zero
One Edgeworth term takes the normal approximation's error at two standard deviations from 38% to 8% on ten exponential draws, and stretches the range within 10% of the truth from 1.66 to 3.09 standard deviations at a hundred. It also turns negative in the short tail at every sample size — past 3.13 standard deviations at a hundred draws and 9.83 at a hundred thousand — because the region recedes only as the sixth root of n.
An approximation built at the threshold
The saddlepoint approximation reads the tail of a sum of five exponential draws to within 0.19% six standard deviations out, where the normal is short by a factor of more than sixty thousand. It is within 2.2% out to ten standard deviations on a single draw, where there is nothing to average, and within 1.1% on a binomial whose expected count is one. It works because it is built where the tail is read rather than at the mean.
A tail the sample never saw
Given the source, the saddlepoint reads the tail of a sum of thirty exponential draws to within a fifth of a per cent six standard deviations out. Given only a sample of thirty from that source, the same tilt reads it at a median of 0.061 of the truth, and nine samples in ten read it low. The error has moved from the approximation to the sample: thirty draws put the skewness at a median of 1.30 against a true 2, and hold nothing past their largest value.
The same correction, inverted
One skewness term added to a tail probability keeps a sum of a hundred exponential draws within 10% of the truth out to 3.09 standard deviations. The same term used to move the critical value instead keeps the test's actual size within 10% of nominal out to 5.10 — a level of about one in six million. In the short tail it has a flaw of its own: below a level of about one in a million at ten draws the corrected critical value turns back, and a test asked for a level of 10⁻⁷ rejects 1,149 times too often.
The correction a t test would use
On thirty exponential draws the one skewness term that held a sum's critical value within 10% out to five standard deviations nearly triples a t test's 1% size when the source's skewness is known, and its upper critical value turns back — at a level of 0.89% on ten draws, where a test asked for one in a thousand rejects 121 times too often. Hall's monotone cubic removes the turn and gains a floor. Built from the sample's own skewness it holds the short side within 20% from 10% down to one in a hundred thousand, and leaves the long side, the one a safety limit reads, at twice its level: the samples that fail are the ones that hide their skewness.
A tilt for the ratio
A t statistic is a function of two means, so its saddlepoint tilts the joint law of each draw and its square. On an exponential source every such tilt is a truncated normal, and none has a coefficient of variation above one — the exponential is the edge of its own family. Integrated over the samples a tilt can reach, the saddlepoint reads a t statistic's tails on thirty draws within 2.5% from 10% to one in a million on the long side, with nothing fitted. The samples it cannot reach are a third of all samples, they sit at the centre of the statistic rather than in its tails, and a larger sample has more of them.
A proportion's interval near the boundary, and the coin
Wilson's interval for a proportion wobbles a point or two around 95% away from the boundary and has a hole near it: its worst coverage is 83.50% at ten trials and 83.81% at a thousand, at an expected count of 0.1765 each time, because below that count the interval built on one success lies wholly above the truth. The intervals that never fall below 95% pay in average coverage, and the two of them pay for different promises — Clopper–Pearson holds each side under 2.5% and is 10.4% wider than Wilson at thirty trials, Blaker holds only the total and is 4.6% wider. Only an interval that adds a uniform random draw to the count covers exactly 95% at every proportion, at 0.9% more width than Wilson, and it is not used because two analysts with the same data would report different intervals.
