Every essay — page 25
A proportion's interval near the boundary, and the coin
Wilson's interval for a proportion wobbles a point or two around 95% away from the boundary and has a hole near it: its worst coverage is 83.50% at ten trials and 83.81% at a thousand, at an expected count of 0.1765 each time, because below that count the interval built on one success lies wholly above the truth. The intervals that never fall below 95% pay in average coverage, and the two of them pay for different promises — Clopper–Pearson holds each side under 2.5% and is 10.4% wider than Wilson at thirty trials, Blaker holds only the total and is 4.6% wider. Only an interval that adds a uniform random draw to the count covers exactly 95% at every proportion, at 0.9% more width than Wilson, and it is not used because two analysts with the same data would report different intervals.
What a guaranteed minimum costs
Clopper–Pearson's interval never covers less than 95%, and at thirty trials it averages 97.34% and is 10.4% wider than Wilson's. Blaker's interval keeps the same guarantee, averages 96.31% and is 4.6% wider. The difference is not waste: Clopper–Pearson guarantees each side separately, holding both below 2.5%, and Blaker guarantees only their sum — so at ten trials and a proportion of 0.15 it misses on one side 5.00% of the time.
The coin that makes it exact
Every interval for a proportion either covers less than 95% somewhere or more than 95% on average, because a count is discrete. One construction covers exactly 95% at every proportion: it adds a uniform random draw to the count. At thirty trials it is 0.9% wider than Wilson's interval and narrower than both exact ones — and two analysts with the same data report different intervals, and one study in forty that sees nothing reports an empty one.
The coin that is already there
The randomised interval covers exactly 95% at every proportion because it adds a drawn coin to the count, and a drawn coin is why nobody reports it. A sequence of trials already holds one: given the count, the order the successes arrived in is equally likely to be any of its arrangements whatever the proportion is. Used as the coin, it keeps twenty trials' coverage within 0.12 points of 95% from proportions of 0.2 to 0.8, where mid-p strays by 2.8, and it gives the same interval to every analyst. It fails where the count has few arrangements — a count of none has one — and it holds only while the order is read as recorded: an analyst choosing among sixteen orders lifts a 5% test on sixteen trials to 7.68%.
![Coverage against the expected count of the interval whose coin is read from the arrival times. The randomised interval for a Poisson mean, with its coin read from where in the window the events fell, covers exactly 97.5% at every expected count between 0.025 and ln 40 = 3.689 and exactly 95% past it, when a count of zero is given the exact interval's [0, ln 40]; its worst is 95.00%. Given mid-p's [0, ln 20] instead, it falls to 92.50% just past 3.00. Mid-p itself reaches 91.66% at 3.03; the exact interval never falls below 95.62% and averages well above 95%.](/f/coverage-against-the-expected-count-of-the-interval-whose-coin-is-read-db99b051.svg)
The coin a clock supplies
A count observed over a window comes with the times its events arrived, and given the count those times are uniform whatever the rate is — so they are a coin with a continuum of faces at every count from one up. Used to randomise the interval for a Poisson mean, they leave exactly one count without a coin, the count of zero, and what that count is given decides everything: mid-p's limit of ln 20 leaves a hole at 92.5%, and the exact interval's ln 40 is the shortest limit that leaves none, after which the interval covers exactly 97.5% below an expected count of 3.69 and exactly 95% above it.
Two windows and a ratio
Two counts from two windows give an interval for the ratio of their rates through the conditional binomial: given the total, the first window's count is binomial with a proportion that depends only on the ratio. The arrival times in both windows supply that binomial with a continuous coin at every total from one up, and the interval then covers exactly 95% plus a twentieth of the chance of no events at all. A total of zero can only report the whole of (0, ∞). And the exact conditional test's familiar conservatism is not the zero total's: the zero is the larger cause only while the expected total is below ln 2, and at an expected total of ten the exact interval still covers 97.8% for reasons that have nothing to do with it.
An interval read beside something else
A 95% interval is almost never read alone. Read against a replication's estimate it captures an equal-sized one 83.42% of the time, because both estimates are uncertain, and ten replications of one study miss together: none of them lands outside 31.40% of the time against 16.32% if they were independent. Read against a second interval, two that just touch mark a difference with p = 0.0056 rather than 0.05, standard-error bars that touch mark p = 0.157, and in a figure of ten identical groups some pair of 95% intervals fails to overlap one time in seven. Read against a second verdict, two studies of one effect at 50% power disagree about significance half the time, and when they do the difference between them is significant in 9.75% of cases.
Five times in six
A 95% interval is read as a 95% chance that a replication's estimate will land inside it. With the spread known and a replication of the same size, the chance is 83.42% — five times in six — because both estimates are uncertain. An original that landed two standard errors from the truth captures a replication 48.40% of the time; a replication a tenth the size lands inside 44.54% of the time; and among significant originals from studies with 17% power, 66.94%.
Two intervals that overlap
Two 95% intervals that just touch are read as a difference at the edge of significance. With equal standard errors their difference has p = 0.0056, not 0.05; two intervals can overlap by 58.6% of an arm and still differ at exactly 5%; standard-error bars that just touch mark p = 0.157; and when the two estimates are correlated at 0.8, touching intervals conceal a difference of 6.2 standard errors. Read as a test, non-overlap needs 1.66 times the sample for the same power.
Significant in one, not in the other
Two studies of exactly the same effect, each with 50% power, disagree about significance half the time — and when they do, the test of the difference between them is significant in 9.75% of cases. A p of 0.01 beside a p of 0.20 is a difference with p = 0.36. Among four subgroups sharing one effect, at least one significant and one not happens 87.5% of the time, and the test that would tell a real difference apart needs four times the sample the effect itself needed.
The verdicts watch the threshold, not the gap
Two studies whose effects really differ are read through their verdicts as often as through their difference. Halves of an 80%-power study whose effects differ by the whole effect split 72.87% of the time against 49.99% when they do not — a split is never more than 1.86 times as likely under any difference up to twice the effect. And a halving of the effect in two large studies leaves both significant 97.93% of the time, while the test of the difference finds it 80.74%.
A subgroup inside its own trial
A trial with 80% power, significant overall and not in the fifth of it that is women, has told the reader almost nothing about women: that pattern turns up 57.45% of the time when women have the full effect and 58.26% when they have none. The subgroup is part of the whole, so the two verdicts are correlated, and the one test that answers the question — the subgroup against everyone else — can be recovered from the two printed intervals and nothing more.
Sixteen subgroups and one effect
A trial of four hundred with 80% power tabulates its result by eight binary characteristics — sixteen subgroups — and the treatment's effect is the same for every patient. Among trials significant overall, 93.5% report at least one subgroup where it is not, and the median one reports seven. Some subgroup-against-the-rest interaction reaches 5% in 34.6% of trials — eight independent chances — and when the characteristics overlap as two age thresholds do, 18.6%, about four. The effective count is set by the correlation between the characteristics' split indicators, (2/π) arcsin of their latent correlation, and a Bonferroni reading of the eight holds the rate at 5% whatever the overlap.
One characteristic that really matters
A trial of four hundred tabulates eight characteristics and one of them genuinely changes the treatment's effect — the whole effect on one side, none on the other. Tested alone at 5% the real interaction is found 81.0% of the time; at the Bonferroni threshold that protects the table, 53.6%. It is the table's largest interaction in 82.4% of trials when the characteristics are independent and in 54.4% when they overlap like nested age bands, and in that case another characteristic crosses the Bonferroni line in half of all trials and is the top finding in a quarter. A family-wise threshold counts false rows; it cannot say which of two overlapping rows is the cause.
The pattern a cause leaves on its neighbours
A characteristic that modifies a treatment's effect raises every characteristic that overlaps it, in proportion to the overlap, and fitting that pattern was supposed to say which one is the cause. Fitted with the noise's own correlation it names the largest row on every trial, because the noise draws the same pattern — so the 54.4% at which the largest row is right among nested characteristics is the most any reading of the table can manage, and the honest answer is a set of about five.
What partial pooling does to one group, to the set, and to a ranking
Partial pooling minimises the total squared error across groups, and three uses of its estimates are not the total. One group: with each group's standard error equal to the population's spread, every group truly more than 1.73 widths from the centre — 8.33% of a normal population — is estimated worse than by its own mean, and among eight groups with the spread estimated the truly most extreme one is worse off in 61.6% of datasets; capping each group's shift at one standard error improves the total, the truly extreme group and the observed extreme group at once. The set: posterior means spread 0.707 as widely as the truth and count 0.234% of groups beyond two widths against a true 2.28%. The ranking: in a league table of a hundred groups of unequal size, raw means fill 62.1% of the top ten with small groups, posterior means 13.4%, and no ranking recovers more than 5.47 of the true ten.
A group from the population's own tail
Partial pooling halves the total squared error when a group's own standard error equals the spread between groups. Every group whose true effect sits more than 1.73 population widths from the centre — 8.33% of a perfectly normal population — does worse than it would have with its own mean, and its loss grows without bound. Among eight groups with the spread estimated, the most extreme is worse off in 61.6% of datasets. Capping the shift at one standard error keeps the total at 0.528 of the unpooled error and holds every group under twice it.
Estimates that are too alike
Posterior means give each group its least-error estimate, and as a set they are too alike: with each group's standard error equal to the population's spread, they spread 0.707 as widely as the truth. Beyond two population widths lie 2.28% of the true effects, 7.86% of the groups' own means, and 0.234% of the posterior means — a tenth of the truth. Rescaling the estimates to the right spread counts the tail exactly and costs 17% more squared error; summing each group's posterior chance of being beyond the line counts it without changing any estimate.
A league table of a hundred
A hundred groups with sizes from 4 to 400, and a top ten to publish. Ranked by their own means, small groups fill 62.1% of the top ten against their 36.3% share of the true top ten. Ranked by posterior means they fill 13.4%. The ranking built from each group's chance of being in the top ten recovers 5.47 of the true ten, the best of three and barely half; and the group ranked first could hold any rank from 1 to 31.
The small groups one centre protects
In a league table of a hundred groups where larger groups do better on average, pooling every group towards one centre overstates the small groups by half a population width — 0.501 — because it pulls them towards an average they are not part of. The top ten barely notices. The bottom ten, the list that flags poor performers, holds 3.19 of the true bottom ten when pooled to one centre and 4.28 when the groups are ranked by their own noisy means. Pooling towards a centre that rises with size restores 4.35 and puts the small groups back at the bottom where 80.6% of the true bottom ten sit.
The relation the table has to estimate
Pooling a league table's groups towards a centre that rises with size repaired the small groups' bias — when the size relation was known. Fitted from the table itself, a straight line in log size loses almost nothing: the small groups' error is 0.750 against 0.719 with the true relation. When the relation bends below the median size, the fitted line extrapolates the large groups' slope into the region where it is barely observed and misplaces the small groups by −0.173 if the relation flattens and +0.264 if it steepens — smaller than one centre's error, of either sign. A quadratic in log size removes nearly all of it, −0.025 and +0.014, and costs 0.025 in error when the relation was straight.
The interval a small group is given
A league table's 95% interval for a small group, pooled towards a size relation fitted from the table, covers 92.8% of the time when the relation is straight and 83.1% when it steepens below the median — and 77.4% for the groups of eight patients or fewer. Adding the fitted centre's own uncertainty, weighted by how much the group borrows from it, brings the line to 88.4% and a fitted quadratic to 93.7%, and the quadratic's interval covers 94.6% of the smallest groups at every shape tried. One centre, plugged in, covers the smallest groups 35.2% of the time when the relation steepens, and no added variance repairs a centre in the wrong place.
A ranking with its uncertainty
A league table's reader wants a rank, and a rank interval inherits a fitted size relation's errors differently from an effect interval. When the relation steepens below the median size, a fitted line's 95% interval for a small group's effect covers 83.1% of the time; its 90% interval for the same group's rank covers 95.6%, because the line's error lifts the small groups into the crowded middle of the table, where rank intervals are wide. The error surfaces instead in the large groups' ranks, 87.6%, and in width: 36.9 places against the 27.3 the true relation needs. A quadratic's uncertainty has to be drawn once per table, since it moves every small group together — added group by group it over-covers at 92.8%.
A centre chosen by its own table
A league table that picks its centre's shape from its own data — a line unless a curvature test says otherwise — picks the line on 85.5% of tables whose relation flattens below the median, because the bend lives where the groups are smallest and noisiest. Its bottom ten then flags like the line's: 2.61 innocent small groups a table against 2.19 for a quadratic told the shape. Always fitting the quadratic costs 0.07 of a correctly flagged group when the relation is straight and matches the told answer when it bends, and a flag set by probability carries the choice through a mixture of the two.
What a sample-size calculation was given
A sample-size calculation takes a standard deviation, an effect and an outcome as given, and each is usually a choice or an estimate. Sized from a pilot of ten's standard deviation, 55.9% of trials have less than their planned 80% power and 11.1% less than half, although the planned sample is right on average; sizing from the pilot's 80% upper confidence limit leaves 19.8% short at 1.65 times the patients. With the effect believed to be half a standard deviation give or take a quarter, a trial of sixty-four per arm succeeds 69.2% of the time, and no trial of any size can succeed more often than the planners believe the effect is positive. Cutting the outcome at a threshold keeps at most 63.7% of its information, and a responder threshold one standard deviation out needs twice the patients.
The spread a pilot supplies
A trial sized for 80% power from a pilot's standard deviation is sized from an estimate that is too small more often than not. With a pilot of ten, 55.9% of the trials it sizes have less than 80% power and 11.1% less than 50%, although the planned sample is right on average. Sizing from the pilot's 80% upper confidence limit instead leaves 19.8% short, at 1.65 times the sample; from its 90% limit, 10.0% short at 2.12 times.
The chance a trial succeeds
A trial of sixty-four per arm has 80% power at an effect of half a standard deviation. If the effect is only believed to be about half a standard deviation, give or take a quarter, the chance the trial reaches significance is 69.2%; give or take a half, 61.4%. Reaching 80% then takes 113 per arm, or 1,268 — and when the belief is uncertain by three quarters of a standard deviation no number of patients reaches 80%, because the chance can never exceed the 74.8% probability that the effect is positive at all.
An outcome cut in two
Replacing a measured outcome with whether it crossed a threshold keeps 63.7% of the information when the cut is at the mean, 34.2% at the top tenth and 13.1% two standard deviations out. A trial that needs 63 patients per arm on the measured outcome needs 102 cut at the mean and 185 cut at one and a half standard deviations. The responder rates that result read as a share of patients who respond — 50.0% against 69.1% — when every patient moved by the same amount; and a cut chosen after looking turns a 5% test into a 17.7% one.