Field

Intervals, counted

An interval that claims 95% is making a checkable statement about a procedure. Build every possible sample and count. The interval taught first fails, the failure is worst where proportions are most often reported, and more data does not monotonically help.
0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted

What the 95% refers to

An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time.

0.8000.8500.9000.950255075100sample sizecoverage at a true proportion of 0.15n = 19n = 20WilsonWaldexact coverage at every n from 10 to 120more data is not automatically better here

More data is not monotonically better

Coverage of an interval for a proportion does not improve smoothly as the sample grows. It oscillates, and there are larger samples that cover materially worse than smaller ones — a sample of twenty covers twelve points worse than a sample of nineteen.

the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval

Twenty intervals and one expected miss

The 95% belongs to the procedure, not to the interval in front of you. Twenty intervals from twenty samples make that visible in a way no definition does, and the one that misses is not a mistake.

Wald0.241covers 94.2%Wilson0.247covers 96.5%Agresti–Coull0.259covers 96.5%Clopper–Pearson0.273covers 98.3%expected width, and what it buysorange: fails its nominal levelthe shortest interval is the one that misses

The shortest interval is the one that misses

Four intervals for the same data, with their widths and their coverage measured together. The narrowest is the one that fails its stated level, which is exactly why it looks the most appealing.

the sample mean93.6%nominal 95%the sample maximum0.0%nominal 95%3,000 samples of 40the same procedure, two statisticsresampling cannot see past the data

Where the bootstrap lies

Resampling is the most generally useful trick in the subject and it has a failure mode that is easy to state: it cannot see past the data. For a statistic that lives at the edge of the sample, coverage collapses from 95% to almost nothing.

00.1000.2000.3000.400-4-2024standard errors from the meandensityt 2.57z 1.96solid: t · dashed: normal31% wider at 5 df

The correction for not knowing the spread

The t distribution exists because the standard deviation is estimated rather than known. At eight observations, using the normal instead makes every interval 12% too short — and the coverage that follows can be measured rather than argued about.

All essays