A coverage table with its own error

The most common figure in a methods paper is a table of coverages: a grid of sample sizes and conditions, and in each cell the share of simulated intervals that contained the truth. A reader scans it for the cells that fall short of 95%, and those cells become the finding.

Every number in that grid is an estimate. A cell reading 94.1% is a count of hits over some number of replications, and it carries an error that the table rarely prints. A seeded simulation reproduces its own answer exactly, which removes one explanation for a surprising cell and leaves the other untouched: the cell may be a correct interval that happened to draw a low count.

The question this essay counts is what that error does to a whole table. It uses an interval whose coverage is known exactly, so that every departure from 95% on the page is simulation error and nothing else.

Twenty cells of an interval that cannot miss #

The t interval for a normal mean covers exactly 95% at every sample size and every spread. That is not a simulation result; it is the definition of the t distribution, and the reason the t interval exists at all. So a table of its coverage has a known right answer in every cell, and whatever the table shows beyond that is the simulation speaking.

At a thousand replications a cell, the twenty cells read from 93.9% to 96.5%. That is a spread of more than two and a half points across cells that are all, in truth, the same number. And two of the twenty are flagged, in the sense a reader of a methods table flags a cell: the estimate plus and minus 1.96 of its own standard error excludes 95%. One of the flagged cells reads above the target and one below, and a reader who had not been told the answer would write a sentence about each.

There is nothing wrong with either cell. There is nothing unusual about the table either, as the next two figures show: two flags in twenty is close to what a correct table produces on an ordinary day.

What one cell’s error is, exactly #

A replication either covers or it does not, and replications on separate seeds are independent. So the number of hits in a cell of R replications of an interval whose coverage is 95% is exactly a binomial count, Binomial(R, 0.95), and the cell’s reading is that count divided by R. Nothing about the interval, the sample size or the parent enters — only R and the true coverage.

At a thousand replications its standard deviation is $\sqrt{0.95 \times 0.05/1000}$, 0.69 points. A cell reading within 1.4 points of 95% either way is doing exactly what a correct cell does, and one reading 93.9% is 1.6 of those standard deviations low.

That claim has a second route that shares nothing with the first. Two thousand cells simulated from scratch spread with 1.038 times the binomial variance, and a chi-square test of their histogram against the binomial’s eight bins reads 5.83 on 7 degrees of freedom — about as close as two thousand draws from the right distribution land. The cells are binomial. Every statement below about a table is therefore arithmetic on binomial counts, and it is exact.

Why a correct cell is flagged more than one time in twenty #

A reader’s flag is a test: it rejects when a cell’s own interval excludes 95%. For one cell of a thousand replications of a correct interval, that test rejects with probability 5.83% — computed exactly by summing the binomial over every count whose interval misses. The nominal rate is 5%, and the gap is not rounding.

Two things push it up, and both are the ones an interval for a proportion misses its level by. The standard error the flag uses is read from the cell itself, $\sqrt{\hat p(1-\hat p)/R}$, so a cell that happens to read high reports a smaller error than a cell that reads low and is flagged more easily for the same distance from 95%. And the count is a whole number, so the set of counts that are flagged jumps rather than slides as R changes, and at most values of R it takes slightly more than 5% of the binomial’s mass. The flag is a Wald interval for the cell’s own coverage, which is the Wald interval for a proportion, the one that undercovers near zero and one, turned on the simulation’s own counts.

The excess shrinks with the replications and does not vanish at any size a table is built at: 7.44% at two hundred replications, 6.83% at five hundred, 5.83% at a thousand, 5.33% at two thousand, and 4.89% at ten thousand, where it has crossed to the other side of 5%. That last reading is the same raggedness a larger sample can show against a smaller one, and it means the per-cell rate is a property of R that has to be computed rather than assumed.

A correct table flags a cell most of the time #

A table is many such tests. Five correct cells flag at least one of them 25.9% of the time; ten, 45.1%; twenty, 69.9%; fifty, 95.0%; a hundred, 99.8%. A hundred simulated tables of twenty were run as a count, and 72 of them flagged at least one cell — inside a standard error of the exact figure. This is exactly the arithmetic of twenty tests of true nulls, which produce at least one false positive 64% of the time, applied to the table a methods section is built on rather than to the analyses it reports.

The obvious repair is more replications, and it does not work in the way it is expected to. More replications shrink every cell’s standard error, and every cell’s flag shrinks with it, so each cell is still flagged at about the same rate. At two hundred replications a cell is flagged 7.44% of the time and a table of twenty 78.7%; at ten thousand, 4.89% and 63.3%. The rate a table of twenty flags a correct cell approaches $1 - 0.95^{20}$, which is 64.2%, and replications do nothing about that number because the number is not about precision. It is about reading twenty intervals at once.

What does repair it is reading the table as a family. Each cell’s interval widened to Bonferroni’s level for twenty, 3.02 standard errors, holds the table to 7.6% at a thousand replications. That is still above 5%, and at two hundred replications it is 41.6%. The reason is instructive and it is again the discreteness: the Bonferroni level asks each cell’s count to fall in the far tail of a binomial, where a normal approximation to a count of two hundred is poor and a cell that happened to hit every time has a standard error of zero. At ten thousand replications the same correction reads 5.4%. So the correction is itself a claim about the table’s arithmetic, and it is honest only when the counts are large enough for its tail to be read.

A real shortfall in the same table #

A table that flags correct cells is only half a problem. The other half is whether it flags incorrect ones, and the cleanest test is to put a real shortfall beside the correct one. The z interval — the same interval with the normal’s 1.96 in place of the t’s critical value — computed on exactly the same draws undercovers by an amount that is also exact: 87.844% at five observations, 91.835% at ten, 93.517% at twenty and 94.430% at fifty.

At a thousand replications the z interval is flagged in 12 of the twenty cells: all five at five observations, four of five at ten, three at twenty and none at fifty. The exact probability that one z cell is flagged is 100.00% at five observations and 97.19% at ten, so those rows are reliably seen. At twenty it is 45.87%, a coin flip. At fifty it is 9.72% — against the 5.83% at which a correct cell is flagged. A real shortfall of 0.57 points sits in the table at almost the rate a correct cell’s noise does, and a reader of this table would call the row at fifty observations fine, which it is not, while calling two correct t cells suspect.

At ten thousand replications the t cells read from 94.49% to 95.42% and two of them are still flagged. The z interval is flagged in 18 of twenty, and the chance of flagging a z cell at fifty observations has risen to 70.56%. So the table has become able to see a shortfall of half a point, and has not stopped reporting spurious ones: it reports the real row seven times in ten and a correct cell somewhere most of the time. More replications change which of a table’s findings are real. They do not change that some of them are not.

The z cells, checked against the t distribution #

The z interval’s coverage figures above are not taken from the simulation; they are exact, and the reason they can be is worth a paragraph because it is the same move every exact coverage calculation makes. For normal data the statistic $(\bar x - \mu)/(s/\sqrt n)$ has a t distribution on n − 1 degrees of freedom whatever the mean and spread, so an interval of ±1.96 standard errors covers with probability $2F_{n-1}(1.96) - 1$. At five observations that is 87.844%; the gap to 95% is the whole of what the t interval’s wider critical value buys.

That gives the table a second route in every row. Pooling the five cells of each sample size, 5,000 replications a row, the counted z coverage reads 88.08% at five observations against the exact 87.844%, 92.06% at ten against 91.835%, 93.52% at twenty against 93.517% and 94.40% at fifty against 94.430%. Each is inside the binomial error of 5,000 replications of the exact value. The simulated table and the closed form agree, which is what licenses reading the flags in each row against a known truth rather than against the table’s own impression of itself.

It also shows how little a single cell could have established. The row at fifty observations is 0.57 points short in truth, and pooling its five cells into 5,000 replications reads it as 0.60 points short — correct, and still less than two of the pooled row’s own standard errors, which are about a third of a point. Five cells of a thousand replications, looked at together, see the shortfall only faintly. Looked at one at a time, as a table is read, they do not see it.

How many replications a stated claim needs #

The honest version of a coverage table starts from the size of shortfall it means to be able to see, and the replications follow from arithmetic that is the same as a trial’s sample size. To detect a true coverage δ points below 95% with 80% power at a two-sided 5% needs

$$R = \left(\frac{1.96\sqrt{0.95 \times 0.05} + 0.84\sqrt{p_1(1 - p_1)}}{\delta}\right)^2, \qquad p_1 = 0.95 - \delta .$$

A two-point shortfall needs 1,031 replications, and exact summation over the binomial at that size gives a power of 79.06%. A one-point shortfall needs 3,932; half a point, 15,329; a fifth of a point, 94,258. Turned around, a table of a thousand replications a cell has 80% power against a shortfall of 2.03 points and no more, and a table of ten thousand against 0.62.

That is the sentence a coverage table should print under itself. A claim that a method “maintains nominal coverage”, read off a thousand replications, is a claim that its coverage is not two points low. It says nothing about one point, which in most applications is the difference that matters, and a table reporting 94.3% in a cell of a thousand replications has not established that the method undercovers either. Both readings are what one run treated as a finding looks like in tabular form.

Cells that are not binomial #

Everything above rests on one premise: that the replications in a cell are independent, so its hit count is binomial. It is easy to break without noticing, so it is worth breaking deliberately to show that the binomial check notices.

Build a table whose cells each draw fifty samples and then reuse them twenty times over — the same fifty datasets resampled into a thousand replications. Every cell still reads a coverage near 95% and still reports a standard error computed as if it had a thousand independent replications. Across its cells the spread is 18.79 times the binomial variance, and the chi-square against the binomial is 664.7. A table like that flags cells far more often than even the arithmetic above, and nothing in any single cell says why.

The tables in this essay keep the premise by construction. Each cell draws from its own block of seeds, the block for cell i starting R seeds after the block for cell i − 1, so no two cells share a draw and no replication inside a cell shares one with another. That is what makes the table-wide figures exact rather than approximate: the chance that none of twenty independent cells is flagged is $(1 - q)^{20}$, and 69.9% is one minus that at a per-cell rate of 5.83%. A table whose cells were computed on overlapping datasets — the same simulated samples scored at twenty settings, which is a common economy — would have correlated flags, and its table-wide rate would be neither that number nor any other closed form.

The same premise is the reason the seeding of the random draws had to be repaired when consecutive seeds turned out to be correlated. Correlated replications do not bias a coverage estimate; they make its error larger than the formula the table uses, which is the one kind of error a reader of the table cannot see.

Which of these numbers are exact #

Exact. The binomial distribution of a correct cell, the probability that a cell is flagged, the probability that a table of k cells flags one, the z interval’s coverage at each sample size, the power at every number of replications and the Bonferroni figures are sums over binomial counts. None of them is simulated, and none of them depends on the t interval, the normal parent or the particular table beyond R and the true coverage.

Counted, as the second route. The two thousand cells that are shown to be binomial, the hundred tables that flag at 72%, and the tables drawn in the figures are simulations, and each is required to sit where the arithmetic puts it.

Particular to this table. Twenty cells, a thousand or ten thousand replications, and a two-sided flag at the reader’s usual level. A table of forty cells, or a reader who flags only undercoverage, changes every figure above and none of the argument: a table is a family of tests whether or not it is read as one.

What a table of coverages can print instead #

Four additions turn a table of point estimates into a statement a reader can check.

The replications per cell and the standard error they imply. At a thousand, 0.69 points near 95%. A reader then knows that 94.1% is not a finding.

The shortfall the table has power to see. At a thousand, two points. A method shown to be “within a point of nominal” at that size has not been shown to be.

The number of cells, read as a family. A sentence of the form “with twenty cells, one flagged cell is expected from a correct method most of the time” costs nothing and prevents the most common misreading.

An exact route, where one exists. For most intervals for a proportion the coverage is a finite sum and needs no simulation at all, as the interval that covers 81.9% at twenty showed. A table that simulates what can be summed has chosen to report an estimate of a known number.

Where this goes next: two methods on the same draws #

A coverage table is rarely about one interval. It is about which of two or three intervals is better, and the comparison it invites is a difference between cells: the z interval covers 1.5 points less than the t interval at twenty observations. That difference is also an estimate, and its error is not the error of either cell. When both intervals are computed on the same simulated datasets — as they were in the tables above — the two cells’ errors are correlated, and the difference can be much more precise than either cell, or, for some pairs of methods, less precise than if the draws had not been shared.

The same draws for both methods measures that, exactly, for every pair of four intervals for a proportion. It is distinct from this essay because the quantity at issue is the difference between two correlated binomial counts rather than one count, and the replications it needs can be several times smaller or larger than this essay’s formula says.