The same draws for both methods

A simulation that compares two intervals almost always computes both on the same datasets. It is cheaper — one set of draws instead of two — and it is widely recommended, under the name common random numbers, as a way to make the comparison more precise: whatever luck a dataset carries, both methods see it, and it cancels from their difference.

That recommendation is usually true, sometimes by a large factor, and sometimes false. Whether it holds for a given pair of methods is not a matter of experience or of the size of the simulation. For intervals for a proportion it is an exact computation over the samples each interval covers, and the answer differs sharply between pairs that look alike.

The measurement also exposes the part that matters more than the precision: the standard error a comparison reports has to know whether the draws were shared. A coverage table’s cells are binomial counts with an error that can be written down. A difference between two cells computed on the same draws is not a difference of two independent binomial counts, and the formula that treats it as one can be wrong in either direction.

Four outcomes for every dataset #

A dataset of fifty trials produces a count of successes, and each interval built on that count either contains the true proportion or does not. Two intervals on the same count therefore produce one of four outcomes: both cover, only the first, only the second, or neither. Because the count takes only fifty-one values, the probability of each outcome is a finite sum over the binomial, with nothing simulated.

For Wilson’s interval against Clopper–Pearson’s at a true proportion of 0.2, both cover on 95.1% of samples, only Wilson on 0.0%, only Clopper–Pearson on 1.6%, and neither on 3.3%. So Wilson covers 95.07% and Clopper–Pearson 96.71%, a difference of −1.636 points, and the two intervals almost always agree about a given sample: every sample Wilson covers, Clopper–Pearson covers too, and they part company only on the few counts where the more conservative interval reaches a little further.

A simulation of R datasets estimates that difference by the share of datasets on which only the first covers minus the share on which only the second does. On shared draws the variance of that estimate, per replication, is

$$\operatorname{var}_{\text{shared}} = p_{10} + p_{01} - (p_{10} - p_{01})^2,$$

where $p_{10}$ and $p_{01}$ are the two one-sided discordant shares. The datasets on which the intervals agree contribute nothing to it at all. On separate draws the variance is the sum of the two binomial variances, $p_A(1-p_A) + p_B(1-p_B)$, and the datasets on which they agree contribute to both. For this pair the second is 4.891 times the first: the correlation between the two intervals’ hits is 0.810, and nearly all of each interval’s binomial noise is noise the other interval shares.

The same comparison four hundred times #

That ratio is exact, and it has a direct route by count. Run the comparison four hundred times at a thousand replications each, once with both intervals on the same draws and once with each on its own.

On shared draws the four hundred estimates spread with a standard deviation of 0.398 points, against an exact 0.401. On independent draws they spread with 0.880 points, against 0.887. Two routes that share no arithmetic agree to within what four hundred draws of a standard deviation allow, and the shared comparison is more than twice as sharp. A simulation that wanted to establish that Clopper–Pearson covers more than Wilson at this setting would see a gap four standard errors wide at a thousand replications on shared draws, and under two on separate ones.

A pair that covers different samples #

The same computation for a different pair reverses the conclusion, and the pair is not exotic. Wilson’s interval against the Wald interval at twenty trials and a true proportion of 0.05 — a small sample and a rare event, the setting in which the Wald interval is known to fail.

Both cover on 56.6% of samples, only Wilson on 35.8%, only Wald on 7.3%, and neither on 0.3%. The discordant shares are large, and they are large on different samples: Wilson covers the count of zero successes, where the Wald interval collapses to a single point at zero, and the Wald interval alone covers the counts of three and four, where Wilson’s interval has moved up past the true 0.05. The correlation between their hits is −0.195. The two intervals’ errors are not shared; they are partly opposed, and a dataset that helps one tends to hurt the other.

With a negative correlation the formula gives the other answer. The variance of the difference on shared draws is 1.164 times larger than on separate draws, so sharing makes the comparison noisier. The difference itself is large — Wilson covers 92.45% and Wald 63.89%, a gap of 28.558 points — which is why nobody would notice: the comparison is decisive either way. The same arithmetic in a pair with a small difference and a negative correlation would quietly cost the precision it was adopted to buy.

Counted, the shared estimates spread with 1.919 points against an exact 1.870, and the independent ones with 1.710 against 1.733. Shared draws are the wider row, as the sum over counts said they would be.

Where sharing pays, across every pair #

Two pairs are two anecdotes, and the exact computation is cheap enough to run over everything. Take the four intervals — Wald, Wilson, Agresti–Coull and Clopper–Pearson — all six pairs of them, at twenty and fifty trials, and at ten true proportions from 0.05 to 0.50.

Of the 120 settings, 44 are ones at which the two intervals cover exactly the same set of counts. There the difference in coverage is exactly zero on every dataset, and on shared draws a simulation of any size reports it without error — the variance ratio is infinite, and those settings are left off the plot rather than drawn at the top of it. It is the extreme case of the argument for sharing, and it is common: at a given sample size many pairs of proportion intervals disagree only at a handful of true proportions.

Across the other 76 the ratio runs from 0.859 to 5.619, and it sits below one at 5 of them — each a pair including the Wald interval at a small true proportion, where the Wald interval’s failures fall on samples the other interval covers. The curve beside the points is $1/(1 - \rho)$, the ratio that two coverages of equal variance would give; the points follow it where the two intervals’ coverages are close and leave it where one interval is far from 95%, because then the two binomial variances are unequal and the correlation alone does not settle the ratio.

How much sharing can cost, and how much it can save #

The asymmetry between the two sides of the figure is worth reading as numbers, because it decides what a default should be. The five settings where shared draws are noisier are all the Wald interval at twenty trials and a small true proportion: against Wilson at 0.05, a ratio of 0.859; against Wilson at 0.10 and against Agresti–Coull at 0.10, 0.962; against Agresti–Coull and against Clopper–Pearson at 0.05, 0.975. The worst of them makes a comparison on shared draws need 16% more replications than the same comparison on separate draws. The best setting on the other side, Wilson against Clopper–Pearson at fifty trials and 0.3, has a ratio of 5.619: shared draws need less than a fifth of the replications.

So the cost of sharing, where it costs anything, is small and confined to a pair whose difference is enormous, and the saving, where it saves, is large and falls on pairs whose difference is small — which are the comparisons that need precision. Sharing draws is the right default for a simulation comparing intervals. What it is not is a default that can be left out of the analysis: the gain is only collected, and the occasional loss only contained, by a standard error that uses the discordant counts.

The 44 settings where the two intervals cover exactly the same counts deserve a sentence of their own, because they are where that standard error behaves in a way that looks like a bug. On shared draws both discordant counts are zero on every dataset, the paired variance is zero, and the simulation reports a difference of exactly zero with a standard error of exactly zero. That is correct: the two intervals are, at that setting, the same interval, and no replication can say otherwise. A table built on separate draws would instead report a difference of a point or so, with a standard error to match, about two methods that cannot differ.

The rule that comes out of the figure is plain enough to state without the exception disappearing. Sharing draws pays when two methods fail on the same samples, and costs when they fail on different ones. For methods that are refinements of each other it pays, often by a factor of several. For methods built on different principles — an interval that collapses at the boundary against one that does not — it can cost, and the pairs where it costs are the pairs whose comparison is most often the point of the table.

The standard error has to know #

The precision of a comparison is only useful if the reported standard error describes it, and here the sharing matters in a way that does not depend on whether sharing helped.

A comparison built on shared draws and reported with the standard error for independent draws — $\sqrt{(p_A(1-p_A) + p_B(1-p_B))/R}$, the formula anyone would write for a difference of two proportions — has a nominal 95% interval whose true coverage is, to a normal approximation, $2\Phi(1.96\sqrt{\text{ratio}}) - 1$.

For Wilson against Clopper–Pearson the ratio is 4.891, and the unpaired interval on shared draws covers 100.0% counted and 100.00% exactly. It is not a conservative interval in any useful sense: it is more than twice as wide as the comparison’s actual spread, and a real difference four of its own standard errors away would be reported as inconclusive. For Wilson against Wald the ratio is below one, and the unpaired interval on shared draws covers 90.0% counted and 93.07% exactly — too narrow, for the opposite reason.

The paired standard error, built from the discordant shares the dataset itself reveals, is the one that belongs to shared draws, and it covers 92.5% for Wilson against Clopper–Pearson (exactly 92.65%) and 93.3% for Wilson against Wald (exactly 94.97%). Neither is quite 95%, and the shortfall for the first pair is its own small instance of the oldest finding about intervals for a proportion: the paired interval is a Wald-type interval on a count of discordant datasets, and at this setting Wilson never covers a sample Clopper–Pearson misses, so one of the two discordant counts is always zero and the normal approximation to it is poor. On independent draws the unpaired interval covers 93.5% and 96.8%.

The shortfall for Wilson against Clopper–Pearson can be computed rather than described. At a thousand replications the expected number of discordant datasets is about sixteen — 1.636% of a thousand — and every one of them is a dataset on which only Clopper–Pearson covers. The estimated difference is minus that count over a thousand, and its paired standard error is built from the same count, so a run that happens to draw eleven discordant datasets reports both a smaller difference and a smaller standard error, and misses. Summing the trinomial distribution of the two discordant counts over every outcome whose interval contains the true difference gives the 92.65% exactly, with the summation window holding all of the probability to twelve decimal places. It is the plain Wald interval for a proportion, failing in the way it always fails when the count it rests on is small, and the repair is the same one: a Wilson-type interval on the discordant count rather than a Wald-type one. That repair is not measured here.

So the correct report of a comparison on shared draws is not a matter of taste about precision. It is the paired standard error, and a table that prints a difference of two coverages from a common simulation with a standard error computed from the two cells separately has printed a number that is wrong about itself, in a direction the reader cannot tell without the four-outcome table.

The replications a comparison needs #

Everything above becomes a count of replications once the size of difference to be detected is fixed, and the arithmetic is the same as a trial’s, with the variance of a single replication in place of the variance of a single subject.

The actual difference between the two intervals at this setting is 1.636 points. On independent draws a simulation needs 2,309 replications to detect it with 80% power; on shared draws it needs 472. The ratio is 4.891 at every size of difference, since it is the ratio of two per-replication variances and the size of the difference divides both equally.

That is the practical content of the essay in one pair of numbers. A comparison designed with the formula for independent proportions, and run on shared draws, runs nearly five times longer than it needs to and then reports an interval more than twice too wide. A comparison of Wilson against Wald designed the same way runs slightly shorter than it needs to and reports an interval too narrow. Neither error is visible in the output.

A paired design by another name #

Sharing draws between two methods is the simulation’s version of a decision made, in two forms, in the design of real studies. Arranging units in pairs before any outcome exists removes the variance the pair has in common and costs a degree of freedom; the saving depends on how alike the pair is. And subtracting a baseline reading from a follow-up makes a randomised comparison less precise when the two readings correlate below one half — a paired difference whose pairing is too weak to pay for itself.

The coverage comparison is the same bargain with one feature the others lack: the correlation can be negative, because two methods can fail on opposite samples in a way two readings of one person cannot. That is where sharing turns from a saving into a cost, and it is why the shortest interval on a table and the most reliable one are exactly the pair for which the common recommendation is least safe.

What is exact here and what is counted #

Exact. The four outcome probabilities, both coverages, the difference, the correlation, the two variances and their ratio at every setting of the grid, the true coverage of each interval for the difference, and the replications needed are sums over binomial counts for intervals whose endpoints are deterministic functions of the count. No simulation enters any of them.

Counted. The spreads of four hundred simulated comparisons and the coverages of the intervals for a difference are simulations, each required to sit within its own standard error of the exact value.

Particular to proportions. The exactness comes from the count being the whole of the data. For intervals on continuous data — the t and z intervals of the coverage table — the four outcome probabilities are integrals rather than sums, and the ratio has to be estimated from the simulation itself. The direction of the conclusion carries over unchanged: sharing pays when the methods fail together and costs when they fail apart.

Where this goes next: a simulation that decides when to stop #

Both essays so far on the simulation’s own error have fixed the number of replications before a single one was run. In practice that number is often chosen while the simulation runs: it is extended until the estimate looks stable, or until it clears the target, or until an interval for the coverage excludes 95%. Each of those is a stopping rule, and a stopping rule is part of what a p-value is relative to.

A simulation that stops when it looks settled measures what three such rules do to the number a simulation reports about an interval whose coverage is known. It is distinct from this essay because the question is no longer how precise a fixed comparison is, but what the decision to stop does to the comparison’s meaning — and one of the three rules, measured, does something different from what intuition predicts.