A simulation that stops when it looks settled

A coverage table’s cells and a comparison of two methods were both measured at a number of replications fixed before a single one ran. Simulations are rarely run that way. They are started, watched, and stopped when the answer looks done — when an estimate has clearly departed from its target, or has reached it, or has stopped changing. Each of those is a reasonable description of finishing, and each is a stopping rule.

A p-value is relative to the sampling plan that produced it, and testing a trial five times at the nominal level rejects a true null 14% of the time. A simulation is a sampling plan too, with replications in place of patients, and the arithmetic does not change because the thing being sampled is a computer’s output. What this essay measures is what three natural rules do to the number a simulation reports about an interval whose coverage is known.

A monitor that checks for a departure #

The first rule is the one a careful simulation adopts to save time. Every 250 replications, compute the running estimate of the coverage and a 95% interval for it; if the interval excludes 95%, the interval being studied has been shown to miss its level, so stop and report it. Otherwise continue, to ten thousand replications at most, which is forty looks.

The interval being simulated covers exactly 95%, so every stop is a false alarm. In the figure, eight of the twenty runs stop before ten thousand, and that is not an unlucky twenty. The chance of stopping can be computed exactly, with nothing simulated: after each look the distribution of the running count of hits is convolved with the next block of 250 Bernoulli trials, the probability on every count the rule stops at is removed and recorded, and what is left carries forward. At ten thousand replications the recursion gives 29.54%.

The rate climbs the way every sequential error rate climbs. After the first look it is 4.02% — close to the nominal rate, a little under it because a count of 250 is discrete. After two looks it is 7.54%, after five 13.25%, after ten 18.33%, after twenty 23.93%. Checking every hundred replications instead, a hundred looks, it reaches 35.61%; checking every thousand, ten looks, 18.83%. The stops are not one-sided: of the 29.54%, 15.97% stop with the estimate below 95% and 13.57% above it, so the monitor produces both false findings of undercoverage and false findings of conservatism.

The recursion has a second route that shares only the rule with it: four thousand streams of Bernoulli hits, each from its own seed, checked the same way. They stop 29.13% of the time, within a standard error of the exact figure.

The practical content is simple and uncomfortable. A simulation study that ran each cell “until it was clear”, with a check every few hundred replications, has reported roughly one correct method in three as having missed its level at some setting — the same inflation as twenty honest analyses of nothing, arriving through time rather than through choice.

A boundary for the monitor, and what it costs #

The repair is the one sequential trials use: spend the 5% across the looks rather than at each one. The simplest version reads every look at the same, stricter critical value, and the value can be found by bisection on the exact recursion.

Reading the running interval at 2.7628 standard errors instead of 1.96 — 0.573% a look — holds forty looks to 4.990% for a correct interval. That is the constant boundary a trial that spends its error rate would call Pocock’s, solved here for this monitor rather than read from a table.

It is not free. Against an interval whose true coverage is 94.5%, the bounded monitor detects the shortfall 46.84% of the time. A single analysis of all ten thousand replications, with no looks at all, detects it 62.67% of the time. The unbounded monitor detects it 78.97% of the time, and most of that advantage is the same false alarms it raises against a correct interval. Against a 94% interval the bounded monitor detects 97.28%, where a single analysis reaches 99.29%. Looking early buys the chance to stop early and pays in power, which is the whole of the trade a sequential design makes.

A rule that stops when the estimate reaches the target #

The second rule is less deliberate and more common. A simulation of a new method is run until it produces the answer the author hoped for — here, until the running estimate of coverage reaches 95% — and then reported. Nobody describes this as a stopping rule; it is described as running the simulation long enough.

The interval here covers 94%, a full point short of its level, and seven of the twenty runs stop by reading 95% at some look. Computed exactly, a run stops with an estimate of at least 95% on 37.21% of runs, and four thousand counted streams give 37.43%.

Across true coverages the pattern is what a search for an answer looks like. An interval at 93% is reported as reaching 95% on 12.28% of runs; at 93.5%, on 21.70%; at 94.5%, on 61.84%. Run to a fixed ten thousand replications instead, the 94% interval reads at least 95% with probability 0.0009%, and the 94.5% interval with probability 1.40%. The rule has converted a simulation that would almost never have endorsed the interval into one that endorses it more than a third of the time. And a correct 95% interval is reported as reaching 95% on only 91.94% of stopped runs, because a run that starts low and never crosses ends below.

Averaged over every run, the stopped simulation of the 94% interval reports a coverage of 94.59%. That is not a noisy 94%; it is a biased one, and the bias is the selection. Each run reports the estimate at the moment it was highest relative to the target, which is the arm chosen for being ahead and the study that reached significance under another name.

A rule that stops when the estimate settles #

The third rule sounds like the most innocent and is the one most often recommended: run until the estimate stops moving. Here, stop at the first look where the running estimate differs from the previous look’s by less than a tenth of a point.

The obvious worry is that this, too, is a selection — that a run is more likely to settle while it is on one side of the truth than the other. It was expected to bias the reported number, and the counts were set up to measure how much. It does not, and the expectation is what the counts refuse.

For the 94% interval, the settled rule reports 93.99% on average over four thousand streams, using 1,496 replications on average, and ends reading at least 95% on 6.15% of runs. For a correct 95% interval it reports a coverage whose bias is −0.0008 points, with a standard error of 0.0099 points, using 1,436 replications, and the estimate’s own 95% interval contains the truth 94.5% of the time. There is no bias here to within a hundredth of a point.

The reason is in what “settled” reads. The change between two looks is the new block’s departure from the running estimate, divided by the number of looks so far. How large that change is depends on how variable a block is, which near 95% barely depends on where the estimate sits, and on how many looks have passed — not on whether the estimate is currently above or below the truth. So the moment of stopping is close to independent of the direction of the error, and a rule that does not select on the direction cannot bias the estimate towards either side. That is an argument, not a proof, and it is only measured here: the rule depends on the path through two consecutive estimates, which the exact recursion used above does not carry.

What the settled rule does do is choose the number of replications, and it chooses it from the tolerance rather than from the question. With a tolerance of a tenth of a point it stops at about fifteen hundred replications; with a twentieth of a point, at 1,915 for a correct interval. The settled simulation of the 94% interval reads at least 95% on 6.15% of runs, and a fixed run of fifteen hundred replications does so on 5.45% — the two are nearly the same experiment. So a simulation that stopped when it settled has not measured coverage to within the tolerance it settled at. It has measured it with fifteen hundred replications, which a coverage table’s arithmetic says can see a shortfall of somewhat under two points and cannot see one of one point.

When each rule stops, and what the stopped runs say #

The rates above are averages over every run, and the recursion can also say when the runs stop and what each group reports, which is where the two unhealthy rules show their shape most clearly.

The target rule stops early or not at all. Of the 37.21% of runs that report a 94% interval as reaching 95%, 25.99% of all runs stop at the very first look, after 250 replications — seven stops in ten happen there — and 35.68% have stopped by the fourth look, a thousand replications. The mean look at which a stopped run stops is 1.625. That is exactly where a running estimate is noisiest: a block of 250 hits from an interval covering 94% reads 95% or more by chance quite often, and a thousand replications rarely do. The stopped runs report 95.64% on average; the 62.79% that never stop end, at ten thousand replications, just under 94%. The reported average of 94.59% is a mixture of a noisy early number that was kept because it looked right and an accurate late number that was kept because nothing else happened.

The departure monitor spreads its stops out. A correct interval’s false alarms arrive at look 10.86 on average, about 2,700 replications in; 39.2% of them come in the first four looks and 62.1% in the first ten. So a monitor’s false alarms are not confined to a noisy start that a burn-in period could remove, and a rule of the form “ignore the first thousand replications, then monitor” would still raise most of them.

The settled rule stops on a coincidence rather than on stability. Halving its tolerance, from a tenth of a point to a twentieth, does not double the replications, as a precision target would; it moves a correct interval’s average from 1,436 to 1,915, and the 94% interval’s from 1,496 to 1,989. The estimate at a twentieth of a point is still unbiased — −0.006 points, with a standard error of 0.0089 — and its own interval still covers 94.6%. What the rule responds to is one small change between two looks, which happens by chance long before the estimate is known to within the tolerance, and the replications it chooses grow far more slowly than the precision it seems to promise.

A tolerance is not a precision #

That last point deserves to be separated from the rest, because the settled rule is the one most likely to be written into software as an automatic stopping criterion, and its failure is not bias, which it avoids, but mislabelling.

A report that a simulation “ran until the estimate changed by less than 0.001” invites the reading that the estimate is known to about a tenth of a point. At fifteen hundred replications the estimate’s standard error near 95% is 0.56 points, more than five times the tolerance, and its 95% interval is more than two points wide. The tolerance describes the size of one step between looks; the precision is set by the number of replications, and the rule chose that number by chance.

The honest translation is mechanical: replace the tolerance in the report with the number of replications actually used and the standard error it implies. The same translation shows why a tighter tolerance is a poor substitute for a planned number of replications. A plan to detect a shortfall of one point needs about four thousand replications, and no tolerance setting of this rule reliably delivers that; it delivers whatever a chance small step happens to deliver.

What each rule reports #

Set side by side for the same 94% interval, the four ways of ending a simulation make four different claims. A fixed ten thousand replications reports 94.00% on average and never reports 95%. A fixed fifteen hundred reports 94.00% and reports 95% on 5.45% of runs, which is its honest error at that size. Settling reports 93.99% and 95% on 6.15%, which is the fixed fifteen hundred wearing a different description. Stopping at the target reports 94.59% and 95% on 37.21%, and it is the only one of the four that is wrong about the interval rather than imprecise about it.

The monitor for a departure belongs on the same table from the other side. Against a correct interval it is wrong about the interval 29.54% of the time, and against a short one it is right more often than a fixed analysis only because it is wrong more often in general. Its boundary-corrected version is honest and slower to find a real shortfall.

Which of these numbers are exact #

Exact. Every figure for the departure monitor, the boundary and the target rule is a recursion over the binomial distribution of the running count, look by look, with the probability on stopping counts removed as it goes: the 29.54%, the rates at every look and every spacing, the boundary of 2.7628 and its 4.990%, the detection rates, the 37.21% and its neighbours, and the mean coverage the target rule reports.

Counted. Four thousand streams check the monitor (29.13%) and the target rule (37.43%) against their recursions. The settled rule has no recursion here — it reads two consecutive estimates — so its bias, replications and coverage are counted only, with their standard errors stated.

Particular to this monitor. Checks every 250 replications to a limit of ten thousand, a Wilson interval for the running estimate, and a tolerance of a tenth of a point for settling. A monitor that checks more often flags more, as the 35.61% at every hundred shows; the direction of every conclusion is unchanged by the particular spacing.

What a simulation report has to say about when it stopped #

One sentence, and it is the same sentence a trial has to write: the number of replications and whether it was fixed in advance. A fixed number is the cleanest report. A monitor for a departure is defensible with its boundary stated. A rule that stops at the target is not defensible under any description, because the number it reports is conditioned on having reached the answer. And “run until the estimate was stable” is defensible and should be translated into what it is — a number of replications, with the shortfall that number can detect.

Where this goes next: when no number of plain replications is enough #

Every rule here assumed that a simulation of enough replications would, eventually, see the answer. For a coverage near 95% that is true: the event of interest happens on nineteen replications in twenty. For the probabilities a normal distribution is most often asked for — the far tail of a distribution, where every approximation in the subject is read — it is not. A tail probability of three in ten million needs hundreds of millions of plain draws before a simulation has seen enough of the event to estimate it at all.

The draws aimed at the tail takes that case: importance sampling, which draws from a distribution aimed at the tail and reweights, and which reaches the answer in a few hundred draws — or, with the wrong aim, produces estimates that look settled by every rule above and an interval that does not cover. It is distinct from this essay because the failure there is not in when the simulation stops but in what its standard error can see.