A bootstrap that repeats the design

What a design chosen from the data costs ran a twelve-run experiment on a Michaelis–Menten curve twice: once with its settings fixed at a guess of the half-saturation constant made in advance, once with the settings chosen from the experiment’s own early responses. The Wald interval for the constant fell short of 95% after both — 92.6% after the fixed design, 91.3% after the adaptive one — and the profile interval, which inverts the residual sum of squares and needs no derivative, covered 94.3% and 94.9%. So the shortfall belonged to the interval’s quadratic approximation, and the adaptivity added about a point to it.

That essay named two answers it did not build. Exact conditional inference, which conditions on the design actually realised. And bootstrap intervals, which it called the practical answer, at the cost of a refit per resample per experiment. A bootstrap after an adaptive design has a choice the fixed design never faces, and it is the choice the earlier essay’s question was about: the resamples can be drawn at the settings the experiment happened to run, or the whole procedure — first stage, estimate, redesign, second stage — can be repeated on every one, so that the resamples vary the design the way the experiment did.

This essay builds both, in the two forms a bootstrap interval usually takes, and sets them beside the Wald and profile intervals on the earlier essay’s own experiments.

Two bootstraps, three intervals each #

Both bootstraps are parametric: they draw new responses from the fitted curve, with the fitted constant, maximum and noise level, and refit. They differ in where.

Design kept draws the new responses at exactly the twelve settings the experiment ran, and refits. It treats the design as fixed — which is what the Wald interval does too, implicitly, when it reads the standard error off the information matrix of the settings it was given.

Design re-run repeats the experiment: eight runs at the local design for the original guess, a fit, two runs at the local design for that fit’s estimate, another fit, two more runs, and the final fit — every redesign recomputed from the resampled responses. It is the only construction here that knows the settings were chosen.

Each set of 199 resampled constants gives three intervals: the percentile interval, the resamples’ own 2.5th and 97.5th percentiles; the basic interval, those percentiles reflected about the estimate; and the studentised interval, which resamples the Wald t-ratio rather than the estimate and reads its quantiles in place of $\pm 1.96$.

After a threefold wrong guess #

The Wald interval covers 91.3% and the profile interval 94.9% — the earlier essay’s numbers, on the earlier essay’s experiments, because the bootstraps were run on exactly the experiments it measured.

The percentile bootstrap covers 90.6% with the design kept and 91.6% with it re-run. The basic bootstrap covers 89.5% and 88.5%. The studentised bootstrap covers 93.0% and 92.0%.

Two things are plain at once. Re-running the design moves no interval by more than a point, in either direction, on eight hundred experiments where a coverage’s standard error is about 0.8 of a point. And neither the percentile nor the basic interval beats the Wald interval, while the studentised one closes about half of the gap to the profile interval and no more.

Why repeating the design buys nothing here #

The resampled constants’ spread says why. With the design kept, the 199 resampled estimates have a standard deviation of 0.540 on average; with the design re-run, 0.547. The estimate’s actual spread across the eight hundred experiments is 0.536. Both bootstraps reproduce it, and the one that re-runs the design adds about one per cent.

That is the earlier essay’s finding seen from inside a single experiment. A design chosen from the data costs a point of coverage because the settings it ends up at are themselves random, and that randomness is small here: the redesigns move between local designs that all put their runs in roughly the same two places, the top of the range and somewhere near the estimated constant. A bootstrap that repeats them is adding a variability that was nearly absent. So the choice the adaptive design seemed to force on the bootstrap is, on this problem, a choice between two nearly identical distributions.

The design that stops guessing measured the design’s side of it, as an efficiency: a first stage of a few runs recovers most of what a threefold wrong guess costs, and a design that finds its way that quickly has little left to vary between one experiment and the next. A bootstrap that repeats it is a second instrument reading the same small number.

Why the percentile bootstrap is no better than Wald #

The more instructive result is that the percentile bootstrap, which reproduces the estimate’s spread correctly, covers no better than an interval that only approximates it. The reason is in what neither of them carries.

Each resample is drawn with the noise level the experiment estimated, and twelve runs fitting two parameters leave ten degrees of freedom for that estimate. The percentile interval is the spread of the constant given that noise level; it is not widened for the noise level’s own uncertainty, just as a Wald interval read against 1.96 is not. At ten degrees of freedom that uncertainty is worth a t multiplier of 2.23 rather than 1.96, about fourteen per cent of width, and it is the part of the shortfall both intervals share.

That reading can be tested directly, and the test is the most consequential number in this essay. Take the Wald interval and change one thing: read it against Student’s t on ten degrees of freedom rather than against 1.96. Nothing about its shape changes — it is still the estimate plus or minus a multiple of the curvature’s standard error — and it is the allowance for an estimated noise level and nothing else.

After the design chosen from the data the Wald interval read against t covers 94.8%; the profile interval covers 94.9%. After the fixed design, 95.4% against 94.3%. So at a threefold wrong guess the whole of the Wald interval’s shortfall is the multiplier. The quadratic approximation that the earlier essay held responsible costs nothing that eight hundred experiments can see: its control compared a Wald interval read against a normal with a profile interval read against an F, and the difference between those two references is the difference it measured.

That revises the earlier diagnosis and leaves its conclusion standing. The adaptivity still costs about a point, the profile interval is still the honest one, and the reason it is honest is now plainer: it carries the noise level’s uncertainty in its F, which a t multiplier also carries and 1.96 does not.

The studentised bootstrap resamples the t-ratio, and a resampled t-ratio carries its own resampled standard error, so the noise level’s uncertainty is partly in it — partly, most likely because the resampled ratios are drawn around the fitted noise level rather than around the true one, and their spread of denominators is the spread an estimate from ten degrees of freedom would have if the fitted level were exact. That is why it closes half of the gap rather than all of it, and why a t multiplier, which carries the same allowance exactly, closes all of it.

The widths order the same way. The Wald interval averages 1.995, the percentile bootstrap 2.061, the studentised bootstrap 2.212 with the design kept and 2.230 re-run, and the profile interval 2.395. The Wald interval read against t is 2.23 over 1.96 times its normal width, about 2.27, so it reaches the profile interval’s coverage at about five per cent less width — the profile interval’s extra is its asymmetry, which at this guess buys nothing in coverage.

After a design fixed at a twelvefold wrong guess #

The earlier essay’s most revealing cell was the one where the guess is badly wrong. Started at K = 0.25 with the truth at 3, the fixed design puts every run where the curve is already flat, and the constant is estimated from almost no information.

Here the intervals come apart. The Wald interval covers 89.9% and the profile interval 94.5%, as the earlier essay found. Read against t the Wald interval covers 91.9%: the multiplier is now worth two points and the shape another two and a half, so this is the one cell in the essay where the quadratic approximation genuinely costs coverage — on a design whose runs are all where the curve is flat, the residual sum of squares is far from a parabola in the constant. The percentile bootstrap covers 90.5%. The studentised bootstrap — the construction that did best of the three bootstraps after the adaptive design — covers 77.8%, and its intervals average 54.5 wide for a parameter whose true value is 3.

Both numbers have one cause. The studentised interval divides each resampled error by the resample’s own Wald standard error, and on a design whose runs are all where the curve is flat that standard error is enormous on some resamples and small on others: the information matrix is nearly singular, and how nearly depends on where the resampled responses happened to put the estimate. A t-ratio built on a denominator that unstable has quantiles set by its rare small denominators, which are large in both directions, so the interval is very wide on most experiments and placed badly on the ones where it is not. The profile interval has no denominator; it asks which constants fit nearly as well, and on a flat design the honest answer is a long stretch of them, which it reports at an average of 25.1 wide.

The design chosen from the data rescues all of them. Started from the same twelvefold wrong guess, the sequential design moves its runs towards the curvature after its first fit, and the intervals behave as they did from the milder guess: Wald 91.1%, percentile 90.5% kept and 90.4% re-run, studentised 92.6% and 92.5%, profile 93.5%, at widths between 2.8 and 3.6.

So the one place in this essay where the bootstrap’s construction mattered a great deal is the one place the design did not adapt. That is a reversal of the question the bootstrap was brought in to answer — whether adaptivity needs a special bootstrap — and the answer is that it is a design that fails to adapt that needs one.

Where the design’s own guess enters the interval #

The two guesses measured here are the two the field has used since the design for the worst case: a threefold error, where a local design is still worth most of what the right one is worth, and a twelvefold error, where it is worth almost nothing. Where the minimum is attained found the worst case of a design over a range sitting at the end of the range the design was not built for, and the intervals behave the same way: at the mild guess every interval in this essay covers between 88.5% and 95.4%, and at the severe one, under the design that could not move, they spread from 77.8% to 94.5%.

So the choice of interval is a second-order decision wherever the design is reasonable and a first-order one wherever it is not — and the design is unreasonable exactly when the guess was bad, which is the one thing an experimenter cannot see from inside the experiment. A rule that picks the interval after looking at how wide it came out has quietly changed what its 95% refers to. The protocol that survives both cells is the one that fixes the profile interval in advance, or the t-multiplied Wald interval wherever the design is known to adapt, and the bootstrap nowhere it was not needed.

What to report after a design that read the data #

Three practical readings come out of the two tables.

The design does not need to be repeated. On this problem a bootstrap that keeps the design and one that repeats it agree to within a point everywhere they were measured, after a mild wrong guess and a severe one. The second costs a sequential procedure per resample; the first costs a refit. Unless the adaptive rule is much more variable than the one here, the cheaper bootstrap reports the same interval.

The cheapest repair is the multiplier. The Wald interval read against t on the residual degrees of freedom covers 94.8% and 95.4% after a threefold wrong guess, which is as well as anything here, and costs nothing to compute. The percentile bootstrap, which ignores the noise level’s uncertainty exactly as 1.96 does, covers no better than the Wald interval it was meant to improve. The studentised bootstrap carries part of the allowance and closes half of the gap. The profile interval carries all of it and is the only interval within a point and a half of 95% in every cell — including the flat design, where the t multiplier alone is not enough and the shape matters too.

And a studentised bootstrap after an uninformative design is a hazard. Its failure at 77.8% is silent: each experiment produces an interval, and nothing in the interval says that its denominator was unstable, except its width — which a reader of one experiment has nothing to compare with. What studentising costs found the same hazard in a block bootstrap, where the scale a studentised interval divides by rested on two or three block means and the interval came out twice as wide as the percentile one; here the scale rests on an information matrix the design barely supplies, and the effect is larger.

The earlier essay’s experiments, with 199 resamples each #

Eight hundred experiments of twelve runs in each of the two starting guesses, the same experiments as the earlier essay’s figure — the generator is advanced through the calls that measurement makes, in its order, so every Wald and profile coverage here is that table’s to the experiment — and 199 parametric resamples per experiment for each bootstrap, on a stream of their own. The noise standard deviation is 0.05 against a maximum response of 1. The fitted constant is found by a search over a logarithmic grid of six hundred values between 0.02 and 50, which quantises every estimate to about 1.3%, and no comparison above is made at that resolution.

The sequential design in the re-run bootstrap is the earlier essay’s exactly: eight runs at the local design for the original guess, then redesigns after every two runs at the local design for the latest estimate, floored at 0.05.

Repeating the design is worth under a point; the multiplier was the shortfall #

The claims are that on twelve runs a parametric bootstrap that repeats the adaptive design and one that keeps the realised design cover within a point of each other after both a threefold and a twelvefold wrong guess; that the percentile and basic bootstraps cover no better than the Wald interval; that the Wald interval read against t on ten degrees of freedom covers 94.8% and 95.4%, so that at a threefold wrong guess the whole of the earlier essay’s shortfall is the multiplier and none of it the quadratic approximation, while at a twelvefold wrong guess under a fixed design the approximation costs another two and a half points; that the studentised bootstrap closes about half of the gap to the profile interval after the adaptive design; and that after a design fixed at a twelvefold wrong guess it covers 77.8% with intervals more than twice as wide as the profile interval’s.

Not claimed: anything about designs whose adaptive rule is more variable than this one — a rule that switches between widely separated settings would make repeating it matter — about models whose guess has two numbers in it, or about larger experiments, where every interval here converges on 95%; and nothing about exact conditional inference, the other answer the earlier essay named.

Still open: conditioning on the design the experiment ran #

The bootstraps here either keep the design or repeat it. Exact conditional inference does a third thing: it treats the realised design as the ancillary it nearly is, and asks for an interval whose coverage is right among the experiments that ended at the same settings. On a sequential design whose settings vary as little as these do, the conditional and unconditional intervals should nearly coincide; after a fixed design at a badly wrong guess, where the information the experiment collected varies enormously from one experiment to the next, they should not. Whether conditioning on the information actually observed — the curvature of the residual sum of squares at its minimum — gives an interval that keeps 95% within each stratum of that information, where every interval here averages over strata that cover very differently, is the measurement that would say whether the profile interval’s success is calibration or luck.