Augmenting a design that has already run

Every design in this field is chosen from nothing. The search is handed a region, a model and a budget, and it returns an arrangement as though no experiment had happened.

That is almost never the situation. A screening factorial has been run and the surface is the next question. Four runs went down one edge because that is where the equipment was. A pilot was done. The question is where to put the next few runs, and the answer has to take the old ones as given.

What changes in the theorem, and it is one term #

Kiefer and Wolfowitz certify a design measure $\xi$ by the condition that the largest prediction variance $d(x) = f(x)'M^{-1}f(x)$ anywhere in the region equals $p$. The derivation is one line: perturb $\xi$ towards a point mass at $x$ and require the directional derivative of $\log\det M$ to be non-positive everywhere.

With runs already spent the object being perturbed is only part of the design. Write $M = \lambda M_{f} + (1-\lambda)M_{\xi}$, where $\lambda$ is the share of the total runs already made and $M_{f}$ is their information. Perturbing $\xi$ changes $M$ by $(1-\lambda)$ times the usual rank-one difference, so the condition becomes

$$\max_{x} d(x) = \operatorname{tr}(M^{-1}M_{\xi}) = \frac{p - \lambda\operatorname{tr}(M^{-1}M_{f})}{1-\lambda}$$

using $\operatorname{tr}(M^{-1}M) = p$, which is an identity. When $\lambda = 0$ it is $p$ and the theorem is the one already stated.

Everything else survives intact. It is still an equality rather than an approximation, it is still “if and only if” and therefore still a certificate, and the search is still a multiplicative update with the target as its denominator.

The first version of this essay’s arithmetic got the target wrong and it is worth saying how, because the error is the natural one. Dropping the fixed part’s own contribution gives $p/(1-\lambda)$, which is the target if the fixed runs carried no information about the model at all. At two added runs that is 18 where the attainable maximum is 7.1, so the search chased a number no design could reach and never converged. The identity $\operatorname{tr}(M^{-1}M) = p$ is what fixes it, and the corrected search reaches its target to 7 × 10⁻¹¹.

The identity that makes it work is worth isolating, because it is the reason the derivation is short and the reason the first attempt was wrong. For any design measure, $\operatorname{tr}(M^{-1}M) = p$ — the trace of the identity, six here — and splitting $M$ into its fixed and new parts turns that one identity into the relation between the two contributions. The target is not derived from the fixed design at all; it is what is left of $p$ after the fixed design has taken its share. That framing is also how the claim that a design’s efficiency is a statement about a region generalises: the share depends on what the other part of the design is, so neither piece has a value on its own.

The bound is a diagnostic #

The changed target is not bookkeeping. It carries the answer to a question the unconditional theorem cannot be asked.

When the target comes back to $p$, the combined design is the unconditional optimum: the new runs have been placed so that what was already spent is exactly the part of the optimal measure those settings carry, and nothing has been lost. When the target is above $p$, the fixed runs cannot be absorbed and the experiment can no longer reach the design it would have chosen.

So the number answers is this experiment still recoverable, and it answers it before the new runs are made:

The last column is the payoff. Wherever the target is $p$, the determinant is the unconditional optimum’s 0.474594 to six digits. Wherever it is above $p$, the determinant is below it, and by how much is exactly what the excess is measuring.

What the recovery costs, in runs #

Reading the table by how many runs each recovery took is the practical form.

A 2² factorial takes four further runs to be fully absorbed. Four runs down one edge — a worse design, concentrated in one corner of the region — takes twelve. A one-at-a-time design of five runs takes eight.

So the cost of a poorly-placed start is not that the experiment is ruined; it is that recovering costs runs in proportion to how badly placed the start was. The edge design needs three times as many additional runs as the factorial to reach the same place, and the one-at-a-time design twice as many.

And every one of those recoveries happens. There is no fixed design in this field’s region that cannot be absorbed given enough further runs, because the optimal measure has full support on nine settings and any finite fixed design is a bounded perturbation of it. The question is always how many, never whether.

The recovery is not gradual #

The last column of the table moves in a way worth looking at, because it is not the shape a “progressive recovery” would have.

After four runs down an edge, the combined design’s determinant is 0.469550 at four further runs, 0.474567 at eight and 0.474594 at twelve, where it stops. The unconditional optimum’s value is 0.474594.

So the recovery is essentially complete by eight added runs — within six parts in a hundred thousand — and the target is still 6.0180 there rather than six. The determinant arrives before the certificate does.

That is the certificate being the more sensitive instrument of the two, and it is the reason to read it rather than the criterion. A determinant six parts in a hundred thousand short is indistinguishable from optimal by any practical measure; a target of 6.0180 against 6 says exactly how much of the fixed design is still not absorbed, and it says it as a number with a scale rather than as a ratio of two very similar values.

An equality is a better diagnostic than a near-equality of two large numbers, which is the general reason the theorem is worth more here than the criterion, and is the same argument the theorem that says when to stop makes.

What conditioning is worth #

The theorem gives an exact answer to the conditional problem. Whether using it is worth anything is a separate question, and the answer is modest.

At two added runs the ignorant design is 84.74% efficient, which in runs is about 18% more of them to reach the same information. At four it is 91.97%, at eight 96.75%, and at sixteen 98.89%.

The gain from conditioning falls away fast, and the reason is that the unconditional optimum is spread over the whole region. It is a good design to add to almost anything, because it covers every direction the model has. Conditioning improves on it by shifting weight away from the directions the fixed runs already cover, and once the new runs outnumber the old there is little left to shift.

That is a genuinely useful negative result. An experimenter with a fitted screening design and a reasonable budget for the surface stage loses under five per cent by ignoring the past entirely, and the loss is largest exactly when the budget is smallest.

What is never worth doing #

The third line in both figures is the option experimenters actually take, and it is not on the same scale as the other two.

Running the same design again — replicating what was already done — gives a combined design supported at the same settings as the original. A 2² factorial visits four settings, the model has six parameters, and four settings cannot determine six parameters however many times they are run. The combined information matrix is singular at every replication count, the determinant is zero, and the efficiency is zero.

That is not a poor design, it is not a design. And it is the default action: the equipment is set up, the protocol is written, and doubling the runs is the cheapest thing to authorise.

The general statement is the one the support count makes: replication buys precision and never buys a setting, so a design that cannot fit a model cannot be made to fit it by running it again. The only repair is a setting the design has not visited.

Where the conditional theorem is the whole method #

One class of design is built entirely out of this arithmetic and is worth naming, because it is where the conditional form stops being a repair and becomes the plan.

A sequentially constructed design places one run at a time: compute $M$ from what has been run, find the setting where $d(x)$ is largest, run it, repeat. Each step is the conditional problem with $\lambda = N/(N+1)$ and one run to place, and the rule “put the next run where prediction is worst” is what the conditional condition reduces to there.

That construction converges to the optimal measure, it needs no search over weights, and it has the property an adaptive design needs: it is valid at every stopping point rather than only at a planned total. What it gives up is that each step is greedy, so a sequence of locally best choices is not the best design of that size — the same gap between a search and an optimum this field measured for exact designs.

What this says about planning in stages #

The whole of this is about a two-stage experiment, and putting the numbers together gives the planning advice rather than the arithmetic.

Stage one should be a design, not a convenience. The three fixed designs here differ by a factor of three in how many further runs they need to be absorbed, and the factorial — which is what a screening stage would be anyway — is the cheapest to recover from. Four runs down an edge is the expensive start and it is the one nobody plans and everybody occasionally has.

The budget should be decided before stage one, not after it. The efficiency of ignoring the past is 84.74% at two added runs and 98.89% at sixteen, so the cost of getting stage two wrong is largest exactly when stage two is small — and stage two is small precisely when the budget ran out during stage one.

And the certificate is worth computing even when the design is not. Placing the extra runs at the unconditional optimum is within five per cent of the conditional answer at any reasonable budget, so the design itself can be looked up. The target, on the other hand, is a number nothing else supplies: it says whether the experiment can still reach the arrangement it would have chosen, and it costs one matrix inverse.

That asymmetry is the useful shape. The hard part of the conditional problem is worth almost nothing and the easy part is worth having, which is the opposite of how the problem is usually presented.

Why the target is above p, read as a quantity #

The excess of the target over $p$ is the essay’s central number and it deserves an interpretation rather than only a value.

Rearranging the condition, the excess is

$$\max_x d(x) - p = \frac{\lambda}{1-\lambda}\bigl(p - \operatorname{tr}(M^{-1}M_{f})\bigr)$$

so it is the share of runs already spent, over the share remaining, times how much less than $p$ the fixed design’s own information contributes to the combined inverse. A fixed design that happened to be a scaled copy of the optimum would contribute exactly $p$ and the excess would vanish at any $\lambda$; one that contributes less leaves an excess in proportion to how much of the experiment it has already used up.

Both factors read the way they should. A fixed design that is a small part of the total — $\lambda$ near zero — cannot do much harm whatever it was, which is the sixteen-added-runs row. And a fixed design that is nearly right contributes nearly $p$ whatever share it is, which is why the factorial recovers in four runs and the edge design takes twelve.

The quantity $\operatorname{tr}(M^{-1}M_{f})$ is worth naming, because it is the fixed design’s contribution measured in the metric of the design it is being combined with rather than on its own. A set of runs is not good or bad in isolation; it is good or bad relative to what else is going to be run, and that is exactly what makes a conditional problem conditional.

What is claimed here, and what is not #

The claim is the equivalence theorem conditional on runs already made: that the stationarity condition becomes $\max d = (p - \lambda\operatorname{tr}(M^{-1}M_{f}))/(1-\lambda)$ and reduces to $p$ when nothing is fixed; that the target equals $p$ exactly when the combined design attains the unconditional optimum, so the number is a diagnostic rather than bookkeeping; that a 2² factorial is absorbed by four further runs, an edge design by sixteen and a one-at-a-time design by eight; that placing further runs at the unconditional optimum is 84.74% to 98.89% efficient; and that replicating the design already run is singular at every replication count.

Every number is a determinant or a prediction variance computed from the designs. The searches converge to their targets to within $10^{-10}$.

What stays out: criteria other than D, where the conditional condition has the same form with the criterion’s own directional derivative and its own average — the three-row table this field already drew would gain a fourth column rather than a new derivation; exact conditional designs, where the new runs must be integers and the measure is not attainable; and the case where the fixed runs were chosen adaptively on the response, which makes the conditioning argument a different one entirely and is what the adaptive field is about.

The efficiencies compare against a measure rather than against an exact design of the same size, so they are upper bounds on what an experimenter with a whole number of runs could achieve. The ordering between the three options is what the essay rests on and it is unaffected.

Still open: a criterion with no derivative #

Every certificate in this field, conditional or not, is built the same way: differentiate the criterion, find the directional derivative, set its maximum equal to its average. Three criteria have been put through it and all three worked.

The fourth does not. E-optimality maximises the smallest eigenvalue of the information matrix — the criterion a reader who cares about the worst-determined direction actually wants — and the smallest eigenvalue of a symmetric matrix is differentiable only while it is simple. At the E-optimal design it is not: two eigenvalues sit at the minimum together, the objective has a corner exactly there, and a search built on a derivative stops at 37.2% of the optimum. That is the criterion with no derivative.

The check, and the refusal #

Two claims are gated and the pair is the point. That the prediction variance under the combined design reaches the conditional bound — required to within $10^{-6}$ of the target on the search’s own candidate set, and to within half a per cent on a grid four times finer, because the optimum over a grid is not the optimum over the square it approximates and pretending otherwise would be a claim about the grid rather than about the design. And that the search stops exactly where the condition says rather than near it, which is what makes the number a certificate and not a tolerance.

The refusals are two, and both are required to fail. The design that ignores what was already run must never be better than the one that conditions on it — an inequality that would catch a sign error in the combination. And the design that repeats what was already run must be worse than either, at every size. It is, by being singular, which is the strongest form that refusal can take: not a worse number but no number at all.