A design is a number

Every design this site has measured came out of a catalogue. A two-level factorial, a factorial with centre runs, a central composite design at α = √2. Somebody worked those out, they are in a book, and the work here has been to compute what each one delivers: the variance it removes, the curvature it can see, the prediction variance it achieves around a ring.

That is the right arithmetic asked in the easy direction. Here is a design; what is it worth?

Ask it the other way — here is what a design is worth; find me one — and the same computation becomes a search. That is the whole of this field, and the first thing to establish is that the question is even well posed, because “find me a good design” is not an instruction a computer can follow until “good” is a number.

Everything is a functional of one matrix #

A design is a list of settings to run. Under a model — here the full quadratic in two factors, with six parameters — a design determines one matrix and everything downstream follows from it.

Write the model vector at a setting x as f(x) = (1, x₁, x₂, x₁x₂, x₁², x₂²). Then the design’s information matrix is

M = X′X / N = (1/N) Σ f(xᵢ) f(xᵢ)′

summed over the runs. Dividing by N is what makes designs of different sizes comparable: M is the information per run, so a design cannot score better simply by being larger.

Everything an experimenter cares about is a function of M and nothing else:

• the variance of any estimated coefficient is a diagonal entry of M⁻¹, over N, times σ²;
• the variance of a prediction at a setting x is f(x)′M⁻¹f(x), over N, times σ²;
• and if M is singular, some coefficient is not estimable at all.

That last one is not a corner case. A two-level factorial with centre runs — a design a great many real experiments use — makes M singular under this model, because at every corner x₁² and x₂² are both 1 and at every centre run both are 0, so those two columns are identical. The design cannot see a curve and the algebra says so before any data is collected.

What the singular case is missing, counted #

The factorial-with-centre-runs example is worth taking one step further, because the algebra says not only that M is singular but by how much.

The six columns are 1, x₁, x₂, x₁x₂, x₁² and x₂². At every corner both squares are 1 and at every centre run both are 0, so x₁² and x₂² are the same column — an indicator for being at a corner. That column is new, so the six columns have rank five, not four.

A deficiency of exactly one. The design determines every linear combination of the six coefficients except one, and the one it misses is the difference $\beta_{11} - \beta_{22}$: it can estimate the sum of the two curvatures perfectly well and cannot separate them.

Which is the same statement the curvature contrast makes from the other direction — one number where the model has two — arriving here as a rank count before any contrast is written down.

Nine points out of a hundred and twenty-one #

Two things about the search’s answer are worth checking against what is known in advance, because a search that returns a known answer is a search that can be trusted with an unknown one.

How many support points there can be. An information matrix in six parameters is a point in a space of dimension $p(p+1)/2 = 21$, so by Carathéodory’s theorem any achievable M is a mixture of at most twenty-one candidate settings, and at least six are needed for M to be non-singular. Nine sits three above the minimum and twelve below the maximum, which is where a well-conditioned answer to this problem should sit.

Whether the grid could have found the right answer at all. The classical optimum for the full quadratic on a square is supported on the 3 × 3 grid — the four corners, the four edge midpoints and the centre — at levels −1, 0 and +1 in each factor.

A hundred and twenty-one candidates is 11 × 11, at a spacing of 0.2, and every one of those nine points is on it. That is not automatic: a 10 × 10 grid over the same square has no point at zero in either factor, so it contains neither the centre nor any edge midpoint, and the search would have had to approximate five of the nine support points with pairs of neighbours.

So the odd-numbered grid is doing more work than its resolution suggests. The relevant property is not that the spacing is fine; it is that the candidate set contains the answer, and for this model on this region that requires an odd number of levels per axis.

Turning a matrix into a number #

M is six by six. A criterion has to compress it to one number, and the choices are not equivalent — that is the subject of a later essay in this field. This one uses the oldest and most common.

D-optimality maximises the determinant of M.

The determinant is a volume. Geometrically, the confidence region for the whole parameter vector is an ellipsoid whose volume is proportional to |M|^(−1/2), so maximising |M| shrinks that ellipsoid. It is a statement about all six parameters at once rather than about any one of them.

In practice the quantity reported is |M|^(1/p) rather than |M| itself, where p = 6 is the number of parameters. Taking the p-th root puts it on a per-parameter scale, so that comparing a design with six parameters against one with three is not comparing a volume with an area.

For the D-optimal design found below, |M|^(1/6) is 0.474594. That number means nothing on its own — its units are the model’s — which is why everything in this field is reported as an efficiency: this design’s value over the best attainable, a number in (0, 1] where 1 is the optimum.

Weights, and the object that is not a design #

There is a distinction here the field cannot do without, and it is worth drawing sharply.

A design is a list of runs: thirteen settings, some possibly repeated, each of which will actually be performed. A design measure is a list of settings with weights that sum to one — 0.1458 of the experiment at this corner, 0.0802 at that edge — and it is not something anybody can run.

The measure is the object the mathematics is about, because optimising over measures is a convex problem with a unique answer and a check for having found it, and optimising over integer run counts is none of those things. The measure is the theory; the design is what an experiment does; and the distance between them is the last essay in this field.

Both have the same information matrix formula, so both go through the same three functions here, and the difference stays where it belongs: in what can actually be run.

The search #

The algorithm is one line, and it is worth setting out because what it does not know is the point.

Offer the search a grid of candidate settings — 121 of them, eleven by eleven across the square. Start with equal weight everywhere. Then repeat:

wᵢ ← wᵢ · d(xᵢ) / p

where d(x) = f(x)′M⁻¹f(x) is the prediction variance at candidate i under the current weights, and p = 6.

Read it as a rule: a setting the current design predicts badly — d above average — gains weight. One it predicts well loses weight. Renormalise so the weights still sum to one, and go again.

The update cannot leave the simplex, so every iterate is a design measure rather than only the limit. It converges in 665 iterations here, and what it converges to is the interesting part.

It is worth noticing what the rule has not been told. It has not been told that corners are special, or that the answer should be symmetric, or that a grid is a sensible thing to end up on, or how many settings to keep. It has been told one thing — put weight where prediction is currently poor — and the arithmetic of that instruction is what discards a hundred and twelve of the candidates. The stopping condition is similarly uninformed: the loop halts when the largest d anywhere falls to p, and nobody told it that p was the right place to stop. That coincidence is not a coincidence, and it is the next essay.

What it returns #

Nine settings out of 121, and they are these:

• the four corners, at weight 0.1458 each,
• the four edge midpoints, at weight 0.0802 each,
• the centre, at weight 0.0962.

The four corners take 58% of the experiment between them, the edges 32%, and the centre 10%.

Those nine settings are the 3² factorial. They are the design an experimenter would have written down without any of this: every combination of low, middle and high on two factors. The search was given a hundred and twenty-one places to put weight, including everything at ±0.2 and ±0.6 and every asymmetric combination, and it put all of it on the nine points of the obvious grid.

That is a genuinely reassuring result and it is the first thing this field has to establish. If optimal-design theory returned exotic settings nobody would run, it would be a mathematical curiosity. It returns the catalogue.

And then says something the catalogue cannot #

The weights are not equal.

A corner is worth 0.1458 of the experiment and an edge midpoint 0.0802 — a factor of 1.82. The 3² factorial as anybody runs it puts one run at each of nine settings, which is weight 1/9 = 0.1111 everywhere, and that is not what the criterion asked for.

The reason is geometric. The quadratic model has to estimate curvature in two directions and an interaction, and the corners are where the interaction term x₁x₂ is largest in absolute value. They carry more information per run about the parameters that are hardest to see, so they are worth more runs. The centre carries almost no information about anything except the intercept and the curvature contrast — but it carries that, uniquely, which is why its weight is not zero either.

Running the nine equally is 97.40% as efficient as the optimum on D’s own scale. That is a small loss and it is a real one, and no amount of reading the catalogue would have revealed it, because a catalogue lists settings and has nowhere to put a weight.

The grid is an approximation, and it says so #

Every number above is computed over a grid of candidate settings, and a grid is not a region. This is the one place where the answer could be an artefact of the machinery rather than a fact about the problem, so it is checked rather than asserted.

Refining the candidate set from eleven points a side to twenty-one — 121 candidates to 441 — leaves the optimal design’s D value at 0.474594, unchanged to six figures, and leaves the support on the same nine settings.

That is the expected result and it is expected for a reason worth stating: the optimum’s support points are corners and midpoints of the region, which are on both grids exactly. A finer grid adds candidates between them, and the algorithm gives those candidates no weight. Had the optimum wanted a setting at 0.6, the coarse grid would have had to round it and the refinement would have moved the answer.

What thirteen runs of it look like #

A measure cannot be run, so the obvious next question is what the nearest thing that can be run looks like. Multiply each weight by N, round, and give the spare runs to the largest remainders.

At N = 13 that gives two runs at each corner, one at each edge midpoint, and one at the centre. Eight plus four plus one, and it happens to come out exactly right — thirteen runs land on the weights better than any other size in the range, reaching 99.77% of the optimal measure’s D value.

The number in each circle is doing the job the weight was doing: an integer count of runs is the only way a real experiment can say “this setting is worth more than that one”. Thirteen runs give it three levels of emphasis — two, one, one — where nine equal runs give it none.

Whether rounding the weights is the best way to get from a measure to N runs is a separate question with a surprising answer, and it is the subject of the last essay in this field. For now it is enough that the gap between what the criterion asked for and what an experiment can do is a real gap with a size.

The same search somewhere else #

Nothing above depended on the region being a square, and it is worth seeing the search run once on a different one, because the region turns out to matter more than anything else in this field.

The optimum on a disc is not the 3² factorial re-scaled. It is supported on the boundary circle and the centre, because the corners of the square — the settings carrying the most information about the interaction — are not available. A region is not a detail of the setup; it is half the problem, and an essay in this field is about how completely it reverses the ranking of two designs anybody would recognise.

What the criterion is not #

Two things worth saying before the field goes further, because “optimal” is a word that invites more belief than it has earned.

It is optimal for a model. Every number here is computed for the full quadratic in two factors. A design chosen to estimate six parameters as precisely as possible is, by construction, a design that spends nothing on detecting that those six parameters are the wrong six. The 3² factorial has nine distinct settings and six parameters, leaving three degrees of freedom for lack of fit; a design with fewer distinct settings and more replication would score better on D and have nothing left to check the model with. The criterion cannot see that trade, because model adequacy is not a function of M.

And it is optimal for a criterion that was chosen. D is a volume of a confidence ellipsoid, which is a sensible thing to want small and is not the only sensible thing. An experimenter who cares about predicting well across the region rather than about estimating coefficients precisely is asking a different question, and will get a different design — with two and a half times the weight at the centre, as it turns out.

What is being claimed here, and what is not #

This field takes optimal-design theory as a criterion — the functionals of X′X, the search that optimises them, the theorem that certifies the answer, and the gap between a measure and a design an experiment can run. That was named as explicitly not claimed when the response-surface field was written, and it is claimed now.

What stays out, and is named so a later phase knows it was a decision: mixture designs, where the factors sum to a constant and the region is a simplex; designs for non-linear models, where M depends on the parameters being estimated and the whole problem changes character; blocked and split-plot optimal designs; and the industrial-practice literature about how any of this is used. None of those is measured here.

The boundary against numerical-linear-algebra.com is the one the response-surface field already drew and this one stays inside. Determinants and inverses are computed here, in six dimensions, and none of it is a claim about conditioning or about numerical method. A design is the subject; the linear algebra is apparatus.

The check, and the refusal that makes it mean something #

Two claims are gated in this field’s own library, and they are the two halves of this essay. The optimum’s support is exactly nine settings and they are exactly the 3² factorial’s — asserted by testing every supported coordinate against 0 and ±1 rather than by comparing to a stored list, so a search that drifted onto a different set of nine would fail. And the weights are not equal, asserted as a ratio above 1.4, so a future change that quietly equalised them would be caught.

The refusal is the design that cannot fit the model at all. A two-level factorial with centre runs is in the comparison table throughout this field, and it scores zero on every criterion, because its information matrix is singular and the inverse the criteria need does not exist. That is not a failure of the search; it is the search reporting, correctly, that the question has no answer for that design — and a table in which every row scores something would be a table that had quietly repaired the one design whose whole interest is that it cannot be repaired.