Normal Distribution
nothing is called 95% until it has been counted
Concept

Prediction variance — where it appears

The variance of a fitted model's prediction at a setting, scaled by the number of runs so that designs of different sizes can be compared. Minimising its largest value over a region is G-optimality, and minimising its average is a different criterion with a different answer.

Named by 10 essays across 3 fields — each of them below, with the objects they name alongside it.

Where a D-optimal design puts its runs. The D-optimal measure over 121 candidate settings on a square region. It keeps 9 of them and discards the rest, and the 9 it keeps are the settings a catalogue would have offered without any of this arithmetic. What the search adds is the weights: 0.1458, 0.0962, 0.0802, which nine equal runs cannot express.

A design is a number

A standard design is taken from a catalogue and then measured. Turn the arithmetic round and a design becomes the answer to an optimisation — and over 121 candidate settings the search keeps nine of them, which are exactly the nine a catalogue would have offered, at weights nine equal runs cannot express.

optimality · Criterion The variance touches p and never crosses it. d(x) = f(x)′M⁻¹f(x) along the diagonal of a square region, for the D-optimal measure. The line at 6 is the number of parameters in the model. Kiefer and Wolfowitz's theorem says a design is D-optimal exactly when the largest d anywhere in the region is p — not approximately, equals — so the optimal curve is tangent to that line at its support points and below it everywhere else. Here the largest value anywhere on a 41×41 grid is 6.000000000.

The theorem that says when to stop

A search that maximises the volume of the information has no way of knowing it has finished, because nothing tells it what the maximum is. Kiefer and Wolfowitz's equality does — a design is D-optimal exactly when the worst prediction anywhere in the region equals the number of parameters, which is 6.000000000059 here, gated at machine precision.

optimality · Equivalence Four criteria on 5 designs, on a square region. Each design scored as an efficiency — its value over the best attainable — so four criteria in as many different units sit on one scale where 1 is the optimum. Rows are ordered by D. D picks 13-run exchange; A picks face-centred composite; G picks 13-run exchange; I picks face-centred composite. Every design has been scaled to just fit the region first, because a design run at settings the region does not contain is not a competitor on it. The disagreement is the point: the letter is a choice, and it is almost never reported as one.

Four letters and two camps

D, A, G and I are four ways of turning one matrix into one number, and they do not agree. The design that wins on D is the worst thing here on I. And the same two designs swap places entirely when the region changes from a square to a disc — on all four criteria at once.

optimality · Criterion A central composite design, 13 runs. Adding 4 axial runs at ±√2 gives every factor three levels, which is the least that can estimate a squared term. The normal matrix now inverts, so each βᵢᵢ has an estimate of its own — and at exactly this axial distance the design is rotatable, which the next figure measures.

Three levels, and the ring where the design says the same thing

A central composite design puts its axial runs at ±α, and α is not a matter of taste. At F to the quarter the prediction variance depends only on how far a point is from the centre and not at all on which direction it lies in — a property with no simulation in it, exact or absent.

surface · Factorial Adding a run can make the design worse. The D-efficiency of the best N-run design at each size, against the optimal measure. It is not a rising curve. 13 runs reaches 99.77% and 14 falls to 99.44%, because the optimal weights are real numbers and N runs is an integer approximation to them, so how good a design can be depends on how well N divides. A Wald interval behaves the same way: a larger sample sometimes makes its coverage worse, for exactly this reason.

The design that has to be integers

The optimal design is a set of real weights and an experiment is a set of runs, so the theory's answer is never available. Thirteen runs reach 99.77% of it and fourteen reach 99.44% — adding a run makes the design worse per run, and the search that finds it does not always find the same one.

optimality · Criterion What visiting fewer settings costs, 6 parameters. Carathéodory's bound puts the support of an optimal measure between 6 and 21. 6 settings: D-efficiency 88.90%, G-efficiency 57.18%, 0 degrees of freedom for lack of fit; 7 settings: D-efficiency 94.54%, G-efficiency 61.22%, 1 degrees of freedom for lack of fit; 8 settings: D-efficiency 95.99%, G-efficiency 64.60%, 2 degrees of freedom for lack of fit; 9 settings: D-efficiency 97.40%, G-efficiency 82.76%, 3 degrees of freedom for lack of fit. The saturated design has none, and buying the first one costs about five points of efficiency to get back.

How many places a design goes

Carathéodory's bound puts an optimal design's support between six and twenty-one settings, and every design in this field that can fit the model visits exactly nine. The count is not a choice anybody makes, it decides how many degrees of freedom are left to check the model with, and the first spare setting costs six points of efficiency to get back.

optimality · Equivalence The Box–Behnken design in three factors: 15 runs, none at a corner. Twelve runs at the midpoints of the cube's edges and 3 at its centre. Every run holds one factor at zero, so no run puts all three factors at an extreme — which is what makes it runnable where a corner is not. The three panels are the design's coordinate projections, with repeated positions marked.

The design that refuses the corners

Box–Behnken runs three factors in fifteen runs and puts none of them at a corner, which is what makes it usable where a corner cannot be run. It predicts the corner 1.84 times worse than the seventeen-run design that goes there, and 1.31 times worse at the middle of a face, and all three numbers are matrix computations with no simulation in them.

design · Factorial The certificate when 4 runs are already spent. a 2² factorial has already been run and 2 further runs are to be placed. The stationarity condition is no longer max d = p; it is max d = (p − λ·tr(M⁻¹M_fixed))/(1 − λ) with λ = 0.6667 the share of runs already spent, which is 7.0985 here. The search reaches 7.098508790 against it, and the largest value anywhere on a 41×41 grid is 7.098508790.

Augmenting a design that has already run

The equivalence theorem still certifies when some runs are already spent, and one number in it changes: the bound is no longer p but (p − λ·tr(M⁻¹M_fixed))/(1 − λ). It equals p again exactly when the runs already made can still be absorbed into the design that would have been chosen — so the certificate says whether the experiment is still recoverable.

optimality · Equivalence The ridge, when the fitted optimum is outside the region. One fitted surface. Its stationary point is at a radius of 2.289 and the fit calls the shape a maximum. The ridge is the best setting at each radius, found by the Lagrange condition (B̂ − μI)x = −ĝ/2; the fitted response rises along it from 59.93 at the centre to 62.10 at the edge. The true optimum is at (0.4, 0.3).

When the best setting is outside the region

On a flat surface at twice the noise the fitted optimum lands outside the experimental region on 24.9% of studies and more than three coded units out on 11.8%. The answer is a ridge — the best setting at each radius, with a closed form — and the two obvious rules for using it turn out to be within four per cent of each other.

surface · Optimum What a confirmation run at the chosen setting would find. The true optimum is worth 62.348. At σ = 2 the fit predicts 62.679 at the setting it recommends and the truth there is 61.821 — a gap of 0.858, which is 0.72 of the prediction's own standard error. The setting itself gives up 0.527 against the best available.

The run that confirms it

The setting a response-surface analysis recommends was chosen because the fitted surface was highest there, so the height the fit predicts at it is a maximum over a random field. At twice the noise the fit predicts 0.858 more than is there — 0.72 of the prediction's own standard error — and the gap is not noise, it is the selection.

surface · Optimum

Named alongside it

The objects these essays reach for when they reach for this one.

Experimental designCentral composite designD-optimalityInformation matrixOptimal designDesign measureFactorial designResponse-surfaceCurvatureG optimalityRotatabilityCentre point

All concepts