Normal Distribution
nothing is called 95% until it has been counted
Concept

Factorial design — where it appears

Varying every factor at once so that each run informs every effect, rather than moving one factor at a time. It is more efficient at the same number of runs and it is the only arrangement that can see an interaction at all.

Named by 13 essays across 5 fields — each of them below, with the objects they name alongside it.

Where a D-optimal design puts its runs. The D-optimal measure over 121 candidate settings on a square region. It keeps 9 of them and discards the rest, and the 9 it keeps are the settings a catalogue would have offered without any of this arithmetic. What the search adds is the weights: 0.1458, 0.0962, 0.0802, which nine equal runs cannot express.

A design is a number

A standard design is taken from a catalogue and then measured. Turn the arithmetic round and a design becomes the answer to an optimisation — and over 121 candidate settings the search keeps nine of them, which are exactly the nine a catalogue would have offered, at weights nine equal runs cannot express.

optimality · Criterion Four runs, and the term they cannot reach. Every run sits at a corner, so x₁² and x₂² are 1 at every run and both columns are copies of the intercept. The normal matrix is singular: the design has no information about curvature at all, and no analysis can recover it.

The design that cannot see a curve

A two-level factorial has every run at a corner, where every squared term equals one — so the column that would estimate curvature is a copy of the intercept, and the design has no information about it at all. A few runs at the centre buy one number back, and only one.

surface · Factorial Twenty walks up the same hill, σ = 2. Each walk fits a plane to the same four-corner factorial, takes its gradient as a direction, and steps along it until a run comes in below the one before. The true optimum is the cross. 80% of the walks stop before the best point on their own path — not because the direction was wrong, but because one noisy run is enough to stop them, and the direction error costs only 3.9% of the available gain.

Walking up the gradient

The fitted gradient is wrong by an angle with a closed form, σ/(|β|√N), and what that angle costs is its squared cosine — twelve per cent at twenty degrees. What costs a third of the gain is not the direction at all. It is deciding where to stop.

surface · Optimum Two factors that interact — where each design looks. The true response at the four corners is -10, 6, 4, 0. One factor at a time visits three of them, sees that raising either factor alone helps, and recommends raising both — a corner it never ran, and one that is worse than either single change. It picks the best corner 0.0% of the time against the factorial design's 99.4%, on the same number of runs.

One factor at a time

Changing one thing per experiment estimates each effect from two conditions; changing everything at once estimates each from every run. The ratio is (k+1)/2 and it is exact — and when two factors interact, the one-at-a-time design recommends a setting it never tried.

design · Factorial A central composite design, 13 runs. Adding 4 axial runs at ±√2 gives every factor three levels, which is the least that can estimate a squared term. The normal matrix now inverts, so each βᵢᵢ has an estimate of its own — and at exactly this axial distance the design is rotatable, which the next figure measures.

Three levels, and the ring where the design says the same thing

A central composite design puts its axial runs at ±α, and α is not a matter of taste. At F to the quarter the prediction variance depends only on how far a point is from the centre and not at all on which direction it lies in — a property with no simulation in it, exact or absent.

surface · Factorial Adding a run can make the design worse. The D-efficiency of the best N-run design at each size, against the optimal measure. It is not a rising curve. 13 runs reaches 99.77% and 14 falls to 99.44%, because the optimal weights are real numbers and N runs is an integer approximation to them, so how good a design can be depends on how well N divides. A Wald interval behaves the same way: a larger sample sometimes makes its coverage worse, for exactly this reason.

The design that has to be integers

The optimal design is a set of real weights and an experiment is a set of runs, so the theory's answer is never available. Thirteen runs reach 99.77% of it and fourteen reach 99.44% — adding a run makes the design worse per run, and the search that finds it does not always find the same one.

optimality · Criterion What a 8-run fraction of 4 factors confounds. The defining relation is I = ABCD, so the resolution is 4. A is estimated as A + BCD; B is estimated as B + ACD; C is estimated as C + ABD; D is estimated as D + ABC. Each of those is an identity about the design rather than an approximation about the data.

The word a fraction costs

A half fraction estimates each main effect as an exact sum of that effect and everything it is confounded with — no error term, no sample-size argument. With every interaction at 0.8 the design reports a true effect of −1 as −0.20, and the design cannot test the assumption that makes the number mean anything.

design · Factorial What the main effects cost the rest of the model. The compound optimum of α·log Ds(main effects) + (1 − α)·log D on an 11-by-11 grid of the square, at 14 weights from α = 0 to 0.999, placed by its D-efficiency against the D-optimal design and its main-effect efficiency against the factorial. The D-optimal design (α = 0) gives the main effects 74.3% of what the factorial gives. At α = 0.5 the design keeps 94.6% of D and gives 85.6%; at 0.9, 64.9% and 96.8%; at 0.99, 31.3% and 99.7%. The factorial itself (open, at the left edge) gives the main effects everything and the model nothing: its D-efficiency is zero. The design optimal for the two quadratic terms (red) gives the main effects 50.0% at a D-efficiency of 83.6%.

A subset the model cannot use

Nominate the two main effects of a quadratic response surface as the parameters of interest and the optimal design is the 2² factorial — four corners, a quarter each — which leaves the full model's information matrix at rank four of six and the curvature with nothing at all. That is why the field's algorithm would not converge on it. Approached through a compound criterion instead, the trade is explicit: 96.8% of the main effects' best information costs the model 35% of its D-efficiency, and the last three points cost nine tenths of what is left about the curvature.

criteria · Criterion The Box–Behnken design in three factors: 15 runs, none at a corner. Twelve runs at the midpoints of the cube's edges and 3 at its centre. Every run holds one factor at zero, so no run puts all three factors at an extreme — which is what makes it runnable where a corner is not. The three panels are the design's coordinate projections, with repeated positions marked.

The design that refuses the corners

Box–Behnken runs three factors in fifteen runs and puts none of them at a corner, which is what makes it usable where a corner cannot be run. It predicts the corner 1.84 times worse than the seventeen-run design that goes there, and 1.31 times worse at the middle of a face, and all three numbers are matrix computations with no simulation in them.

design · Factorial Seven products of three splits. The correlation of every product of three centred median splits with every other, at a Spearman rank correlation of 0.4 between each pair of covariates. Under a Gaussian copula a main effect correlates with the other mains and with the triple (0.273), and with no pair at all, because the third moment is 0.000. Under a Frank copula, whose pairs are radially symmetric, the third moment is 0.078 and every other cell fills in: a main correlates 0.082 with the pair it is not in. In both panels a main is exactly uncorrelated with a pair it belongs to, which is the known zero.

The zero that needed two

A rule balancing two median splits removes nothing of their interaction, on every copula, because a centred split squares to a constant. Add a third covariate and the same arithmetic stops protecting the thing it looked as if it protected. The product of three splits is not orthogonal to the mains: at a rank correlation of 0.4 a rule on the mains removes 14.47% of it under a Gaussian copula. And the symmetry that decides the rest belongs to the triple rather than the pair, so Frank's copula, whose pairs are radially symmetric, leaks where Clayton's, whose pairs are not, barely does.

compound · Adjustment What a stated power leaves of the main effects. The main-effect efficiency of the best design reaching a power of 0.8 at level 0.05, as N|δ|² rises through 50, 60, 70, 80, 100, 130, 170, 220, 300, 400, 600, 1000. a saddle, two-degree test, the compound path: out of reach, out of reach, 0.806, 0.841, 0.881, 0.914, 0.937, 0.952, 0.966, 0.975, 0.984, 0.990. a bowl, two-degree test, the compound path: 0.782, 0.823, 0.851, 0.871, 0.898, 0.923, 0.941, 0.955, 0.967, 0.976, 0.984, 0.990. a bowl, its own one-degree contrast, the compound path: 0.827, 0.859, 0.880, 0.896, 0.918, 0.937, 0.952, 0.964, 0.973, 0.980, 0.987, 0.992. a bowl, a factorial with centre points: 0.914, 0.930, 0.940, 0.948, 0.959, 0.969, 0.976, 0.982, 0.987, 0.990, 0.993, 0.996.

A design chosen for a test

A screening experiment that wants its main effects and a check on curvature has been told to trade them with a weight nobody can state. It can state a power. Read against a power of 80%, the frontier collapses to one curve in runs times the squared size of the curvature, and a large enough trial pays almost nothing. But the curve belongs to a test. A factorial with a few centre points beats the entire frontier on a bowl, spending 4% of its main-effect information where the frontier spends 10%, and against a saddle of the same size it has exactly its 5% level.

criteria · Criterion What one lost run costs a 16-run factorial fitting 11 coefficients. Every run is worth the same: dropping any one multiplies every coefficient's variance by 1.2000, which is 1 + 1/(N − p) with N = 16 and p = 11, and gives every pair of coefficients a correlation of 0.1667 where the complete design had exactly zero.

The run that did not happen

Lose one run from any orthogonal design and every coefficient's variance is multiplied by exactly 1 + 1/(N − p), and every pair of coefficients acquires a correlation of exactly 1/(N − p + 1) where there was none. The price is set by the design's spare capacity and by nothing else, and a saturated design cannot survive it at all.

design · Factorial What the sixteen-run factorial's surviving runs estimate when about 1.3 runs are lost, by why they were lost. A 2⁴ factorial fitting eleven coefficients; true A = 1, B = 0.5, AB = 0.5, everything else 0, noise 1; 20,000 datasets. Runs are lost when the response exceeds 2.5, or at random at the high corner of A and B, or at random anywhere, each losing 1.26 runs on average. Bias of the fit on the surviving runs — lost at random: A 0.001, B 0.003, AB 0.000, C -0.002; lost at the high corner: A 0.003, B 0.001, AB 0.000, C -0.000; lost when the response passes the threshold: A -0.131, B -0.120, AB -0.117, C -0.000.

A run lost for a reason

A sixteen-run factorial that loses a run at random loses precision and nothing else. One that loses a run because the process failed — the response passed a threshold — loses about the same number of runs and biases exactly the effects that pushed the response there: an interaction of 0.5 reads 0.383, a main effect of 1 reads 0.869, and the interval around each still covers about 94%. Treating the lost runs as censored above the threshold overshoots with eleven coefficients and works with six, because a run's own leverage decides how much a censored observation pulls its own estimate.

design · Factorial

Named alongside it

The objects these essays reach for when they reach for this one.

Experimental designCurvatureLeast squaresCentral composite designCentre-pointD-optimalityDesign measurePrediction varianceDegrees of freedomInformation matrixInteractionOptimal design

All concepts