A design chosen for a test

A subset the model cannot use found the problem with naming the main effects of a quadratic surface as the thing cared about. The optimal design for them alone is the $2^2$ factorial, four corners with a quarter of the runs on each, which gives the two main effects everything and has no information at all about curvature. So it introduced a compound criterion, with a weight $\alpha$ on the main effects and $1 - \alpha$ on the rest of the model, and traced what the trade looks like. Raising main-effect efficiency from 74.3% to 96.8% costs 35 points of D-efficiency, the centre leaves first, and the edges stay last.

The essay’s last paragraph put its finger on what is wrong with that. Nobody can state an $\alpha$. It is a number on a scale of log determinants, with no interpretation outside the optimisation it was invented for, and a statistician handing a client a compound criterion will be asked what 0.75 means and will not have an honest answer. But a screening experiment is usually followed by a test. Is curvature present? A test’s currency is power against a stated alternative, and a client can state one: “a curvature of about this size, at 80%”.

So the frontier can be read in a different unit. For a stated power against a stated curvature, find the design with the most main-effect information that delivers it. It is a short calculation.

It also leads somewhere the question did not expect, because the answer depends on what the test is for.

One curve #

With a known error variance, the two curvature coefficients have an estimator whose information, after everything else in the model, is a matrix $C$ read directly off the design. A test of both at level 5% then has power $P(\chi^2_2(\lambda) > 5.99)$, with noncentrality

$$\lambda = \frac{N}{\sigma^2}\,\delta' C \delta,$$

for $N$ runs and a curvature $\delta$. For 80% power, $\lambda$ has to be 9.63. That is the whole of the test. Everything else is the design’s $C$ along the compound path, and all the design needs from the trial is the product $N|\delta|^2/\sigma^2$. So the frontier is not one curve per sample size and effect size, it is one curve in one number. Twenty runs and a curvature of two, and eighty runs and a curvature of one, are the same problem.

The figure at the head of this essay is that curve, for three things a trial might want to detect. Reading the two-degree test first, for a bowl, with the same curvature in both factors. Below an $N|\delta|^2$ of about 44, no point on the path reaches the power at all, because the D-optimal design at one end of the path has only so much curvature information to give, at 0.221 per unit. At 50 the design has main-effect efficiency 78.2%. At 80 it has 87.1%, at 100 it has 89.8%, at 200 it has 95.0%, at 500 it has 98.1%, and at 1,000 it has 99.0%.

So a trial with a good deal of signal to detect loses very little. And the curve has a simple tail: past about 200, main-effect efficiency falls short of one by about $9.63 / N|\delta|^2$. Each extra doubling of signal halves what the check on curvature costs.

What the weight was #

This also says what $\alpha$ meant.

On a bowl, $\alpha = 0.5$ is an $N|\delta|^2$ of about 70, $\alpha = 0.75$ is about 130, $\alpha = 0.9$ is about 300, and $\alpha = 0.95$ is about 600. So a weight has an interpretation after all. A half means “a trial in which a curvature of about one standard deviation, on seventy runs, should be detected four times in five.” Any statistician can have that conversation with a client. The numbers on the path do not change, since it is the same path with a new label.

The saddle needs a slightly higher $N|\delta|^2$ for the same weight, and it cannot be reached at all below about 60: at 80 it has $\alpha$ of 0.44 and main-effect efficiency of 84.1%, against the bowl’s 0.55 and 87.1%. So a design has more difficulty seeing a saddle, and the reason is in where the runs go.

Where the curvature information comes from #

Along the path, the share of runs at the corners rises from 58.3% to about all of them, and the centre goes first. It falls from 9.6% at the D-optimal end to 3.1% at $\alpha = 0.5$, under one per cent at 0.75, and to zero by 0.95. The edges follow, from 32.1% to 6.1% at 0.9.

So when the path is up in the region a well-powered trial wants, it is running a factorial plus a handful of edge midpoints, not centre points. That is unlike what anyone is usually taught, and it is not arbitrary. A centre point only sees the sum of the two curvatures: it compares a point with $x^2 + y^2 = 0$ with the corners, where it is 2. An edge midpoint sees them separately, because it has $x^2 = 1$ and $y^2 = 0$ and vice versa. The compound criterion’s goal includes the whole model, so it needs both coefficients, and hence edges.

That explains the shape difference between a bowl and a saddle. A saddle has its two curvatures cancelling at the centre, and a centre point cannot see it. So on a path with centre points still on it, a bowl has about 1.4 times the information of a saddle, 0.221 against 0.160 at the D-optimal end. Once the centre’s gone, at about $\alpha = 0.9$, they are equal to within a few per cent, 0.0317 against 0.0307, because only the edges are left and they treat both alike.

The design nobody put on the path #

And that leads to the awkward part. The design anyone with a screening experiment would actually use is not on this path at all. It is a factorial with a few replicated centre points, which is in every textbook, and it has a one-degree test for curvature: compare the mean of the corners with the mean of the centre. Giving a share $w$ of the runs to the centre, main-effect efficiency is $1 - w$, and the contrast’s noncentrality, on a bowl of unit norm and size $d$, is $2Nd^2 w(1 - w)$. So 80% power needs a contrast noncentrality of 7.85, and everything has a closed form.

It is the blue line in the first figure. It lies above the whole compound frontier, and by a lot. At $N|\delta|^2 = 100$ it has main-effect efficiency of 95.9%, against 89.8% for the compound path’s two-degree test on a bowl. At 44, where the path has only just come into reach at 74.6%, it has 90.1%. Even the path’s own one-degree contrast along the bowl direction reaches only 91.8% at 100. So a practitioner reading the compound frontier’s account of what a curvature check costs has been told a much higher price than the one the commonest design in the field actually pays.

The commonest design is not dominated by the design theory built to replace it. That is the result this measurement was not looking for, and it is a large one. The reason is that a centre-point design is buying only what it is asked for: one contrast, in the direction of a bowl. It uses a share of its runs that is about half what the same power takes from a design that has to pay for two coefficients.

Yet that cheapness has a price, and it is in what the design cannot see.

Consider two designs at about the same main-effect efficiency, at $N|\delta|^2 = 100$. The factorial with 4.1% of its runs at the centre has a power of 80.0% against a bowl, 50.8% against a ridge, and 5.0% against a saddle. That last number is the test’s level. It is not low power, it is no power: the contrast compares corners and centre, and on a saddle $x^2 - y^2$ takes the value zero at both. No trial size fixes it.

The compound design of about the same main-effect efficiency, 96.8%, has much less power against a bowl, 33.8%, because it is only carrying a sliver of edge points, and about the same against every shape: 33.3% against a ridge and 32.8% against a saddle. It is a design that does not care which curvature turns up, and a design that does not care has to pay for all of them.

In the units a protocol uses #

It helps to put this into run counts. Take a curvature of 1.5 error standard deviations in the norm of the two coefficients on the coded square, a fair guess at something a screening trial would want to catch. Then $N|\delta|^2/\sigma^2$ is $2.25N$, and 80% power on a bowl comes at these costs:

Two things stand out. Thirteen runs cannot reach this power on the compound path at all, whereas the factorial with centre points can, at some real cost to its main effects: a small trial gets a usable curvature check only by giving up on the rest of the model. And the number of centre points barely moves with the trial. Two of them do about the same job at twenty runs and at eighty, because what the contrast needs is a share of a small absolute number of observations at the centre. That is the practitioner’s habit of “three or four centre points, whatever the size of the factorial”, and it is a better rule than it looks, though it should be checked against the pure error it also supplies.

One factor at a time and what a fraction of a factorial costs priced what a screening design gives up in main effects and interactions. This is the same accounting applied to a curvature check, and three levels and a ring is what the trial turns into once the check comes back positive.

Which test a screening trial is running #

So the question that replaces $\alpha$ is not just “what power?”. It is “against what?” Three cases are worth writing out.

If curvature, if it exists, is expected as a bowl or a dome, because the response has one optimum in the region, which is the classic situation for a response surface, then factorial plus centre points is right, cheap, and the frontier should not be used. It needs only a closed form: $w(1 - w) = 3.92/N|\delta|^2$.

If the shape of curvature is unknown, or a saddle is a real possibility, the centre points are not protecting anything. A trial that writes “no curvature was detected” after a centre-point contrast has said nothing about a saddle, and the protocol should say so. A small number of edge points fixes it. By the frontier’s own reckoning at 200, it costs about 5% of main-effect information, in exchange for roughly uniform sensitivity.

If what is wanted is both coefficients estimated, and not merely a test, the frontier is what it always was, and the path is right, along with the design that hedges if the model itself is uncertain, and the height at the chosen setting if the aim is a prediction. A confirmed curvature has to be modelled, and that requires knowing what it is.

The earlier essay’s phrase was that a compound weight is a statement of how much the main effects matter. It is more useful to say it is a statement of what the trial is trying to detect, and that the design choice follows without needing the weight at all.

There is also a reporting consequence, for an analysis that has already been run. A screening trial with centre points that reports “no evidence of curvature” has a specific meaning, which is no evidence of a bowl, read through one contrast. The word a fraction costs made the same observation about an alias chain, and the phrase should be held to the same standard: the sentence is only as broad as what the design could have seen.

How the numbers were made #

The model is a full quadratic in two factors on the square, with a grid of $11 \times 11$ candidate points. The compound path is solved at each of ten weights by the multiplicative algorithm from the earlier essay, to a gap of about $10^{-9}$ except at the last point. $C$ is read off the information matrix by the same partition. Power is the noncentral chi-square’s tail, as a Poisson mixture of central ones, with the critical value from the two-degree or the one-degree central law. Reported efficiencies are interpolated in the compound weight between neighbouring points of the path.

Two assumptions should be stated. The error variance is known throughout. At small $N$, a test using an estimated variance on few residual degrees of freedom has less power, and these figures flatter small trials, for both designs. And the path is at the level of a design measure: a real design rounds to whole runs, and at 13 runs a share of 3% is not a thing that can be had. So what is read here is the shape of the trade, and a real plan should check its own rounded design with the same closed form.

Four things are required. At 100, all three of the bowl, the saddle and the centre-point designs must reach the power. The saddle must cost more than the bowl. The centre-point design must beat the compound frontier on a bowl. And the claim that centre points detect a saddle is refused: their noncentrality against one is exactly zero, and the reported power is the level.

Still open: estimated variance, and a few edge points on a factorial #

How small a trial this is honest for. Replacing the known variance with one estimated from a pure-error term has a noncentral $F$ in place of the chi-square. That changes small trials a great deal, and it changes the comparison too, since centre points give pure error for free. A centre-point design has replicates built in, and an edge-point design may have none. That is a real advantage for centre points that does not appear in this essay’s frame, and it might move the recommendation at very small $N$.

The hybrid. The obvious design that the frontier and the textbook between them leave out is a factorial plus centre points plus a very few edge points: cheap on a bowl and not blind to a saddle. It is not a point on any path here, and the arithmetic of it is a two-parameter family with a closed form. Finding the point of that family that holds 80% on a bowl and some stated power on a saddle, at the least cost in main effects, is exactly the question that a client would ask next.

Which curve a trial is on #

This essay takes the compound design for a quadratic surface, read against a stated power for a test of curvature. Its claims are that all of this depends only on $N|\delta|^2/\sigma^2$; that for 80% power the compound path has main-effect efficiency 87.1% on a bowl at 80, 95.0% at 200 and 98.1% at 500; that a compound weight of a half corresponds to a bowl at about 70, and three quarters to about 130; that a saddle costs slightly more, 84.1% at 80; that a factorial with centre points has 94.8% at 80 and 98.0% at 200 on a bowl, beating the whole frontier; and that against a saddle of the same size that design has power exactly equal to its level.

What stays out, as a decision: an estimated error variance, named above; more than two factors, where a centre-point contrast reads the sum of many curvatures and can cancel in more ways; and any statement about which curvature an experiment should expect, which belongs to the experimenter. The boundary against a subset the model cannot use is that it asked how to trade information, and this one replaces that with a question somebody can answer, and finds the answer depends on a test the trade never mentioned. The boundary against the design that cannot see a curve is that it showed what a factorial alone cannot see, and this one prices what the commonest repair sees and does not.