The design that hedges

The previous essay ended with a design that is perfect at one value of a number nobody has and 23.9% efficient at the edge of a sixteenfold range around it. That is not a defect in the design; it is what “locally optimal” means, and the local design is doing exactly what it was asked.

The question is what to ask instead. And the honest answer is that the experimenter has more information than the guess: they have a sense of how wrong the guess might be. A design can use that, and the two standard ways of using it are the two ways of summarising a range — average over it, or protect against its worst point.

Averaging the criterion #

The construction is one line. Instead of maximising log|M(ξ, K)| at a guessed K, maximise its average over a prior:

maximise Σₖ πₖ · log|M(ξ, Kₖ)|

The directional derivative averages the same way — δ(t) = Σₖ πₖ·f(t,Kₖ)′M(Kₖ)⁻¹f(t,Kₖ) — so the same multiplicative algorithm runs unchanged with the derivative summed over the prior. Nothing new is required and nothing about the equivalence theorem changes: the search still stops when max δ reaches the number of parameters.

The choice of log|M| rather than |M| is not cosmetic. Averaging determinants would let one value of K with a large determinant dominate the average; averaging their logarithms is averaging on a scale where a factor of two counts the same wherever it happens, which is what a criterion that is trying to be uniformly acceptable ought to do. It is the same reason D-efficiency is reported as a p-th root rather than as a raw determinant.

What averaging the logarithm actually costs the alternative #

“Not cosmetic” is worth a number, because Jensen’s inequality says exactly how far apart the two criteria can be.

Take two values of K at which a design’s determinant is 1 and 100. Their arithmetic mean is 50.5; the mean of their logarithms corresponds to a geometric mean of 10. A factor of five, produced by a design that is superb at one value and useless at the other.

So maximising an average of determinants would hand the prize to exactly that design, and maximising an average of logarithms would not. Averaging on the log scale is averaging ratios rather than levels, and since $|M|^{1/p}$ is the quantity a D-efficiency is a ratio of, the averaged criterion is literally an average of log-efficiencies across the prior.

The stopping rule is also a certificate #

One more property of the equivalence theorem is worth stating because it is free and the algorithm has already computed it.

The search stops when $\max_t \delta(t)$ reaches p. Before it stops, the same quantity bounds how far from optimal the current design is: any design satisfies

$$\text{D-efficiency} \;\ge\; \frac{p}{\max_t \delta(t)} .$$

So on six parameters a design whose maximum directional derivative has fallen to 6.3 is guaranteed at least 95% D-efficient, without the optimum ever being computed.

That matters more for the averaged criterion than for the plain one, because the averaged optimum is not in any catalogue and there is nothing to check the answer against. The bound is the check, it holds at every iteration, and it holds for the averaged criterion for the same reason the theorem does — the derivative averages, and the stopping value stays p.

What comes out is a wider design, and the width is an integer #

The interesting result is not the efficiency. It is the shape.

A locally optimal design for this two-parameter model has two support points. The hedged design has more — and how many more is an integer that changes at particular prior widths.

• A prior reaching a factor of three either side of the guess is still answered by two settings — the same two the local design uses, shifted.
• The third setting arrives at a spread of 3.36.
• The fourth at 8.86.

Nothing put those thresholds in. They were found by bisecting on the design the algorithm returns, and they are properties of the model and the range rather than choices.

This is the concrete content of “hedge the guess”, and it is more specific than the phrase suggests. Hedging does not mean spreading runs out generally. It means spending runs at settings that would be right if the guess were wrong, and a two-point design has nowhere to put them — so the first thing a prior buys is not better weights, it is another place to stand.

At a spread of four the design is 0.246 of the runs at t = 0.42, 0.283 at t = 1.53, and 0.471 at the end of the range. The middle point is close to where a local design at the guess would have put its lower run; the new point at 0.42 is where a local design would have put it had K been four times smaller. The design is, quite literally, hedging: it is running part of somebody else’s experiment.

What it buys and what it costs #

Both designs scored across the range the prior claims, which is the only fair comparison and is a correction this field had to make — scoring a design hedged against a factor of sixteen on a range that only reaches four charges the hedge for runs spent where the comparison never looks, and makes hedging appear worse the more of it is bought.

Three readings.

At a factor of two, hedging buys nothing. The local design’s worst case is already 90.2% and the hedged design’s is 90.5%, and it costs nothing at the guess because the design has not changed. An experimenter who knows the answer to within a factor of two should stop reading here and run the local design.

At a factor of sixteen it more than doubles the worst case, 23.9% to 56.3%, and costs 31.3 points at the guess. That is a real trade and it is a trade an experimenter can make: the local design is better if the guess is right and catastrophic if it is not, and which of those matters is a question about the consequences rather than about the arithmetic.

And what it costs at the guess grows with the hedge. 0.0, 10.4, 27.8, 31.3 points as the prior widens. Hedging is not free and it is not cheap at the widths where it is most needed.

The comparison this essay had to fix #

There is a correction worth recording, because it changed the sign of the conclusion and it was made by the arithmetic rather than by review.

The first version of the comparison scored every design across a fixed range of true values — the same seven values of K whatever the prior said. That is the obvious thing to write and it produces a table in which hedging looks worse the more of it is bought: at a spread of sixteen the hedged design came out at 53.8% worst case against the local design’s 66.7%, which reads as a clear recommendation not to hedge.

It is not a comparison. A design hedged against a factor of sixteen has spent runs at settings that would be right if K were sixteen times smaller, and a scoring range reaching only a factor of four never looks there — so the hedge is charged for the runs and credited with none of their purpose. The range scored has to be the range the prior claims, and once it is, the same numbers run the other way: 23.9% against 56.3%, and hedging wins by more the wider the prior gets.

The general form is one this site keeps meeting. A comparison between two procedures has to be scored on the situation both of them were built for, and a fixed evaluation range is a silent third assumption that belongs to one of them. It is the same error as comparing an adaptive design’s power against a fixed design’s without holding the two to the same size, which the exact field had to make explicit before its power comparison meant anything.

The prior is a real input and it is not a small one #

A prior that is wrong is worse than no prior, and the arithmetic above says how.

The hedged design is optimal for the range it was told about, and the range is a claim. Told a range of four when the truth is at a factor of ten out, it is a locally optimal design for the wrong neighbourhood plus some scatter — better than a point guess, worse than a design that had been told the truth. There is no width that is safe, because widening the prior is not free either.

This site has measured the analogous thing directly in the Bayesian hierarchical field, and the result there is the one worth carrying: three proper priors on the population spread span 54% of the widest answer at small τ and still 15% at large, so the prior never stops mattering at eight groups. The reassuring story — the data takes over and the prior washes out — is true about whether a difference exists and false about how large it is.

Here it is worse in one respect and better in another. Worse, because the prior is used before any data exists, so there is no data to take over from it; the design is a function of the prior and nothing else. Better, because the consequence is an efficiency rather than a coverage — a design built on a poor prior wastes runs and does not produce an interval that lies about its own coverage.

There is a second difference and it decides how much of this an experimenter should worry about. In the hierarchical field the prior is part of the analysis, so a reader of the published interval inherits it whether or not they agree with it, and the only defence is to report the sensitivity. Here the prior is part of the design, and it is discharged the moment the runs are taken: the analysis of the resulting data need not mention it, the estimate is whatever least squares says, and a reader who thinks the prior was wrong is entitled to say so and has lost nothing but efficiency.

That makes a design prior a considerably softer commitment than an analysis prior, and it is the strongest argument for using one. The cost of a prior in an analysis is measured on this site in observations — a prior is worth a stated number of them, and it moves the answer. The cost of a design prior is measured in efficiency and moves nothing at all, because by the time anybody reads the result the design has already happened.

What a support point is worth, and why the answer is discrete #

The staircase deserves more than a paragraph, because a quantity that changes in integer steps is unusual on this site and the reason it does is instructive.

A design measure is a continuous object: weights are real numbers and can be moved by arbitrarily small amounts. So it is not obvious that anything about the answer should be discrete. What makes the support count an integer is the equivalence theorem, which says the optimum puts weight only where the directional derivative reaches its maximum. As the prior widens, the derivative’s shape changes continuously — and a second local maximum rises continuously towards the ceiling until, at one particular width, it touches. Before that moment the setting gets zero weight; after it, positive weight. The weight is continuous in the width; the support is not.

That also says what happens either side of a threshold, and it is the reassuring part. Just past 3.36 the third support point exists and carries almost nothing, so the design there is barely different from the two-point design just before it. The staircase is a discontinuity in the description rather than in the experiment, and an experimenter who mis-specifies the prior slightly near a threshold gets a design that is slightly different rather than a different design.

Where the runs actually go #

It is worth reading the hedged design’s settings rather than only its count, because they say what it is protecting against.

At a spread of four, the three settings carry 0.246, 0.283 and 0.471 of the runs, at t = 0.42, 1.53 and the end of the range. The local design at the guess would have run at 0.83 and the end of the range, half and half. So the hedge has done three things: it has kept the run at the end of the range and given it more weight, it has moved the lower run outward, and it has added a run well inside where the lower one was.

The run at the end of the range surviving is the least surprising and the most useful. It is there for the asymptote, which is the parameter whose best measurement point does not depend on K at all — so it is the one part of the design that needs no hedging, and the criterion recognises that by loading it. The hedge is concentrated entirely in the part of the design that was sensitive to the unknown, which is what a well-behaved answer should look like and is not something the algorithm was told.

The two lower points bracket the local design’s single lower point, which is the geometric statement of what averaging did: one setting that is right at K = 1 has been replaced by two that are right at K = 0.25 and K = 1.5 respectively, and the average of the criterion over the prior prefers the pair.

Protecting the worst point instead #

Averaging is one of the two summaries and the other is a maximin criterion: maximise the minimum efficiency across the range, rather than the average of the criterion.

It is the natural thing to want, and it is the harder of the two for the same reason E-optimality is harder than D: a minimum over a range is not a differentiable function of the design, so the multiplicative algorithm has nothing to follow. The two problems are the same problem — a maximin design is the E of this field — and the family’s lesson transfers unchanged: the criterion whose statement is simplest is the one whose optimum sits on a corner.

What is measured here is the averaged design’s worst-case efficiency, which is a lower bound on what a maximin design would achieve. The 56.3% at a spread of sixteen is therefore a floor rather than the answer, and how much a maximin design would improve it is not measured.

What is being claimed here, and what is not #

This essay claims Bayesian D-optimality on one two-parameter non-linear model: the averaged criterion, the support count as a staircase in the prior’s width with the two thresholds located by bisection, and the worst-case efficiencies against the local design at four widths.

What stays out: maximin designs, named above and not built; priors that are not three-point discrete approximations, which change the numbers and not the shape; and sequential design, where the prior is updated between runs and the whole problem becomes a different one. Sequential design is the obvious neighbour of everything in these two essays and is the reason the boundary needs stating: it is not a better hedge, it is a different experiment, with an error-rate question of its own that the adaptive field would recognise immediately.

The checks #

One claim is gated in this field’s library and it has three parts, because any one of them alone would pass for the wrong reason.

The hedged design has more support points than the local one — which fails if the prior is too narrow, and is therefore checked at a width where the design is known to have three.

Its worst efficiency over the range is materially better, by more than five points, which is the part that says the extra points are doing something rather than merely existing.

And it gives something up at the guess, asserted as an inequality in the other direction: the hedged design has to be below the local one at the guess and above 85%. A hedge that cost nothing would be a hedge that had not changed the design, and a check that only ever confirmed the first two parts would pass for a design that had simply been given more runs.