a-design-against-its-guarantee

Two designs for one model, and the weights are not equal

Where the runs go, for the same two-parameter model at K = 1 and a ceiling of T = 10. The D-optimal design for both parameters is the familiar one: half the runs at 0.8333 and half at the ceiling. The design for the half-saturation constant alone moves the lower setting down to 0.6040 — where the response curve is still bending, which is where K is visible — and, unlike every design in the two fields before this one, it does not split the runs evenly: the weights are exactly 1/√2 and 1 − 1/√2, 0.7071 and 0.2929, at every K, V and T. The equal weights of D-optimality were a consequence of asking about both parameters at once, and nobody had to notice while that was the only question being asked.

What the design is asked to guaranteeslider: the true value the designs are built at, 3 positionswide4 views

What else it draws

The same object, drawn to answer the other questions the essays put to it.

Every design scored by its Ds-efficiency for K at 13 true values across a 16-fold range. The peaked curve is the subset design built at the guess K = 1: 100% there and 42.1% at the worst point of the range. The flat curve is the maximin-Ds design, never above 64.8% and never below 61.2%. Between them is the maximin design for the pair — a robust design, protecting something else, and worth 41.9% at worst here. The lowest curve is the D-optimal design at the guess, which is what an experimenter who wanted K and looked up a design for the model would actually run: 29.1% at the worst point, against 61.2% available.

Both stopping rules at five requirements, 1,500 experiments each, with a first stage of 5. The upper curve is the two-stage rule: 97.1%, 96.2%, 95.9%, 96.1%, 96.0% — at or above 95% at every point, which is a theorem rather than a tendency, because its interval is built from a spread estimated before the stopping point was chosen. It pays 2.06×, 2.01×, 2.00×, 1.99×, 1.99× the observations that knowing σ would need. The lower curve is the rule that re-estimates after every observation: 94.5%, 89.3%, 91.0%, 91.5%, 94.3%, on 0.98×, 0.87×, 0.88×, 0.93×, 0.96×. The second rule is the one anybody would run and the first is the one whose claim is true.

Coverage against spend at a half-width of 0.4 and a first stage of 5, all exact. A floor on the second stage at 5, 8, 10, 12, 15, 18, 21, 24 observations covers 88.4%, 91.4%, 92.4%, 93.1%, 93.9%, 94.6%, 95.3%, 96.0% at 0.86, 0.94, 0.97, 1.00, 1.02, 1.05, 1.09, 1.14 times the oracle's spend. The second stage reading with t, with its fraction dialled from 0.2 to 1, covers 88.1%, 89.0%, 90.6%, 92.1%, 93.4%, 94.4%, 95.2%, 95.9%, 96.4% at 0.84, 0.91, 1.02, 1.17, 1.34, 1.52, 1.70, 1.89, 2.08 times. A constant at the end covers 88.4%, 89.4%, 90.4%, 91.3%, 92.1%, 92.9%, 93.6%, 94.2%, 95.0%, 95.7% at 0.86, 0.90, 0.94, 0.98, 1.02, 1.06, 1.10, 1.14, 1.20, 1.26 times.

Where it is used

8 essays draw this figure, each at the numbers its own argument is about, so the same picture answers 8 different questions.

All 80 figures

FieldsThreadsSeriesConceptsAll essaysSearch