Normal Distribution
nothing is called 95% until it has been counted
Series

Local design — the series

10 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. Where to look depends on the answer. The information a single run at time t carries about the rate of an exponential decay, (∂η/∂θ)² = t²·exp(−2θt), at three values of θ. Each curve has one maximum and it is at t = 1/θ exactly — marked, and found by a search over 8,001 settings that was never told the formula. Nothing in a linear model behaves this way: there the information matrix is X′X and the parameters are not in it, so a design can be chosen once and used whatever the answer turns out to be. Here the design is optimal at a guess, and the three curves are three different experiments for one model.

    The design that needs the answer

    Every design this site has computed is optimal whatever the experiment turns out to say, because X′X does not contain the parameters. For a non-linear model it does, so the best place to take a measurement is a function of the number the measurement exists to find — and guessing it three times too low costs two and a half times more than guessing it three times too high.

    part 1 · criteria
  2. A design that is right once, against one that is never wrong by much. Two designs for the same two-parameter model, scored at every true value of K across a range of 16-fold. The local design is the two-point optimum for a guess of K = 1: it reaches 100% there and 66.7% at the worst point of the range. The hedged design maximises the average of log|M| over a prior spanning a factor of 4 either side, uses 3 settings rather than two, and is never below 75.4%. What it costs is 10.4 points at the one value the local design was built for — which is the whole trade, and it is only available to somebody willing to say how wrong the guess might be.

    The design that hedges

    A locally optimal design is right at one value of the unknown and 23.9% efficient at the edge of a sixteenfold range. Averaging the criterion over a prior instead buys the worst case back to 56.3% — and buys it by adding support points, at spreads the arithmetic decides rather than the experimenter — a third setting at a factor of 3.36 and a fourth at 8.86.

    part 2 · criteria
  3. A design that is right once, and one that is never wrong by much. Three designs for the Michaelis–Menten model, scored at every true value of K across a 16-fold range. The peaked curve is the two-point local design built at K = 1: 100% there and 66.7% at the worst point of the range. The curve just under it is the design that averages the criterion over a uniform prior on the same range, which is barely different — 67.9% at worst — because averaging is dominated by the middle of the range where the local design is already good. The flat line is the maximin design: 3 settings, never above 80.8% and never below 78.8%. Its worst case is 12.0 points better, and what it gives up is the 21.2 points at the one value the local design was built for.

    The design for the worst case

    A design for a non-linear model is optimal at a guess about the answer. Averaging over a prior repairs that on average; protecting the worst value in a range is a different problem, with a different answer, and it needs a third setting to reach it.

    part 3 · robust
  4. The optimum is a tie, and the tie is at both ends. The maximin design's efficiency across the range, and underneath it the prior that makes the averaged criterion as bad as possible. The efficiency curve is flat to within 2.1 points, and the minimum 78.74% is attained at K = 0.25 and 0.79 and 0.89 and 1.00 and 1.12 and 4.00 rather than at a single value: if it were attained once, the design could be moved towards that value and the worst case improved, so a tie is what having finished looks like. The bars are the least favourable prior's weights, computed by a completely different route — an averaging problem solved under the weighting that hurts most — and it puts its weight exactly where the ties are, reaching 78.63% against the direct search's 78.74%.

    Where the minimum is attained

    A design that protects a range is finished when its worst case is a tie. That is a checkable property rather than a description, it is why the search cannot climb a derivative, and it is the same corner the criteria field found at the end of the Φₚ family.

    part 4 · robust
  5. One experiment finding out where to look. A single run of the fully sequential design: 40 runs, the first 8 placed at the guess K = 1, then the model refitted and the design revised after every 2. The marks are the settings the runs were made at. The horizontal lines are where a design built at the truth K = 3 would have put them — 1.875 and 10.00 — and the rule walks onto them without being told: its estimate of K after the first eight runs was 2.694, and by the end 2.765 against a truth of 3. The whole experiment is 96.5% as efficient as the design that knew the answer, where running all 40 at the guess would have been 81.1%.

    The design that stops guessing

    Every repair so far protects a guess. The alternative is to run part of the experiment, estimate the parameter from it, and design the rest at the estimate — which recovers most of what a threefold wrong guess costs, and has a best moment to stop guessing that is earlier than anyone expects.

    part 5 · robust
  6. What the interval covers, after a design that read the data. 800 experiments of 12 runs, all at the same truth. The first pair is a design fixed in advance; the second is one whose settings were chosen from the first stage's own outcomes. If choosing the design from the data broke the inference, the second pair would sit below the first, and it does not — 91.3% against 92.6%. What does move the coverage is the shape of the interval rather than the design: the Wald interval assumes the estimate is normal around its own standard error and falls short under both designs, and the profile interval, computed from the same residual sums of squares with no derivative in it, covers 94.3% and 94.9%.

    What a design chosen from the data costs

    Two earlier measurements found what happens when a rule reads the data, and the error rate broke both times. A design that reads the data to decide where to put its runs breaks nothing — and the control that proves it also finds what the real shortfall is.

    part 6 · robust
  7. Four designs, scored on the one parameter that was wanted. Every design scored by its Ds-efficiency for K at 13 true values across a 16-fold range. The peaked curve is the subset design built at the guess K = 1: 100% there and 42.1% at the worst point of the range. The flat curve is the maximin-Ds design, never above 64.8% and never below 61.2%. Between them is the maximin design for the pair — a robust design, protecting something else, and worth 41.9% at worst here. The lowest curve is the D-optimal design at the guess, which is what an experimenter who wanted K and looked up a design for the model would actually run: 29.1% at the worst point, against 61.2% available.

    Protecting one parameter over a range

    A design for a non-linear model is optimal at a guess. A design for one of its parameters over a range of guesses is a worst case of a ratio of two determinants, and it is not a special case of either problem it is made of.

    part 7 · guarantee
  8. Six intervals after a design chosen from the data, 12 runs. Over 800 experiments of 12 runs, a first-order design at a guess of K = 1 where the truth is 3, and 199 parametric resamples per experiment. After the sequential design chosen from the data, the coverage of each interval for K: Wald 91.3%; percentile, design kept 90.6%; percentile, design re-run 91.6%; studentised, design kept 93.0%; studentised, design re-run 92.0%; profile 94.9%.

    A bootstrap that repeats the design

    After a design that chose its settings from the data, a parametric bootstrap can keep the settings the experiment ran or repeat the whole procedure from the start. On twelve runs repeating it changes coverage by a point at most, because the adaptivity was nearly free. What matters is the interval's shape: the percentile bootstrap covers 90.6%, no better than the Wald interval's 91.3%; the studentised one 93.0%; the profile interval 94.9%. And after a design stuck at a twelvefold wrong guess, the studentised bootstrap — the textbook repair — covers 77.8%, with intervals averaging 54.5 wide for a parameter of 3.

    part 7 · robust
  9. One curve, three designs, and the disagreement is the weights. The compartmental response exp(−θ₁t) − exp(−θ₂t) at the guess θ = (0.2, 1.2), with three designs underneath it. Each mark is a setting and its height is the share of the experiment spent there. The D-optimal design splits the runs equally between two settings — that is the determinant's answer and it is equal for every model of this kind. The design for the first parameter alone puts 1.9% of the experiment at its early setting and the rest at its late one, because the early runs are there to identify the nuisance and nothing more. The design that protects a 4-fold rectangle needs 3 settings and spends 9.1% at the earliest of them.

    The guess with two numbers in it

    Every optimal design for a non-linear model is optimal at a guess. Where the model has one parameter that moves the settings, that guess is a number and everything about it comes out in closed form; where it has two, three constants become functions and one of them becomes zero.

    part 8 · blind
  10. Where the design puts its runs as the noise stops being constant. The locally Ds-optimal design for the slow rate of the compartmental model at the guess (0.2, 1.2), on a horizon of 10, under each variance function: the settings and the share of runs at each. constant noise: 0.66 with 1.9% and 5.74 with 98.1%; noise follows the response, c = 1: 0.55 with 1.0% and 6.55 with 99.0%; noise follows the response, c = 3: 0.36 with 0.1% and 8.89 with 99.9%; noise follows the response, c = 10: 0.17 with 0.0% and 10.00 with 100.0%; noise follows the clock, c = 1: 0.49 with 4.2% and 4.39 with 95.8%; noise follows the clock, c = 3: 0.41 with 4.4% and 4.16 with 95.6%; noise follows the clock, c = 10: 0.34 with 4.4% and 4.03 with 95.6%.

    A variance that follows the response

    Every design for the two-rate compartmental model assumed the same noise at every sampling time. Let the noise grow with the response, as a combined additive and proportional error does, and the design for the slow rate runs to the tail of the curve, where the response and so the noise are smallest: at three additive standard deviations of proportional error at the peak, the constant-noise design is worth two thirds of the right one. The earlier essay expected a weight function to break the model's scale invariance. A variance that follows the response leaves it exact to six decimals; only one that follows the clock breaks it. And tailoring the design to the noise makes it more fragile to the rates: its worst case over the rectangle falls from 9.2% to 3.3%.

    part 9 · blind

All series