Normal Distribution
nothing is called 95% until it has been counted
Concept

Information matrix — where it appears

X'X divided by the number of runs: the information per run a design carries about a model's parameters, and the object every optimality criterion is a function of. Every optimality criterion in this fleet is a different scalar summary of the same matrix, which is why they can rank the same two designs in opposite orders.

Named by 25 essays across 10 fields — each of them below, with the objects they name alongside it.

The six best bases of 2 functions, and what each protects. Every cell is R²(g | span B) — the share of the imbalance in that shape a rule balancing that basis removes — computed from exact inner products between Hermite functions and indicators, with nothing simulated. The rows are ordered by their worst cell, which is the number an experimenter who does not know the shape is exposed to. The best row here guarantees 26.8% against every shape in the list, and the worst of the six guarantees 15.1%: the difference between them is entirely which subspace was picked, at the same cost per arrival.

A basis is a subspace

A balancing rule cannot tell one basis from another with the same span, so choosing what to hand it is choosing a subspace — and then what it removes of any outcome shape is a projection, computable exactly, with no trial anywhere in it.

basis · Criterion Where a D-optimal design puts its runs. The D-optimal measure over 121 candidate settings on a square region. It keeps 9 of them and discards the rest, and the 9 it keeps are the settings a catalogue would have offered without any of this arithmetic. What the search adds is the weights: 0.1458, 0.0962, 0.0802, which nine equal runs cannot express.

A design is a number

A standard design is taken from a catalogue and then measured. Turn the arithmetic round and a design becomes the answer to an optimisation — and over 121 candidate settings the search keeps nine of them, which are exactly the nine a catalogue would have offered, at weights nine equal runs cannot express.

optimality · Criterion Two designs for one model, and the weights are not equal. Where the runs go, for the same two-parameter model at K = 1 and a ceiling of T = 10. The D-optimal design for both parameters is the familiar one: half the runs at 0.8333 and half at the ceiling. The design for the half-saturation constant alone moves the lower setting down to 0.6040 — where the response curve is still bending, which is where K is visible — and, unlike every design in the two fields before this one, it does not split the runs evenly: the weights are exactly 1/√2 and 1 − 1/√2, 0.7071 and 0.2929, at every K, V and T. The equal weights of D-optimality were a consequence of asking about both parameters at once, and nobody had to notice while that was the only question being asked.

An efficiency that is a ratio

A design chosen for a model is not a design chosen for the parameter somebody wanted. Asking for one of two parameters moves the runs, unbalances the weights, and costs the other question exactly 15.07% — at every setting, because it is algebra.

guarantee · Criterion A design that is right once, and one that is never wrong by much. Three designs for the Michaelis–Menten model, scored at every true value of K across a 16-fold range. The peaked curve is the two-point local design built at K = 1: 100% there and 66.7% at the worst point of the range. The curve just under it is the design that averages the criterion over a uniform prior on the same range, which is barely different — 67.9% at worst — because averaging is dominated by the middle of the range where the local design is already good. The flat line is the maximin design: 3 settings, never above 80.8% and never below 78.8%. Its worst case is 12.0 points better, and what it gives up is the 21.2 points at the one value the local design was built for.

The design for the worst case

A design for a non-linear model is optimal at a guess about the answer. Averaging over a prior repairs that on average; protecting the worst value in a range is a different problem, with a different answer, and it needs a third setting to reach it.

robust · Local design The end of the family is the member the algorithm cannot reach. Every member of the Φₚ family optimised on the same 11² candidates, and two eigenvalues of each answer. The upper curve is the smallest eigenvalue of the information matrix — the quantity E-optimality maximises — which rises from 0.0993 at the D end to 0.1994 at p = 64. The lower curve is the gap between that eigenvalue and the next one up, which falls from 0.0611 to 0.0011. A smallest eigenvalue is not differentiable where it is repeated, and the family is driving the gap to zero: the one criterion here whose meaning fits in a sentence is the one whose optimum sits on a corner of its own surface. The candidate grid is on a slider and it answers a narrower question than it looks. On this square region, refining an odd grid from seven to eleven moves nothing at all — the optimum's support is the corners, the edge midpoints and the centre, and every odd grid from five up contains all of them. An even grid has no centre point and cannot reach the answer at any member of the family. On a disc, where the boundary passes through no grid point, refinement does move it.

The family behind the letters

A, D and E are not three ideas. They are three points of one family with a single dial, and running the dial from one end to the other doubles the smallest eigenvalue of the information matrix while closing the gap above it fifty-three-fold — which is the family driving its own last member to the place where it stops being differentiable.

criteria · Criterion One curve, three designs, and the disagreement is the weights. The compartmental response exp(−θ₁t) − exp(−θ₂t) at the guess θ = (0.2, 1.2), with three designs underneath it. Each mark is a setting and its height is the share of the experiment spent there. The D-optimal design splits the runs equally between two settings — that is the determinant's answer and it is equal for every model of this kind. The design for the first parameter alone puts 1.9% of the experiment at its early setting and the rest at its late one, because the early runs are there to identify the nuisance and nothing more. The design that protects a 4-fold rectangle needs 3 settings and spends 9.1% at the earliest of them.

The guess with two numbers in it

Every optimal design for a non-linear model is optimal at a guess. Where the model has one parameter that moves the settings, that guess is a number and everything about it comes out in closed form; where it has two, three constants become functions and one of them becomes zero.

blind · Local design Four designs, scored on the one parameter that was wanted. Every design scored by its Ds-efficiency for K at 13 true values across a 16-fold range. The peaked curve is the subset design built at the guess K = 1: 100% there and 42.1% at the worst point of the range. The flat curve is the maximin-Ds design, never above 64.8% and never below 61.2%. Between them is the maximin design for the pair — a robust design, protecting something else, and worth 41.9% at worst here. The lowest curve is the D-optimal design at the guess, which is what an experimenter who wanted K and looked up a design for the model would actually run: 29.1% at the worst point, against 61.2% available.

Protecting one parameter over a range

A design for a non-linear model is optimal at a guess. A design for one of its parameters over a range of guesses is a worst case of a ratio of two determinants, and it is not a special case of either problem it is made of.

guarantee · Local design Two rates, not a factor. The standard deviation of the covariate imbalance under three rules, at five trial sizes, 260 trials each, on log axes. The upper line is a coin: its slope is -0.489, against a closed form of exactly −½. The middle line is minimisation on a median split; its slope is -0.519 — the same rate — because inside a category the assignment is still a coin, and what it buys is the constant, 0.654 of a coin's at n = 200. The lower line is the rule that reads x and maximises the information about the treatment effect: slope -0.987, nearly twice as steep. Its advantage is therefore not a number that can be quoted — it is 0.258 of a coin's at n = 50 and 0.065 at n = 800, and it keeps going.

The rule that reads the number

Stop categorising and let the rule read the covariate itself. What it should minimise is not an invented distance but the variance of the effect being estimated — and what comes back is not a better constant but a different rate.

continuous · Assignment The variance touches p and never crosses it. d(x) = f(x)′M⁻¹f(x) along the diagonal of a square region, for the D-optimal measure. The line at 6 is the number of parameters in the model. Kiefer and Wolfowitz's theorem says a design is D-optimal exactly when the largest d anywhere in the region is p — not approximately, equals — so the optimal curve is tangent to that line at its support points and below it everywhere else. Here the largest value anywhere on a 41×41 grid is 6.000000000.

The theorem that says when to stop

A search that maximises the volume of the information has no way of knowing it has finished, because nothing tells it what the maximum is. Kiefer and Wolfowitz's equality does — a design is D-optimal exactly when the worst prediction anywhere in the region equals the number of parameters, which is 6.000000000059 here, gated at machine precision.

optimality · Equivalence Where a Ds-optimal design puts its runs. The Ds-optimal measure over 121 candidate settings on a square region. It keeps 9 of them and discards the rest, and the 9 it keeps are the settings a catalogue would have offered without any of this arithmetic. What the search adds is the weights: 0.2500, 0.1250, 0.0625, which nine equal runs cannot express.

The two terms anybody wanted

D-optimality estimates all six parameters of a quadratic as precisely as possible. Nobody wants that. An experimenter looking for a maximum wants the two curvature terms, and the design that gives them is not the D-optimal one — it is a quarter of the runs at the centre, exactly, and the D-optimal design is 75.3% efficient for the question that was actually asked.

criteria · Criterion Four windows, one line, and one that is off it. The optimism measured for each window at a band of 30 lags, against what that window's weights sum to, on 2000 pairs of independent samples of 120 rows. The diagonal is where a window that spent exactly its summed weights would sit. Three of the points are one shape at three levels — the Bartlett window, its square and its cube, whose sums stand in the ratio 6 : 4 : 3 — and they lie on a line through the origin at 0.767 of the diagonal, with 0.033 between the highest and the lowest. Scaling the weights scales the charge by the factor the weights predict, which is what makes the weights the mechanism. The Parzen window has a comparable sum and a different shape, and it sits at 0.871: its weights stay near one over the first few lags, and the first few lags are where the information is. A weight sum treats every lag as equally informative and no sample does.

What a window leaves free

A Bartlett window's weights sum to exactly half its width, which is a candidate for what the band costs. Varying the weights without varying anything else says the weights are the mechanism; varying the shape at the same weight says they are not the arithmetic.

dimension · Criterion The optimum is a tie, and the tie is at both ends. The maximin design's efficiency across the range, and underneath it the prior that makes the averaged criterion as bad as possible. The efficiency curve is flat to within 2.1 points, and the minimum 78.74% is attained at K = 0.25 and 0.79 and 0.89 and 1.00 and 1.12 and 4.00 rather than at a single value: if it were attained once, the design could be moved towards that value and the worst case improved, so a tie is what having finished looks like. The bars are the least favourable prior's weights, computed by a completely different route — an averaging problem solved under the weighting that hurts most — and it puts its weight exactly where the ties are, reaching 78.63% against the direct search's 78.74%.

Where the minimum is attained

A design that protects a range is finished when its worst case is a tie. That is a checkable property rather than a description, it is why the search cannot climb a derivative, and it is the same corner the criteria field found at the end of the Φₚ family.

robust · Local design Four criteria on 5 designs, on a square region. Each design scored as an efficiency — its value over the best attainable — so four criteria in as many different units sit on one scale where 1 is the optimum. Rows are ordered by D. D picks 13-run exchange; A picks face-centred composite; G picks 13-run exchange; I picks face-centred composite. Every design has been scaled to just fit the region first, because a design run at settings the region does not contain is not a competitor on it. The disagreement is the point: the letter is a choice, and it is almost never reported as one.

Four letters and two camps

D, A, G and I are four ways of turning one matrix into one number, and they do not agree. The design that wins on D is the worst thing here on I. And the same two designs swap places entirely when the region changes from a square to a disc — on all four criteria at once.

optimality · Criterion Where to look depends on the answer. The information a single run at time t carries about the rate of an exponential decay, (∂η/∂θ)² = t²·exp(−2θt), at three values of θ. Each curve has one maximum and it is at t = 1/θ exactly — marked, and found by a search over 8,001 settings that was never told the formula. Nothing in a linear model behaves this way: there the information matrix is X′X and the parameters are not in it, so a design can be chosen once and used whatever the answer turns out to be. Here the design is optimal at a guess, and the three curves are three different experiments for one model.

The design that needs the answer

Every design this site has computed is optimal whatever the experiment turns out to say, because X′X does not contain the parameters. For a non-linear model it does, so the best place to take a measurement is a function of the number the measurement exists to find — and guessing it three times too low costs two and a half times more than guessing it three times too high.

criteria · Local design What a rule reads, against what the outcome uses. The variance of the treatment estimate relative to a coin's, for four things a rule might balance against three shapes the outcome might have, over 350 trials of 200 units. The diagonal is the easy part — a rule that reads the function the outcome uses removes about half the variance. What the table is for is the off-diagonal: reading the covariate alone is worth nothing against a quadratic (0.755), and reading all three is worth nearly as much against every shape as the matching rule is against its own (0.532, 0.493, 0.514).

Three functions of one number

A rule that balances the covariate is exposed to every shape the outcome might have. A rule that balances three functions of it costs two points of variance against the shape the first was built for and takes the worst case from a coin's to about half of it.

shape · Criterion What balancing several numbers at once costs each of them. The criterion generalises without a word changing — the covariate imbalance becomes a vector and the correction a quadratic form — so the question is what it is worth rather than whether it can be done. At n = 200 with 200 trials per point, a rule balancing one covariate leaves 12.7% of a coin's imbalance in it; balancing eight leaves 23.2% in each. The assignment has a fixed amount of freedom and every covariate added takes a share of it. The rule degrades rather than failing: at eight covariates it is still four times better balanced than a coin, and the eight are being held simultaneously rather than in turn.

Balancing more than one number

The criterion generalises to several covariates without a word changing, which makes the question what it is worth rather than whether it can be done. Each one added takes a share of the assignment's freedom, and the imbalance left in every one of them rises.

continuous · Blocking Adding a run can make the design worse. The D-efficiency of the best N-run design at each size, against the optimal measure. It is not a rising curve. 13 runs reaches 99.77% and 14 falls to 99.44%, because the optimal weights are real numbers and N runs is an integer approximation to them, so how good a design can be depends on how well N divides. A Wald interval behaves the same way: a larger sample sometimes makes its coverage worse, for exactly this reason.

The design that has to be integers

The optimal design is a set of real weights and an experiment is a set of runs, so the theory's answer is never available. Thirteen runs reach 99.77% of it and fourteen reach 99.44% — adding a run makes the design worse per run, and the search that finds it does not always find the same one.

optimality · Criterion The interval every package reports first does not cover. Counted coverage of two 95% intervals for the 100-block return level of a normal parent, against the length of the record they were fitted from, over 300 records at each length. The level they are about is known in closed form, so this is coverage of a number rather than agreement between two estimates. The delta-method interval covers 80.3% at 25 blocks and reaches only 89.0% at 200; the profile-likelihood interval sits between 94.0% and 94.7% throughout. The gap is not a small-sample effect that lengthening the record removes — it narrows by 8.7 points for an eightfold longer record.

Two intervals for one return level

Two 95% intervals read off the same fits of the same records, against a level known in closed form. The symmetric one covers 80.3% at twenty-five blocks and reaches only 89.0% at two hundred — and 99.24% of its misses are the interval sitting entirely below the truth, which is not the endpoint anybody expects to fail.

extreme · Extremes What visiting fewer settings costs, 6 parameters. Carathéodory's bound puts the support of an optimal measure between 6 and 21. 6 settings: D-efficiency 88.90%, G-efficiency 57.18%, 0 degrees of freedom for lack of fit; 7 settings: D-efficiency 94.54%, G-efficiency 61.22%, 1 degrees of freedom for lack of fit; 8 settings: D-efficiency 95.99%, G-efficiency 64.60%, 2 degrees of freedom for lack of fit; 9 settings: D-efficiency 97.40%, G-efficiency 82.76%, 3 degrees of freedom for lack of fit. The saturated design has none, and buying the first one costs about five points of efficiency to get back.

How many places a design goes

Carathéodory's bound puts an optimal design's support between six and twenty-one settings, and every design in this field that can fit the model visits exactly nine. The count is not a choice anybody makes, it decides how many degrees of freedom are left to check the model with, and the first spare setting costs six points of efficiency to get back.

optimality · Equivalence The worst case against the number of functions, at four trial sizes. The largest share of a coin's imbalance the rule leaves in any of the four shapes, against how many functions it reads, at 50, 100, 200, 400 units. At 50 units: 103.5%, 38.6%, 37.2%, 37.0%, 39.8%, 41.7%. At 100 units: 105.9%, 37.8%, 38.9%, 30.3%, 27.0%, 24.6%. At 200 units: 95.9%, 44.2%, 42.8%, 32.9%, 28.4%, 11.7%. At 400 units: 108.5%, 41.8%, 41.3%, 32.1%, 26.1%, 5.8%. At fifty units every basis from two functions up leaves the worst shape between 37.0% and 41.7%; at four hundred, six functions leave it 5.8%.

How many functions a trial can hold

A rule that reads more functions of one covariate protects more shapes, and every function it reads costs the shapes it already held: the imbalance it leaves in a linear outcome falls like one over the trial size, with a constant that runs from about 3 for one function to about 16 for six. At fifty units that cost eats the protection, and the worst case over four shapes sits between 37% and 42% of a coin's from two functions to six. At four hundred it falls from 42% to 6%. By six functions the worst shape, at every size, is one the rule is reading.

shape · Criterion Where the design puts its runs as the noise stops being constant. The locally Ds-optimal design for the slow rate of the compartmental model at the guess (0.2, 1.2), on a horizon of 10, under each variance function: the settings and the share of runs at each. constant noise: 0.66 with 1.9% and 5.74 with 98.1%; noise follows the response, c = 1: 0.55 with 1.0% and 6.55 with 99.0%; noise follows the response, c = 3: 0.36 with 0.1% and 8.89 with 99.9%; noise follows the response, c = 10: 0.17 with 0.0% and 10.00 with 100.0%; noise follows the clock, c = 1: 0.49 with 4.2% and 4.39 with 95.8%; noise follows the clock, c = 3: 0.41 with 4.4% and 4.16 with 95.6%; noise follows the clock, c = 10: 0.34 with 4.4% and 4.03 with 95.6%.

A variance that follows the response

Every design for the two-rate compartmental model assumed the same noise at every sampling time. Let the noise grow with the response, as a combined additive and proportional error does, and the design for the slow rate runs to the tail of the curve, where the response and so the noise are smallest: at three additive standard deviations of proportional error at the peak, the constant-noise design is worth two thirds of the right one. The earlier essay expected a weight function to break the model's scale invariance. A variance that follows the response leaves it exact to six decimals; only one that follows the clock breaks it. And tailoring the design to the noise makes it more fragile to the rates: its worst case over the rectangle falls from 9.2% to 3.3%.

blind · Local design What the main effects cost the rest of the model. The compound optimum of α·log Ds(main effects) + (1 − α)·log D on an 11-by-11 grid of the square, at 14 weights from α = 0 to 0.999, placed by its D-efficiency against the D-optimal design and its main-effect efficiency against the factorial. The D-optimal design (α = 0) gives the main effects 74.3% of what the factorial gives. At α = 0.5 the design keeps 94.6% of D and gives 85.6%; at 0.9, 64.9% and 96.8%; at 0.99, 31.3% and 99.7%. The factorial itself (open, at the left edge) gives the main effects everything and the model nothing: its D-efficiency is zero. The design optimal for the two quadratic terms (red) gives the main effects 50.0% at a D-efficiency of 83.6%.

A subset the model cannot use

Nominate the two main effects of a quadratic response surface as the parameters of interest and the optimal design is the 2² factorial — four corners, a quarter each — which leaves the full model's information matrix at rank four of six and the curvature with nothing at all. That is why the field's algorithm would not converge on it. Approached through a compound criterion instead, the trade is explicit: 96.8% of the main effects' best information costs the model 35% of its D-efficiency, and the last three points cost nine tenths of what is left about the curvature.

criteria · Criterion The certificate when 4 runs are already spent. a 2² factorial has already been run and 2 further runs are to be placed. The stationarity condition is no longer max d = p; it is max d = (p − λ·tr(M⁻¹M_fixed))/(1 − λ) with λ = 0.6667 the share of runs already spent, which is 7.0985 here. The search reaches 7.098508790 against it, and the largest value anywhere on a 41×41 grid is 7.098508790.

Augmenting a design that has already run

The equivalence theorem still certifies when some runs are already spent, and one number in it changes: the bound is no longer p but (p − λ·tr(M⁻¹M_fixed))/(1 − λ). It equals p again exactly when the runs already made can still be absorbed into the design that would have been chosen — so the certificate says whether the experiment is still recoverable.

optimality · Equivalence Where each criterion's optimum puts the information. the D-optimal design's smallest eigenvalue is 0.09927, attained once; the A-optimal design's smallest eigenvalue is 0.16516, attained once; the I-optimal design's smallest eigenvalue is 0.17541, attained once; the E-optimal design's smallest eigenvalue is 0.19999, attained 2 times. A criterion that reads the smallest eigenvalue has no derivative where that eigenvalue is repeated, and the E-optimal design is exactly there.

The criterion with no derivative

E-optimality maximises the smallest eigenvalue of the information matrix, and at its own optimum that eigenvalue is attained twice — which is exactly where the function has a corner. The multiplicative search this field's other three criteria use assumes a derivative that is not there, and stops at 37.2% of the optimum.

optimality · Equivalence The E-optimality certificate along the axis x₂ = 0. The certificate — a fifth of the squared distance of the point's squared radius from one, plus a fifth of the squared difference of its two squared coordinates — along the axis x₂ = 0, with its two pieces drawn alone at full weight: the radial piece, a third of the squared distance from the unit circle in squared radius, which reaches 0.333 here, and the cross piece, half the squared difference of the squared coordinates, which reaches 0.500. The certificate's largest value on the cut is 0.2000, reached at x = -1.0, 0.0, 1.0; the smallest eigenvalue of the nine-point design is 0.2.

A certificate for the worst direction

E-optimality was the one design criterion here whose answer was claimed rather than proved. It has a certificate after all, and on the square it fits on one line — (1⁄5)(1 − x₁² − x₂²)² + (1⁄5)(x₁² − x₂²)², which never exceeds 1⁄5 and touches it at exactly the nine points of the 3² grid. That proves the design with 2⁄5 of its weight at the centre, 1⁄10 at each edge midpoint and 1⁄20 at each corner is E-optimal on the whole square, and it is built from two directions mixed three to two, either of which alone fails.

optimality · Equivalence

Named alongside it

The objects these essays reach for when they reach for this one.

Optimal designD-optimalityDesign measureExperimental designEquivalence theoremThe non-linear modelDs-optimalityLocally optimal designMaximin designPrediction varianceClosed formE-optimality

All concepts