Normal Distribution
nothing is called 95% until it has been counted
Concept

Weighted least squares — where it appears

Fitting by minimising a weighted sum of squared residuals. The weights that make the estimate efficient and its variance decomposition exact are the inverse variances. Getting them wrong leaves the estimate unbiased and the interval no longer exact, which is a failure that shows in width rather than in coverage.

Named by 11 essays across 6 fields — each of them below, with the objects they name alongside it.

One experiment, with the blocks getting smaller as the target comes into range. A single run at a requirement of 0.25, with the block sizes 5, 5, 11, 25, 11, 8, 3, 2 and a total of 70 observations in 8 blocks. The rule stops when the observations in hand reach z²σ̂²/d², with σ̂² pooled from the within-block contrasts — an estimate that moves as the run goes on, so the target moves too. Early blocks are large because the target is far away and cannot be overshot; late ones are small because a block is the granularity of the answer. The interval afterwards is built from the 8 block means and from nothing the rule looked at, and it has 7 degrees of freedom against the rule's 62.

A block size that changes

The blinded rule's exactness never needed the blocks to be the same size. Letting the size be chosen from the contrasts as the run goes on leaves the coverage exactly where it was — and runs straight into an identity that says what a schedule can and cannot buy.

pace · Stopping The stopping rule costs more than the weighting does. Coverage over 2000 runs of a trial whose variance ratio drifts by a factor of twenty, at three ways of deciding when to stop. Twelve blocks fixed in advance is the top line and reproduces what a trial of fixed length delivers. Stopping when the reported interval is short enough is the bottom line, and it costs between 3.0% and 5.5% of coverage — including for the rule that is told every block's true ratio, which is what says the shortfall belongs to the stopping and not to the weights. Stopping on a width predicted from the within-arm sums of squares is the middle line, and it is back at the fixed-length values. The standard error on each point is 0.49%.

A width the trial has to stop for

The weighting that covers at 94.9% on twelve blocks covers at 91.5% when the trial stops as soon as its interval is short enough — and so does the rule that is told every block's true variance ratio. The shortfall is the stopping, not the weights.

stop · Stopping Exact in the corner, where nothing was. Coverage of a nominal 95% interval on five designs, at a required half-width of 0.3. The first four are the two-arm field's own and the fifth is its corner — two variances, block sizes that swing by eight, and an allocation that alternates between five to one and one to five — where neither of that field's two conditions holds. The effective-size weights over-cover there at 98.40%; the weights h_b(λ) = (1/m_A + λ/m_B)⁻¹ cover at 94.84%, and at 94.84% when λ is estimated from the within-arm contrasts rather than known. Nothing here is supposed to move.

Weights that need only a ratio

A fixed-width interval about a difference is exact under either of two conditions and under neither in the corner. It is exact there too, and the only thing it needs is how much larger one arm's variance is than the other's.

corner · Nuisance Two promises, and no rule here keeps both. A fixed-width procedure promises two things: that the interval covers at its nominal rate, and that it is no wider than the width asked for. Over 1500 runs of the modelled weighting, a rule that stops when the interval it will report is short enough keeps the width — only 2.0% of runs come out wider than 0.34 — and covers at 91.13% against a nominal 95%. A rule that stops on a width predicted from the within-arm sums of squares covers at 94.80% and comes out wider than promised on 42.3% of runs. The two promises are in conflict because keeping the second one exactly requires conditioning on the very quantity that has to be independent of the stopping time for the first.

Stopping on the arms

The width a trial will report is predictable from quantities the interval is not about. A rule that stops on the prediction covers at 94.5% where one that stops on the interval covers at 91.5, and it costs two blocks and half of the width promise.

stop · Width Three sets of weights, five designs, and no estimator that is exact everywhere. Coverage of the same interval under three weightings. h_b is the inverse variance when the arms share a variance or the allocation is constant; equal weights are right when every block has the same two counts; the estimated precision weights are right in the limit and exact nowhere, because the decomposition needs the weights to be the constants they are only estimating. In the corner — two variances, changing sizes, changing allocation — the two exact estimators are the ones that miss, at 98.45% and 95.65%, and the one with no theorem behind it is at 95.05%. That is the whole statement: there is an exact estimator under either condition, and none under both.

Which weights are the inverse variances

There is an exact estimator when the two arms share a variance and another when every block has the same two counts, and between them they cover every trial anybody designs on purpose. In the corner where neither holds, both cover 98.45% instead of 95%, and the only estimator at its level is the one with no theorem behind it.

contrast · Nuisance The weights may not read the block they weight. A weighted least squares decomposition needs weights that are constants, or at least independent of the differences they multiply. One λ̂ pooled across the trial is estimated on hundreds of degrees of freedom and is effectively a constant; a λ̂ estimated inside each block is estimated on that block's own two or three, and is correlated with the difference it weights. Coverage falls from 94.68% to 82.76% — and the interval gets wider while doing it, 0.5163 against 0.3024, which is the signature of weights that are noise.

The condition that cannot be dropped

The weights may not read the block they weight. Estimate the variance ratio inside each block rather than across the trial and the coverage falls to 83% — on an interval that is at the same time seventy per cent wider.

corner · Allocation A wrong weight costs width; a random weight costs level. Five weightings on a trial whose variance ratio drifts by a factor of 20.1 between the first block and the last, over 4000 runs. The rule that knows every λ_b covers at 95.1% and sets the width. One ratio for the whole trial is wrong for every block and costs nothing in level — 94.8% — while being 20% wider; equal weights are calibrated by an identity and 22% wider. The ratio estimated inside each block is the only rule aimed at the quantity that actually varies, and it is the only one that misses the level, at 92.0%: a weight computed from a handful of degrees of freedom is mostly noise, and noise in a weight is not a wrong weight. Modelling the drift across blocks recovers the oracle's width at 94.8%.

A ratio that changes between blocks

A wrong weight costs width and a random weight costs level. The rule aimed at the quantity that actually varies is the only one that misses its own coverage, and the rule that models it across blocks recovers the whole of what knowing it is worth.

blocks · Nuisance One pooled ratio, two periods. The calibration factor of weights built from one variance ratio pooled across the trial — the factor by which the interval's own scale estimate overstates the estimate's variance — when arm B's ratio is F times higher in one kind of period than the other, closed form on the corner's block compositions. With the small A-heavy blocks the noisier, the factor rises: 1.00 at F = 1, 1.50 at F = 1.5, 2.07 at F = 2, 3.12 at F = 3, 3.91 at F = 4, 4.86 at F = 6, 5.44 at F = 8. With the large B-heavy blocks the noisier, it falls: 1.00 at F = 1, 0.74 at F = 1.5, 0.65 at F = 2, 0.58 at F = 3, 0.55 at F = 4, 0.53 at F = 6, 0.53 at F = 8. A ratio estimated per period gives weights that are the inverse variances again, and its factor is exactly 1 at every F.

A ratio that changes with the period

The corner's exact interval needs one variance ratio, and the corner's own alternating allocation makes two kinds of period that need not share one. When they differ the pooled ratio is precise and wrong: at a fourfold difference the interval covers 100.0% one way round and 85.1% the other. A ratio estimated per period is exact again, blind to the effect, and costs under a tenth of the width — and a ratio that merely wanders at random costs width and no coverage at all.

corner · Nuisance Where the bias lands. The drift in the log variance ratio, fitted across 12 blocks over 4000 trials. E[log λ̂_b] is log λ_b plus ψ(k_B/2) − log(k_B/2) − ψ(k_A/2) + log(k_A/2), which depends on nothing but the degrees of freedom — so the tempting sentence is that it goes into the intercept and leaves the slope alone. It does not, because the blocks alternate between allocations and the alternation is correlated with the covariate being fitted: the lopsided blocks carry 0.5383 of bias and the even ones carry none. Uncorrected the slope reads 1.5597 against a truth of 1.5, which is 8.0 standard errors. Subtracting the two digammas block by block leaves 1.4976.

The bias that lands in the slope

The bias in a log variance estimate depends on nothing but its degrees of freedom, so it goes into the intercept — unless the degrees of freedom alternate with the design, which is exactly what a block-randomised trial makes them do.

blocks · Width A lean, its reflection, and none. The mean error of each weighting's estimated difference at a fixed eighteen blocks, 2,000 trials a shape, with two standard errors. normal: the modelled ratio, every block 0.0031, the modelled ratio, each block left out 0.0028, the modelled ratio, cross-fitted halves 0.0023, the true ratios 0.0037; skewness 0.95: the modelled ratio, every block 0.0092, the modelled ratio, each block left out -0.0134, the modelled ratio, cross-fitted halves -0.0014, the true ratios 0.0002; skewness 2.26: the modelled ratio, every block 0.0166, the modelled ratio, each block left out -0.0298, the modelled ratio, cross-fitted halves -0.0042, the true ratios -0.0029; skewness 4.75: the modelled ratio, every block 0.0255, the modelled ratio, each block left out -0.0455, the modelled ratio, cross-fitted halves -0.0055, the true ratios -0.0051.

A weighting that does not read its own spreads

On skewed outcomes a block's spreads move with its means, so a weighting fitted to every block's spreads leans: +0.0255 at a skewness of 4.75. The obvious repair — fit each block's weight with that block left out — does not remove the lean. It reverses it and nearly doubles it, to −0.0455, because a weighted mean divides by the sum of the weights and every other block's weight still reads the one left out. Splitting the trial into two halves and weighting each by the other's fit does remove it: −0.0055, indistinguishable from the weighting told the true ratios, for a spread about a tenth wider.

stop · Width Where the design puts its runs as the noise stops being constant. The locally Ds-optimal design for the slow rate of the compartmental model at the guess (0.2, 1.2), on a horizon of 10, under each variance function: the settings and the share of runs at each. constant noise: 0.66 with 1.9% and 5.74 with 98.1%; noise follows the response, c = 1: 0.55 with 1.0% and 6.55 with 99.0%; noise follows the response, c = 3: 0.36 with 0.1% and 8.89 with 99.9%; noise follows the response, c = 10: 0.17 with 0.0% and 10.00 with 100.0%; noise follows the clock, c = 1: 0.49 with 4.2% and 4.39 with 95.8%; noise follows the clock, c = 3: 0.41 with 4.4% and 4.16 with 95.6%; noise follows the clock, c = 10: 0.34 with 4.4% and 4.03 with 95.6%.

A variance that follows the response

Every design for the two-rate compartmental model assumed the same noise at every sampling time. Let the noise grow with the response, as a combined additive and proportional error does, and the design for the slow rate runs to the tail of the curve, where the response and so the noise are smallest: at three additive standard deviations of proportional error at the peak, the constant-noise design is worth two thirds of the right one. The earlier essay expected a weight function to break the model's scale invariance. A variance that follows the response leaves it exact to six decimals; only one that follows the clock breaks it. And tailoring the design to the noise makes it more fragile to the rates: its worst case over the rectangle falls from 9.2% to 3.3%.

blind · Local design

Named alongside it

The objects these essays reach for when they reach for this one.

CoverageFixed-width intervalBlindingVariance ratioDegrees of freedomBlockingInterval widthInverse variance weightingBlockEfficiencyEstimated varianceAllocation ratio

All concepts