Normal Distribution
nothing is called 95% until it has been counted
Concept

Calibration — where it appears

Whether an interval's own estimate of its own width is right on average. For fixed weights it is a closed factor in the design and the block variances, equal to one in exactly two cases — weights proportional to the inverse variances, and weights that are all equal.

Named by 9 essays across 3 fields — each of them below, with the objects they name alongside it.

Exact in the corner, where nothing was. Coverage of a nominal 95% interval on five designs, at a required half-width of 0.3. The first four are the two-arm field's own and the fifth is its corner — two variances, block sizes that swing by eight, and an allocation that alternates between five to one and one to five — where neither of that field's two conditions holds. The effective-size weights over-cover there at 98.40%; the weights h_b(λ) = (1/m_A + λ/m_B)⁻¹ cover at 94.84%, and at 94.84% when λ is estimated from the within-arm contrasts rather than known. Nothing here is supposed to move.

Weights that need only a ratio

A fixed-width interval about a difference is exact under either of two conditions and under neither in the corner. It is exact there too, and the only thing it needs is how much larger one arm's variance is than the other's.

corner · Nuisance How wrong the ratio is allowed to be. λ enters only through the weights, so misstating it leaves the estimate unbiased and moves two things — the interval's calibration and its efficiency — both of which are closed forms of the design. Coverage stays at its level over a factor of two in either direction (94.93% at half the truth, 94.27% at twice it) and starts to go at a factor of five. An estimate on hundreds of within-arm degrees of freedom is never wrong by anything like that, which is what makes the feasible rule usable rather than merely definable.

Blinded, and still exact

The one number the exact interval needs is a ratio of within-arm spreads, which is a contrast and contains no mean — so a rule forbidden to look at the effect may compute it, on more degrees of freedom than the interval itself has.

corner · Width Three charges, and only one of them is a test. What each of three thresholds does to the same decision, under AR(1) at 0.8, against the size of a genuine break in the mean at row 60. A chi-square on the 5 coefficients a split adds — 11.07 — declares a break on 73.6% of samples that have none: it is not a test at all. The break search's own 95% point, 69.6, carried into a rule that also chooses its window, fires on 0.0% of null samples and on 0.0% of samples with the largest break measured — the natural way of combining two published corrections does not lose a little power, it switches the test off. The calibrated charge, 27.2, holds 5.6% at no break and reaches 29.2% at the largest.

The charge that is not a sum

Charging two searches what each costs on its own is conservative, and conservative here means the test never fires. At the largest break measured it declares nothing, on every draw, while a calibrated threshold reaches 29%.

twice · Break point The weights may not read the block they weight. A weighted least squares decomposition needs weights that are constants, or at least independent of the differences they multiply. One λ̂ pooled across the trial is estimated on hundreds of degrees of freedom and is effectively a constant; a λ̂ estimated inside each block is estimated on that block's own two or three, and is correlated with the difference it weights. Coverage falls from 94.68% to 82.76% — and the interval gets wider while doing it, 0.5163 against 0.3024, which is the signature of weights that are noise.

The condition that cannot be dropped

The weights may not read the block they weight. Estimate the variance ratio inside each block rather than across the trial and the coverage falls to 83% — on an interval that is at the same time seventy per cent wider.

corner · Allocation One pooled ratio, two periods. The calibration factor of weights built from one variance ratio pooled across the trial — the factor by which the interval's own scale estimate overstates the estimate's variance — when arm B's ratio is F times higher in one kind of period than the other, closed form on the corner's block compositions. With the small A-heavy blocks the noisier, the factor rises: 1.00 at F = 1, 1.50 at F = 1.5, 2.07 at F = 2, 3.12 at F = 3, 3.91 at F = 4, 4.86 at F = 6, 5.44 at F = 8. With the large B-heavy blocks the noisier, it falls: 1.00 at F = 1, 0.74 at F = 1.5, 0.65 at F = 2, 0.58 at F = 3, 0.55 at F = 4, 0.53 at F = 6, 0.53 at F = 8. A ratio estimated per period gives weights that are the inverse variances again, and its factor is exactly 1 at every F.

A ratio that changes with the period

The corner's exact interval needs one variance ratio, and the corner's own alternating allocation makes two kinds of period that need not share one. When they differ the pooled ratio is precise and wrong: at a fourfold difference the interval covers 100.0% one way round and 85.1% the other. A ratio estimated per period is exact again, blind to the effect, and costs under a tenth of the width — and a ratio that merely wanders at random costs width and no coverage at all.

corner · Nuisance An overlap that depends on where it is read. Each pair's overlap on the doubled log-likelihood scale — the sum of the two searches' own quantiles less the joint search's quantile, as a share of the smaller — at five quantile levels, over 1200 draws under the first-order autoregression. two dictionaries of independent columns: -0.176 at 0.5, -0.000 at 0.75, 0.238 at 0.9, 0.320 at 0.95, 0.538 at 0.99; a break, and an independent column: -1.211 at 0.5, -0.406 at 0.75, -0.029 at 0.9, 0.205 at 0.95, 0.161 at 0.99; two dictionaries of step columns: -0.397 at 0.5, 0.145 at 0.75, 0.438 at 0.9, 0.541 at 0.95, 0.682 at 0.99; a break, and a step column it contains: 1.000 at 0.5, 1.000 at 0.75, 1.000 at 0.9, 1.000 at 0.95, 1.000 at 0.99; a break, and a whitening window: 0.512 at 0.5, 0.640 at 0.75, 0.716 at 0.9, 0.732 at 0.95, 0.754 at 0.99. Every pair but the containment overlaps more at the 95% point than at the median, the control included.

The overlap at the critical value

Two searches over independent noise columns are additive on average: the joint search finds what the two find separately, to within a fortieth of the smaller. At the point a test actually uses — the joint statistic's 95% quantile — the same pair overlaps by 0.320, and a rule that adds the two searches' own critical values rejects 1.4% of true nulls instead of 5%. Nothing about the searches causes it. Quantiles of a sum do not add, and on every pair in the field the overlap at the critical value is larger than the overlap the field measured.

apart · Criterion How often two units contain a break that is there. 300 draws at each of 5 break sizes, a shift from row 60 of 120 on first-order autoregressive errors. The share of draws on which the profile's two-unit set contains the true row: at the fixed, unwhitened window 12.0% at 0.3, 19.0% at 0.6, 31.7% at 1, 51.7% at 1.5, 69.3% at 2; with the window searched 22.7% at 0.3, 30.0% at 0.6, 44.3% at 1, 63.3% at 1.5, 75.3% at 2. A 95% interval would sit at the top of the frame.

A break that is there

Put a real break in the regression at row 60 and the profile's two-unit set still misses it on most draws — it contains the true row on 12% of draws at a small break and 69% at a large one, without whitening; with the window searched as well, 23% to 75%. A set that would contain it on 95% of draws needs a threshold of about 19 units without whitening and about 8.5 with it, and either honest set holds most of the sample: fifty-seven of the seventy-three candidate rows at a break of one standard deviation. Searching the window does not move where the break is found.

twice · Break point Four rules for a break and a column. How often each of four rules says something is there, for the pair of searches over a break and a column, under four truths, 400 draws each under a first-order autoregression at 0.8 with every threshold simulated on 1200 null draws. nothing there: joint, calibrated 6.3%, summed thresholds 6.3%, Bonferroni 6.3%, either, uncorrected 12.0%. a shift from row 60: joint, calibrated 65.3%, summed thresholds 64.0%, Bonferroni 55.0%, either, uncorrected 69.3%. a column enters: joint, calibrated 11.8%, summed thresholds 10.8%, Bonferroni 73.0%, either, uncorrected 82.5%. both: joint, calibrated 70.5%, summed thresholds 69.0%, Bonferroni 75.8%, either, uncorrected 84.8%. The first block is each rule's size.

The power a calibration buys

Charging two searches separately makes a joint test conservative, and calibrating the joint statistic on its own null repairs that. Against a real break, on the pair of a break search and a window search, the repair is worth 47 points of power — 62.0% against 14.5%. On the pair of a break search and a column search it is worth nothing, and worse: a real column is found by the calibrated joint test on 11.8% of samples and by a plain Bonferroni split on 73.0%, because a joint statistic takes its threshold from the noisier of its two searches.

apart · Criterion How often each set holds the break. For a shift added to the response from row 60 of 120 on first-order errors at 0.8, 300 samples at each of five sizes and 120 simulations per sample, the coverage of sets for the break row built at a threshold each method supplies. simulated at the truth: 95.0%, 93.7%, 93.7%, 95.0%, 94.0%; estimated break, true persistence: 90.0%, 88.3%, 88.3%, 89.3%, 90.3%; estimated break, persistence matched: 81.0%, 83.3%, 81.3%, 81.3%, 82.3%; estimated break and persistence: 64.0%, 66.0%, 67.7%, 70.7%, 75.7%; a fixed threshold of 21 units: 96.0%, 94.0%, 96.0%, 97.3%, 98.3%; the two-unit set: 12.3%, 19.7%, 35.0%, 49.0%, 66.7%, at shifts of 0.3, 0.6, 1, 1.5, 2. Simulated at the truth the construction is exact; simulated at what one sample can estimate it misses the true row a quarter to a third of the time.

A set calibrated at the estimate

A confidence set for where a regression breaks needs a threshold on the profile likelihood, and no fixed one works: the honest threshold falls as the break grows. Simulating the profile's deficit at the true break, shift and persistence gives a set that covers 93.7% to 95.0% at every break size and narrows from 67 rows to 23. Simulating it at what one sample can estimate gives a set that covers 64% to 76%, and most of the shortfall is one number: a split regression's residuals report a persistence of 0.62 where the errors have 0.8.

twice · Break point

Named alongside it

The objects these essays reach for when they reach for this one.

CoverageMonte CarloChange pointDegrees of freedomEfficiencyFixed-width intervalInverse variance weightingLikelihood ratioVariance ratioBlindingBlockCritical value

All concepts