Normal Distribution
nothing is called 95% until it has been counted
Concept

Chi-square — where it appears

The distribution of a sum of squared standard normals, which is what a variance estimate is a multiple of when the observations are normal. Its degrees of freedom are the count of independent contrasts behind the estimate, which is what a t interval's width is a function of.

Named by 7 essays across 4 fields — each of them below, with the objects they name alongside it.

One experiment, with the blocks getting smaller as the target comes into range. A single run at a requirement of 0.25, with the block sizes 5, 5, 11, 25, 11, 8, 3, 2 and a total of 70 observations in 8 blocks. The rule stops when the observations in hand reach z²σ̂²/d², with σ̂² pooled from the within-block contrasts — an estimate that moves as the run goes on, so the target moves too. Early blocks are large because the target is far away and cannot be overshot; late ones are small because a block is the granularity of the answer. The interval afterwards is built from the 8 block means and from nothing the rule looked at, and it has 7 degrees of freedom against the rule's 62.

A block size that changes

The blinded rule's exactness never needed the blocks to be the same size. Letting the size be chosen from the contrasts as the run goes on leaves the coverage exactly where it was — and runs straight into an identity that says what a schedule can and cannot buy.

pace · Stopping No block size is best at both things the procedure claims. Two claims and one dial. The honest interval's half-width falls as the blocks get smaller, because the interval's degrees of freedom are the number of blocks: 0.2602 at blocks of two against 0.2933 at blocks of sixteen. The fixed-width claim — that the mean is within 0.25 of the truth — gets more reliable as they get larger, because the sample size is less variable: 92.40% against 95.00%. Both are computed from the same runs, and the second is reproduced to within a tenth of a point by E[2Φ(d√N/σ) − 1], which needs the sample-size distribution and nothing else. The schedules sit at the bottom left: as narrow as the smallest fixed block and as few observations, with the rule's spread estimate on half as many degrees of freedom again.

Two degrees of freedom, one total

The block size is a dial, and the two things a fixed-width procedure claims move in opposite directions along it. Divide the width by the square root of the sample size and one of them turns out to depend on the number of blocks and on nothing else.

pace · Width How long an honest forecaster looks broken for. The calibration error shown by a forecaster with no miscalibration in it at all, at five record lengths, drawn against one over the square root of the length so that the closed form is a straight line through the origin. It is 0.1252 at fifty forecasts and 0.0090 at ten thousand, against a closed form of √(2K/πn) times the mean root bin variance which gives 0.1257 and 0.0089. The threshold drawn across it is 0.02, a figure routinely read as evidence that something is wrong; the mean falls under it at 1976 forecasts and the 95th percentile at about 4111. Below that, an honest forecaster and a miscalibrated one are being told apart by a statistic that is mostly the sample size.

The miscalibration a perfect forecaster shows

A forecaster whose true reliability is exactly zero shows a calibration error of 0.1252 on fifty forecasts and 0.0090 on ten thousand. Every one of 1,200 blameless hundred-forecast records exceeds the 0.02 routinely read as evidence of a problem, and the mean does not fall under it until 1,976 forecasts.

calibrate · Calibration The squared estimate 1 standard errors from the flat point, exact and linearised. At δ = √n·μ/σ = 1 the exact law of the squared estimate has mean 2.00, variance 6.00 and skewness 2.177; the delta method's normal has mean 1.00, variance 4.00, no skewness, and 30.85% of its mass below zero, where a square cannot go. The Kolmogorov distance between them is 0.3085.

Where the derivative is zero

The delta method reads a standard error off a tangent line, and at a flat point the tangent says the spread is zero. The interval built on it for a squared mean covers 99.991% there and 85.978% one and a half standard errors away, with nearly every miss on the same side — and the law it should have used is a χ², not a normal.

normal · Clt How often each reading rejects when every null is true. At the optimal combination of eight forecasts, where all eight encompassing nulls are exactly true, the share of 1,200 draws on which each reading rejects at 5%, at 60, 120, 240, 480 origins. The joint Wald test against a chi-square on seven degrees of freedom: 26.6%, 16.4%, 9.8%, 8.6%; with Hotelling's finite-sample F: 15.1%, 12.4%, 8.3%, 8.0%; the largest of the eight read as a single comparison: 25.0%, 25.9%, 24.7%, 23.4%; the largest with Bonferroni over eight: 5.2%, 4.9%, 4.7%, 4.4%.

One test for eight nulls

A combination of eight forecasts can be tested against its parts one comparison at a time or all at once. All at once is a Wald test on the vector of eight encompassing means — which has seven degrees of freedom, not eight, because the means sum to zero on every origin when weighted by the combination. On sixty origins it rejects a true null 26.6% of the time, Hotelling's correction brings that to 15.1%, and handed its own true critical value it is less powerful than Bonferroni over the largest of the eight against both wrong combinations measured: 12% against 18%, and 47% against 96%.

search · Multiplicity The law is the eigenvalues, and nothing else. The mean and the skewness of n(ĝ − g) at the stationary point, measured over 40,000 draws, against the closed forms: half the sum of the Hessian's eigenvalues for the mean, and for the skewness twice root two times the sum of their cubes over the sum of their squares to the power three halves. The worst disagreement anywhere is 0.028. In one variable the second-order law is a single chi-square with one degree of freedom and its sign is the sign of the second derivative; here it is a weighted sum with the Hessian's eigenvalues as weights, so a bowl and a valley differ in both moments and a saddle has both equal to zero.

A flat point with more than one direction

At a stationary point of a function of several means the second-order law is ½ Z′HZ, so the bias is half the Hessian's trace — 2.008 for a bowl, 5.028 for a valley, and −0.006 for a saddle, where the eigenvalues cancel. The saddle's coverage is the worst of the three.

normal · Clt Testing for flatness at 5% before choosing the interval for a squared mean. Coverage of four nominal 95% intervals for μ², exact, against the true mean's distance δ from the flat point in standard errors. The tangent interval Z² ± 2z|Z| has its worst, 85.98%, at δ = 1.45. The second-order interval read off χ²₁ is right only at δ = 0 and falls to 34.22% at δ = 2. The two-stage rule, which uses the second-order interval when a 5% test does not reject flatness, has its worst, 49.72%, at δ = 1.96. The squared exact interval never falls below 95%.

Testing for a flat point first

Test whether the gradient is zero, and if the test does not reject, read the interval off the second-order law that holds at a stationary point. For a squared mean that two-stage rule covers about 50% at its worst whatever the pretest's level — 49.72% with a 5% test, where the tangent interval it was meant to rescue never falls below 85.98% — because its worst sits exactly where the true distance equals the test's critical value, half the samples are declared flat there, and the second-order interval cannot reach the truth from any of them.

normal · Clt

Named alongside it

The objects these essays reach for when they reach for this one.

Confidence intervalCoverageMonte CarloClosed formDelta methodSecond-order delta methodStationary pointBlockingCentral limit theoremDegrees of freedomEigenvalueFixed-width interval

All concepts