Normal Distribution
nothing is called 95% until it has been counted
Concept

Maximum likelihood — where it appears

Choosing the parameter values that make the observed data most probable under the model, all of them together rather than one after another. Where a transformation is involved the likelihood carries its Jacobian, which is the term a rule built on residuals never sees.

Named by 15 essays across 9 fields — each of them below, with the objects they name alongside it.

The dependence, at four removes. Under AR(1) at 0.8, four different sequences all called the dependence. The top line is the law. The middle line is what a sample of 120 errors reports on average — computable exactly, because the expectation of a sample autocovariance is arithmetic once the covariance is known. The lower line is what a candidate's residuals report, which is what every two-step rule in this collection actually reads: a fit removes variance, and it removes more of the persistent part than of the rest. At the first lag the three are 0.800, 0.7773 and 0.7338. The dots are counted from draws and share no arithmetic with the line they sit on; the worst departure is 1.2 standard errors.

A dependence fitted with the line

Every whitening in this collection reads the dependence off a set of residuals, and residuals are not errors. Fitting the two together recovers most of what that costs, and changes almost nothing about the decision it feeds.

together · Dependence More blocks, and the light tail is called wrong more often. The share of records whose three-way family call is right, over 400 records at each block count, at 100 readings a block. A 95% interval for the shape is formed and the record is recorded as calling Fréchet, Weibull or Gumbel according to whether that interval sits above zero, below it, or straddles it. The two signed parents go from 51.5% and 71.0% at 20 blocks to certainty by 100. The light-tailed one goes the other way — 83.0%, 75.5%, 55.0%, 36.5%, 4.8% — because its estimate sits at about -0.1157 whatever the record length, and a longer record only shrinks the interval onto that number.

Three shapes, one limit

A normalised sum has one limit and a normalised maximum has three, indexed by a single number. Twenty blocks put the sign of that number right 97.3% of the time — and naming the family from a light-tailed record gets worse as the record grows, from 83.0% at twenty blocks to 4.8% at five hundred.

extreme · Extremes A weight that balances, and one that unbalances. The standardised difference between the arms on each covariate, integrated over the population rather than counted in a sample. Unweighted, the arms differ by 0.8310 on the first covariate and 0.6015 on the second, which is what makes the raw difference of arm means 2.7102 against a true average effect of 1.0000. Weighting each unit by one over its own assignment probability removes both differences exactly — -2.78e-17 and -5.69e-19, which is machine precision and not a small number — because the weighted density of the treated arm is the population's own whatever the propensity is. Weighting by a score fitted without the second covariate balances the first to 0.0035 and pushes the second out to 0.7057, further apart than doing nothing.

A score that balances

Weighting each unit by one over its own assignment probability drives the standardised difference between the arms from 0.8310 to 2.8×10⁻¹⁷ — exactly, not nearly. A score fitted without the second covariate leaves that covariate at 0.7057, further apart than doing nothing at all.

weights · Weighting One likelihood, three answers. The concentrated Gaussian log-likelihood of one sample of 120 rows under AR(1) at 0.8, as a function of the correlation the errors are whitened at. Three rules put three different numbers on this curve. The two-step rule reads the least-squares residuals and lands at 0.7616, giving up 0.304 of log-likelihood. Iterating moves it to 0.8080 and gives up 0.002. The maximum is at 0.8044. The curve is not flat between them: what a fixed point of the residual update finds is a solution of a different equation, and the difference is the Jacobian term ½log(1 − ρ²), which grows as the correlation does.

Iterating is not maximising

Re-reading a correlation from the generalised residuals and refitting converges in seven steps. What it converges to solves the first-order condition of a sum of squares, and the likelihood has one term more than that.

together · Dependence The threshold buys accuracy and spends exceedances. The mean squared error of the estimated shape against the threshold, split into the square of its bias and its spread, over 600 records of 2000 readings from a a normal parent. At the 0.9 quantile 199 exceedances are left, the bias is -0.1708, the spread is 0.0701 and the total error is 0.0341. The bias falls as the threshold rises because the exceedances get closer to being generalised Pareto; the spread rises because there are fewer of them. The sum is smallest at the 0.925 quantile, at 0.0340, of which 80.6% is still bias — so even the best threshold on this grid is one where accuracy, not spread, is the binding constraint.

The threshold is a dial

A peaks-over-threshold analysis has one knob, and raising it buys accuracy with exceedances. For a normal parent the error is smallest at the 0.925 quantile and 80.6% of it is still bias there — and both diagnostics practitioners use to set the knob lose to a fixed 0.90 rule, one by a factor of 1.590 and one by 11.881.

extreme · Extremes The interval every package reports first does not cover. Counted coverage of two 95% intervals for the 100-block return level of a normal parent, against the length of the record they were fitted from, over 300 records at each length. The level they are about is known in closed form, so this is coverage of a number rather than agreement between two estimates. The delta-method interval covers 80.3% at 25 blocks and reaches only 89.0% at 200; the profile-likelihood interval sits between 94.0% and 94.7% throughout. The gap is not a small-sample effect that lengthening the record removes — it narrows by 8.7 points for an eightfold longer record.

Two intervals for one return level

Two 95% intervals read off the same fits of the same records, against a level known in closed form. The symmetric one covers 80.3% at twenty-five blocks and reaches only 89.0% at two hundred — and 99.24% of its misses are the interval sitting entirely below the truth, which is not the endpoint anybody expects to fail.

extreme · Extremes Estimating a weight you already know is worth doing. The variance of an inverse-probability estimate weighted by a propensity fitted from the sample, over the variance of the same estimate weighted by the true propensity, paired on the same 500 samples of 600 units at each of five settings. Every reading is below one: the stabilised estimator keeps 27.8% of its true-weight variance where the assignment is nearly a coin toss and 72.0% where it is nearly decidable, and the unstabilised one 30.0% and 49.3%. Neither estimator is materially biased, so this is a variance rather than a trade. The true weights are right about the population and know nothing about the draw; the fitted weights are the value that sets this draw's own imbalance to zero, and that imbalance was what the variance was made of.

The estimated weight is the better one

The propensity is known exactly here, so it can be weighted by — and estimating it from the same data and weighting by that gives a variance ratio of 0.4769 on paired draws. The reason is a projection: the draw's own imbalance explains 56.33% of the true-weight variance and 0.05% of the estimated-weight one.

weights · Weighting Five upper limits for the mean of 15 observations from a exponential source, by margin and by failure rate. t limit: exceeded 8.51% at a mean margin of 0.523; Hall's transformation: exceeded 4.61% at a mean margin of 1.038; gamma family, shape known: exceeded 2.59% at a mean margin of 0.787; gamma family, shape by likelihood: exceeded 4.04% at a mean margin of 0.746; gamma family, shape by skewness: exceeded 17.51% at a mean margin of 0.421. The maximum-likelihood shape has a median of 1.10 and the skewness-based shape 3.20, against a true shape of 1.

A limit from the family it came from

On fifteen exponential observations an upper limit for the mean built from the gamma family's own exact pivot, with the shape estimated by maximum likelihood, is exceeded 4.04% of the time at a mean margin of 0.746 standard deviations. Hall's transformation is exceeded 4.61% at 1.038, and the same family limit with the shape read from the sample skewness is exceeded 17.51%. Knowing the family and estimating its one parameter well recovers most of what knowing the population buys — and the parameter is estimated well only from the logarithms.

tails · Student What the two-regime residue was made of, persistent first. Under a break in the persistence at row 60 of 120 (0.95 then 0.65), the selection regret of six whitenings over 200 draws: restart, unscaled, residual persistences 0.04043; restart, unscaled, persistences given 0.03488; restart, unscaled, fitted with the line 0.03466; the law's recursion, residual persistences 0.03856; the law's recursion, fitted with the line 0.01640; the law's own covariance 0.01579. The whitening that restarts at the break and does not rescale its two regimes gives up 0.01910 against the oracle even when it is given the true persistences; the law's recursion, which divides each row by its regime's innovation scale and continues across the break, is the oracle when given them and within 0.00061 of it when they are fitted with the regression.

The residue was the weights

A regression whose errors change persistence at a known row was whitened as two autoregressions and still gave up 0.0246 against the law's own covariance. That residue was put down to estimating the two persistences from residuals, and the repair named was to fit them with the line. Fitting them with the line moves the persistent regime's estimate from 0.81 to 0.89 and recovers a fifth of the residue — exactly what being told the true persistences recovers. The rest was the transform: it never divided the two regimes by their own innovation scales, which differ sixfold. The law's own recursion, told the persistences, is the oracle to machine precision; fitted with the line, it estimates the persistent one at 0.945 and comes within 0.0006 of the oracle.

general · Dependence Six intervals for the white predictor, a five-period moving average. Under a five-period moving average, over 200 regressions of 120 rows with an eight-lag band, how often each 95% interval for the white predictor's coefficient covers it: two-step band, as reported 95.5%; joint fit, as reported 72.0%; joint fit, White's sandwich 71.0%; joint fit, four-lag sandwich 71.0%; joint fit, band maximised again 85.5%; the law's own covariance 92.0%. The bootstrap redraws each regression 24 times from its fitted coefficients and fitted band, maximising the band afresh on every redraw.

An interval that knows the band was maximised

A regression whose coefficients and eight-lag error band are maximised together prints an interval that covers 72% to 83% where 95% was promised, because it reports nearly the precision of a known covariance and delivers less. Two repairs were named. A sandwich standard error read off the whitened residuals repairs nothing — 71% to 83% — because the maximisation has already made those residuals look independent. A parametric bootstrap that redraws from the fitted band and maximises the band again on every redraw repairs about half the gap, to 85.5% to 89%, by pricing the maximisation through repeating it. What it cannot price is that the band it redraws from is the maximised one.

family · Dependence Where two profile likelihoods put the break. Under a break in the persistence at row 60 of 120, and the same break turned over, over 300 draws: how often each search's estimate lands within twelve rows of the truth, with its average and spread beside it. persistent first · one scale for the whole series: 46.7%, at 74.8 with a spread of 11.8; persistent first · a scale per regime: 94.3%, at 61.8 with a spread of 5.8; persistent last · one scale for the whole series: 49.3%, at 45.3 with a spread of 11.7; persistent last · a scale per regime: 93.0%, at 56.7 with a spread of 5.9.

The break a shared scale hid

The break searches in this field all maximised a likelihood with one innovation variance for the whole series, and on a break from 0.95 to 0.65 they found it badly: an average of 74.8 where the truth is 60, within twelve rows on 47% of draws, from a profile everyone called flat. Give each regime its own innovation variance — one more parameter, and the one the law actually has — and the same search lands at 61.8 with a spread of 5.8, within twelve rows on 94% of draws, reporting four times the likelihood. The break was never barely there; the likelihood was blind to the half of its evidence that is in the scale. Under no break the extra parameter costs about one and a half units of manufactured likelihood.

charged · Break point An estimate reported as a function of an assumption. What the slope really is, against a shift in the outcomes nobody saw — line from the closed form, dots counted over 2000 studies of 200 rows at 35.0% missing. Every point on this line produces exactly the same observed data, and the complete-case estimate is the flat line at 0.5996 regardless. The truth moves at -0.2845 per unit of shift, which is a function of the missingness model and the missing fraction and of nothing that can be estimated: across the swept range the true slope runs from 0.8845 to 0.3155, a span of 0.5691 against a value of 0.60 in the world where the shift is zero. Reporting the line is the honest form of the answer.

The mechanism the data cannot see

Two worlds produce identical covariates, identical patterns of what is recorded and identical recorded outcomes, to the last bit. Their true slopes are 0.6 and 0.315452, and the truth moves at 0.284548 per unit of an assumption nothing in the data can inform.

missing · Missingness A peak where the recorded data have none. The profile log-likelihood of a selection model in cy, the coefficient that lets the chance of being recorded depend on the outcome itself, for one study of 800 rows whose missingness is at random, with residuals normal; every other parameter is maximised at each fixed value. The model assumes the outcome is normal given the covariates. The curve peaks at cy = 0.35, where the fitted slope is 0.839, and the values of cy within the 95% cut run from −0.13 to 0.75; the likelihood-ratio statistic against cy = 0 is 1.47. The study was drawn with cy = 0.00. With the outcome's law left free, every value of cy fits the recorded rows equally well and this curve would be flat: its curvature is the normal assumption.

The assumption that identifies the mechanism

A selection model estimates how strongly an outcome decides whether it is recorded — the quantity two identical datasets showed no statistic can see — and it does so by assuming the outcome is normal. Where that holds and the outcome does decide, it repairs a slope complete cases put at 0.4318 to 0.5795. Where the missingness is at random and the residual is merely skewed, it reports selection that is not there, moves the slope from 0.5971 to 1.0319, and rejects missingness at random in 72.5% of studies.

missing · Missingness What the sixteen-run factorial's surviving runs estimate when about 1.3 runs are lost, by why they were lost. A 2⁴ factorial fitting eleven coefficients; true A = 1, B = 0.5, AB = 0.5, everything else 0, noise 1; 20,000 datasets. Runs are lost when the response exceeds 2.5, or at random at the high corner of A and B, or at random anywhere, each losing 1.26 runs on average. Bias of the fit on the surviving runs — lost at random: A 0.001, B 0.003, AB 0.000, C -0.002; lost at the high corner: A 0.003, B 0.001, AB 0.000, C -0.000; lost when the response passes the threshold: A -0.131, B -0.120, AB -0.117, C -0.000.

A run lost for a reason

A sixteen-run factorial that loses a run at random loses precision and nothing else. One that loses a run because the process failed — the response passed a threshold — loses about the same number of runs and biases exactly the effects that pushed the response there: an interaction of 0.5 reads 0.383, a main effect of 1 reads 0.869, and the interval around each still covers about 94%. Treating the lost runs as censored above the threshold overshoots with eleven coefficients and works with six, because a run's own leverage decides how much a censored observation pulls its own estimate.

design · Factorial How often a skewed study reports selection that is not there, by the strength of a variable that moves recording only. 120 studies of 800 rows at each strength, missing at random on the regressor with a right-skewed residual, 35% unrecorded. The variable w shifts the chance of being recorded and has no place in the outcome; its share of the recording index's variance is 0.0%, 9.3%, 29.1%, 62.1% at strengths 0, 0.5, 1, 2. The selection model that includes w rejects missingness at random on 75.8%, 27.5%, 7.5%, 5.0% of studies; the same model fitted without w rejects on 78.3%, 85.0%, 100.0%, 100.0%.

A variable that moves recording

A selection model fitted to a skewed outcome that is missing at random reports selection that is not there on three studies in four. Give it a variable that shifts who is recorded and has no place in the outcome — strong enough to carry 29% of the recording index's variance — and the false reports fall to 7.5%, the slope from 1.034 to 0.613 against a truth of 0.6. Leave the same variable out of the model and it does harm instead: false reports reach 100%, and where the outcome really does decide recording, the estimated pull falls from 1.18 to 0.78.

missing · Missingness

Named alongside it

The objects these essays reach for when they reach for this one.

AutocorrelationClosed formConfidence intervalGeneralised least squaresMissing at randomMissing not at randomModel misspecificationMonte CarloProfile likelihoodRegretStandard errorBias

All concepts