Normal Distribution
nothing is called 95% until it has been counted
Concept

Heteroskedasticity — where it appears

Error variance that is not the same everywhere, usually because it depends on the design or on the past. A resampling that detaches a residual from its own row destroys it, which is why a multiplier that leaves the residual in place is the repair for it and a block resampling is not.

Named by 17 essays across 10 fields — each of them below, with the objects they name alongside it.

The same copula, turned over. A Clayton copula and its reflection, at the same Spearman correlation of 0.40 and the same Kendall tau of 0.275, against the covariate's marginal. With a symmetric covariate the two are the same number to nine decimals — 7.707% apiece — because the leak then depends on how much asymmetry the copula has and not on which way it points. Skew the covariate and they come apart: at a skewness of 2.26 the lower-tail copula leaves 3.431% and the upper-tail one 36.213%, a factor of 10.6. Both halves of the dependence are asymmetries and an asymmetry has a direction; a lower-tail copula concentrates the dependence where a right-skewed marginal is compressed and the two distortions partly undo each other, and an upper-tail one concentrates it where the marginal is stretched.

A symmetry that was not enough

A heavy-tailed symmetric covariate has a skewness of zero and leaks exactly nothing under three copulas. Under the two asymmetric ones it doubles the leak, from 7.707% to 14.229%.

compound · Adjustment How far each reference distribution's 95% point falls short. Seven constructions on rows that repeat each other, at a block length of 5 and 200 draws, against the statistic's own 95% point of 3.0224 computed from three thousand draws of the same world. Reading down: a multiplier on every row keeps no dependence at all and is 44% short; a multiplier shared along a block keeps the triangle; a fixed block keeps the same triangle and is 8% closer, which is the pair that says a taper is not what decides this; the stationary bootstrap; the two tapered blocks, both further short than the untapered one at this block length; and errors generated from a fitted model, which is the only construction here not bounded by what the residuals report.

A taper and a critical value

Two constructions whose tapers visibly differ give the same critical value, and two that share a taper exactly do not. Adding a construction whose taper is a decision rather than an accident says which half of that is true.

taper · Reference Three sets of weights, five designs, and no estimator that is exact everywhere. Coverage of the same interval under three weightings. h_b is the inverse variance when the arms share a variance or the allocation is constant; equal weights are right when every block has the same two counts; the estimated precision weights are right in the limit and exact nowhere, because the decomposition needs the weights to be the constants they are only estimating. In the corner — two variances, changing sizes, changing allocation — the two exact estimators are the ones that miss, at 98.45% and 95.65%, and the one with no theorem behind it is at 95.05%. That is the whole statement: there is an exact estimator under either condition, and none under both.

Which weights are the inverse variances

There is an exact estimator when the two arms share a variance and another when every block has the same two counts, and between them they cover every trial anybody designs on purpose. In the corner where neither holds, both cover 98.45% instead of 95%, and the only estimator at its level is the one with no theorem behind it.

contrast · Nuisance What each error is a claim about, and what the claim comes out as. Each variance estimate's average over 20000 draws, divided by the variance the slope actually has across those same draws, at 80 rows with the error variance leaning towards the edges of the design (γ = 0.8). One is a standard error that is right. The model-based estimate reads 0.6081 of the spread, so its standard error is 77.98% of the one it should report; the four robust corrections read 0.9576, 0.9821, 0.9961, 1.0362. Two further routes agree with the count and share none of its arithmetic: n times the counted variance is 4.8905 against a population sandwich of 4.9200, and the counted ratio of the two standard errors is 1.2799 against a closed form of 1.2806.

The bread and the filling

The robust standard error is not a safety margin. At one setting of the error variance it is 1.2806 times the model-based one and at another it is 0.8246 times it, and the sign of a single dial decides which.

sandwich · Misspecification Which repair goes with which defect. The share of true nulls rejected at a nominal 5% by a reference distribution generated from the fitted benchmark, over 120 draws with 59 resamples each. Where the errors are well behaved every resampling is fine and all four are conservative. Where the variance is a function of the design, the two that detach a residual from its own row reject 5.8% and 8.3% — and the block bootstrap, which is the resampling three earlier fields on this site reach for, repairs nothing at all, because the dependence it is built for is between origins and the rolling scheme reproduces that on its own. Where the errors are skewed the symmetric multiplier is the one that is wrong, and Mammen's two-point version is the only one of the four that is right in both columns.

Residuals that keep their own variance

A reference distribution for a search has to be generated from a fitted model, and the generator draws residuals. Four ways of drawing them keep four different things — and the one this site has reached for three times repairs nothing at all here.

select · Bootstrap Two constructions on one triangle, and a third that is not. Three resamplings that all keep runs of neighbours, on the same residuals at a block length of 5, with the lags running past ℓ so that the tapers separate. A blocked multiplier never moves a residual; a fixed-length moving block moves every one; and they attenuate identically, worst gap 1.4 standard errors, both sitting on γ_resid(k)(1 − k/ℓ)⁺ and both exactly zero past ℓ — so the attenuation is the block boundary rather than the multiplier. The third is the stationary bootstrap, whose runs are geometric rather than fixed: its taper is γ_resid(k)(1 − 1/ℓ)^k, it agrees with the other two at the first lag and at no other, and at lag 6 it still carries 0.0081 where they carry -0.0005.

The triangle that was not the multiplier's

A resampling that leaves each residual on its own row can keep only what the residuals have, times a triangle. A construction that moves every one of them has the same triangle — and the one in this collection's own table has a different taper entirely.

banded · Bootstrap Each repair is for its own defect, and one is for both. The 95% point of the statistic's own distribution in each world, against the mean 95% point of five reference distributions built from one sample. Where the error variance is a function of the design, the two resamplings that detach a residual from its row fall short and the two multipliers that keep it there do not; where the rows repeat each other it is the other way round. With both defects at once the blocked multiplier — drawn once per run of 5 rows, so the residual never moves and its neighbours share a sign — is the closest of the five, at 2.999 against a truth of 3.803. It is still short by 0.804, and that shortfall is the next figure.

Two defects and one resampling

Four resamplings, each the repair for one defect and wrong about the other. Put both defects in the same world and the statistic's 5% point is 3.8028, where the best of the four reaches 2.8326 — until a multiplier that stays on its own row and shares a sign with its neighbours reaches 2.9988.

proxy · Bootstrap One promise, kept on average and inside neither group. What each calibration scheme covers inside each of two equally common groups whose noise scales are 1 and 3, over 6000 draws with 200 calibration points. One interval for everybody covers 100.00% of the quiet group and 90.66% of the noisy one, averaging to 95.28% — and the closed form for that population says 99.9999% and 90.0001% at a half-width of 4.9346, from two normal cdfs and no simulation. Dividing by an estimated per-group scale gives 95.29% and 95.48%; calibrating separately inside each group gives 95.49% and 96.01% against a closed-form expectation of 95.4645%. Only the last of those is a guarantee rather than a repair, because the rank argument runs inside each group.

Marginal is not conditional

One exactly valid interval covers 100.00% of a quiet group and 90.66% of a noisy one, and the floor is arithmetic rather than a measurement — a group of share π is guaranteed only 1 − α/π, which is zero when the group is as rare as the miss rate.

conformal · Exchangeability Robust, at the sample sizes it is reached for. Counted coverage of five 95% intervals for a slope, at six sample sizes, under an error variance leaning towards the edges of the design (γ = 0.8), over 20000 draws at the small end. The model-based interval sits at about 87.06% everywhere and does not improve with the sample, because it is a claim about a variance it is not estimating. The robust ones do improve: HC0 covers 88.73% at 20 rows, 92.70% at 50 and 94.93% at 1,000. Its promise is asymptotic and its use is not, and the gap between those two facts is this picture. The leave-one-out correction read against a t on n − 2 is the only line that is near its promise at the small end: 94.55% at 20 rows.

Robust is not free

A robust standard error's promise is asymptotic and its use is not. Its 95% interval covers 88.73% at twenty rows, and under mild heteroskedasticity it is the worse of the two intervals until a hundred.

sandwich · Misspecification What each construction carries, against what there was. The autocorrelation of a resampled error series at five lags, averaged over 60 samples of 40 resamples each. Three facts are in the picture. The residuals lie below the errors at every lag, which is the ceiling a multiplier cannot exceed. The blocked multiplier and the fixed-length block lie on top of each other below it — they attenuate identically, because the attenuation is the join — while the stationary bootstrap, whose runs are geometric rather than fixed, sits above them both. And the sieve is the exception in kind rather than in degree: at lag six it carries 0.0638 where the residuals have 0.0300 and the multiplier has -0.0011, because a fitted model extrapolates past the lags it was told about and a truncated sample sequence cannot.

Errors generated from a fitted model

The one construction that is not bounded by the residuals, because a model extrapolates past the lags it was told about and a truncated sample sequence cannot. It is nearly exact where the only defect is dependence, and it pays for it where there are two.

banded · Reference A bias against a variance, with the answer in between. How wrong one sample's reference distribution is, split into the two things it is wrong by. Sharing the multiplier over more rows keeps more of the dependence and closes the bias from 1.688 to 0.835; every row it is shared over also removes an independent sign from the 101 the sample started with, and the spread of the resulting quantile rises from 1.307 to 2.172. The distance a practitioner with one sample is actually exposed to is the two together, and it is smallest at ℓ = 5.

How long a block a multiplier shares

Sharing a sign over more rows keeps more of the dependence and leaves fewer independent signs to build a distribution from. The bias falls from 1.6885 to 0.8479 and the spread rises from 1.3073 to 2.1716, and the rejection rate walks straight through its nominal level on the way from 11.3% to 1.3%.

proxy · Reference Six scores, one coverage. What each nonconformity score's interval covers, over 2000 draws with 200 calibration points, against the 95.0249% the rank argument promises. The column runs from 94.30% to 94.90%, a spread of 0.60% against a standard error of a difference of 0.69% — one number, six times. That includes a score aimed five units off the fit and a score that never reads the response at all, because the rank argument does not read the score either: it needs the scores exchangeable and nothing else. Every decision a modeller makes has to show up somewhere else, and the next two readings are where.

The score is the modelling

Six nonconformity scores on the same draws cover within 0.60 points of each other, against a standard error of a difference of 0.69 — one number six times. Their widths run over a factor of 2.361 and their adaptivity over a factor of 8.377.

conformal · Exchangeability The ceiling a multiplier cannot reach past. A wild-type resampling forms e*_t = e_t·w_t with the multiplier independent of the residual, so what comes out has autocovariance γ_resid(k)·γ_w(k) — the residuals' own, multiplied by the multiplier's. Since |γ_w| ≤ 1 the reference distribution's dependence is bounded above by the residuals', and the residuals' is already below the errors'. The two shortfalls compose. For a block of ℓ the multiplier's autocorrelation is exactly the triangle (1 − k/ℓ)⁺, drawn here as the dashed prediction against the realised resamples at ℓ = 5; the bound is attained only at ℓ = n, where the reference distribution is built from one sign.

What a multiplier cannot keep

Two reasons were named for the quarter a blocked resampling falls short, and taking either away makes the gap larger. What is left is a bound — a multiplier can only take dependence out, and the residuals' own is already below the errors'.

effective · Reference The price of insurance is noise, not coverage. What a robust standard error costs under a constant error variance — the case where the model-based one is exactly right — at five sample sizes over 20000 draws at the small end. It is not coverage: the leave-one-out interval read against a t on n − 2 covers 95.52% at 20 rows against the model-based 95.06%. It is a wider interval, by a factor of 1.0689 at 20 rows falling to 1.0049 at 250, and it is a variance estimate 2.57 times as variable at 20 rows and 1.85 times at 250 — against a denominator that is exactly V²·2/(n − 2), so only the numerator is counted. The uncorrected estimate is the cheaper of the two at small samples and the dearer at large: 1.31 against 1.76.

Right for the wrong reason

A robust standard error costs no coverage where the risk is absent — 95.52% against 95.06% at twenty rows. It costs a 6.89% wider interval and a variance estimate 2.572 times as variable, and the pre-test that would avoid paying recovers 15.9% of what the insurance is worth.

sandwich · Misspecification An exact test rejecting a true hypothesis a fifth of the time. How often each analysis reports an effect when the average treatment effect is exactly zero and the effect varies between units, at 150 units with 25% treated. The permutation test on the difference in means reads 4.20% where the effect is constant — where the two nulls coincide and its exactness applies — and 22.93% where the effect varies with a standard deviation of 3. The same test on the studentised difference reads 6.27% there, and the ordinary large-sample t, which makes no exactness claim at all, reads 6.60%.

The null the exactness is for

A permutation test is exact under the hypothesis that the treatment changed nothing for anybody. Under the hypothesis it changed nothing on average, with a quarter of the units treated and the effect varying between them, it rejects a true null 22.93% of the time.

exact · Nuisance One statistic that is right under both hypotheses. Rejection rates for both statistics under both nulls, at 25% of 150 units treated, with the weak-null readings taken at an effect spread of 3. The difference in means is exact under the sharp null and rejects 22.93% of true weak nulls. The studentised difference is exact under the sharp null — 4.07% — and reads 6.27% under the weak one. The repair is a change of statistic inside the same construction: the same re-randomisations, the same fixed outcomes, a different number compared across them.

A statistic that is exact twice

Dividing the difference in means by its own separate-variance standard error before permuting takes the rejection rate under a true weak null from 20.47% to 6.07%, keeps the exactness under the sharp null at 4.07%, and costs 0.8 points of power against a real effect. At an even split it changes nothing at all, in every draw.

exact · Nuisance Three detectors for one departure, all at 5%. How often each of three checks on the calibration scores fires, against the size of the drift, with every critical value simulated under no drift so that all three sit at 5.0% exactly. The incumbent — a rank comparison of the first half of the scores against the second — reaches four-in-five power at a growth factor of 4.31. Reading each score's rank against its position reaches it at 2.65, and the largest running departure of the scores from their mean at 2.12. The ordering of the three is the ordering by how much of the sample's arrangement each one uses.

A detector built for the ordering

The best of three checks for a drifting scale fires at half the growth factor the standard one needs — 2.12 against 4.31 — and still leaves 6.50 points of coverage gone before it does, against 0.51 for serial correlation. The reversal was not a property of the test.

conformal · Exchangeability

Named alongside it

The objects these essays reach for when they reach for this one.

Reference distributionMonte CarloCritical valueBlock bootstrapError rateResidualWild bootstrapAutocorrelationResamplingClosed formCoverageBootstrap

All concepts