Normal Distribution
nothing is called 95% until it has been counted
Concept

Hat matrix — where it appears

The projection that takes a response to its fitted values. Its trace is the parameter count, its diagonal is the leverage, and its trace against the error covariance is the optimism a criterion's penalty is estimating — which is the parameter count only when the errors are independent.

Named by 18 essays across 10 fields — each of them below, with the objects they name alongside it.

The dependence, at four removes. Under AR(1) at 0.8, four different sequences all called the dependence. The top line is the law. The middle line is what a sample of 120 errors reports on average — computable exactly, because the expectation of a sample autocovariance is arithmetic once the covariance is known. The lower line is what a candidate's residuals report, which is what every two-step rule in this collection actually reads: a fit removes variance, and it removes more of the persistent part than of the rest. At the first lag the three are 0.800, 0.7773 and 0.7338. The dots are counted from draws and share no arithmetic with the line they sit on; the worst departure is 1.2 standard errors.

A dependence fitted with the line

Every whitening in this collection reads the dependence off a set of residuals, and residuals are not errors. Fitting the two together recovers most of what that costs, and changes almost nothing about the decision it feeds.

together · Dependence The correction is not a property of the sample. tr(HΩ)/q for each of fifteen candidates, at ρ = 0.7. Two candidates that fit the same number of coefficients need corrections that differ by as much as 1.49, because one of them is fitting the persistent predictors and the other is not — so no single number can be right for both, and the scalar n/n_eff = 5.537 is above every one of them. The four predictors carry persistences 0.9, 0.6, 0.3, 0; at one persistence for every column the whole spread collapses and a scalar looks exactly as good as the trace.

A penalty is a trace

Akaike's 2q is not a count of coefficients. It is the answer a trace collapses to when the rows are independent — and once they are not, the trace is still the right object and is no longer the count.

effective · Order selection What the rule blocks is not where it splits. How much of the separating direction each probe carries, over 100 designs of 14 units whose admissible set is enumerated and split into two pieces. The deferral this field answers proposed the constraint's active set — which exchanges the tolerance box actually blocks — as a better probe than the design's own leverage, on the ground that leverage is a heuristic and the active set is the quantity. Modelled from the design and the tolerance, it reads 0.4272 against leverage's 0.5395, at 4.43 paired standard errors the wrong way. Counted exactly over the enumerated set — at a cost no trial can pay — it reads 0.3854, worse again. Both beat a random direction at 0.2622, so they are probes; neither beats the two the earlier field already had.

What the rule blocks

A balancing rule breaks the admissible set into pieces by refusing exchanges. Which exchanges it refuses is computable from the design and the tolerance alone, before any assignment exists — and it makes a probe.

blocked · Randomisation How far apart the two components are, on each probe. The median separation between the two components of the admissible set — the difference in their mean probe values, over the spread inside a component — over the 100 of 200 designs whose set is enumerated and found split. The separating direction carries 10.565 and needs the enumeration. The fourth power as the earlier fields use it carries 1.543; projected off the span the rule balances, 5.080. The design's own leverage, which uses no dictionary and no outcome, carries 3.836. A random direction in the same subspace carries 0.942, and a direction chosen by looking for concentrated structure carries 0.543 — below random, and the one heuristic here that is worse than not choosing at all.

A probe chosen from the design

The design's own leverage aligns with the separating direction four times better than a random direction in the same subspace. The concentrated direction the argument invites is worse than random.

aimed · Randomisation The one candidate an effective sample size is right about. n/n_eff with the finite-sample inflation Σ(1 − |k|/n)ρ^|k| is not an approximation to tr(HΩ) for a fit with only an intercept — it is that trace, to machine precision, because the hat matrix of a constant column is 1/n everywhere and its trace against Ω is the mean of Ω. The quoted limit form n(1 − ρ)/(1 + ρ) is not even right about that one. And the average correction the table's fifteen candidates actually need is 3.318 per parameter, well below the scalar, so applying it to all of them over-charges every one.

One number for a table of candidates

An effective sample size is a real quantity, it is exactly right about one thing, and that thing is a mean. Substituted into Akaike's criterion it changes nothing at all, because the penalty it is meant to fix has no sample size in it.

effective · Dependence How much memory a fit takes out, candidate by candidate. Under AR(1) at 0.8, the lag-one autocorrelation a candidate's residuals report, computed exactly for each candidate on 200 draws. The upper line is the law at 0.8000. A candidate that is an intercept alone reports 0.7773 — which is exactly what a sample of 120 errors reports, because an intercept annihilates the sample mean and nothing else, and the two arithmetics agree to the last bit. Every predictor after that takes more out, down to 0.7341 at the fullest candidate. That is the collision this field is about: the rule every whitening here uses estimates its nuisance once, from the fullest candidate, so that the criteria stay comparable — and the fullest candidate is the one whose residuals report the least.

The fit that takes the memory out

A candidate's residuals report less dependence than its errors do, and how much less is arithmetic rather than noise. The rule used for a good reason reads the series that has lost the most.

together · Dependence What each probe can see. How far apart the two components of the admissible set are on each probe, over the spread inside a component, on 100 designs whose set is enumerated and split. It is the population quantity a chain is trying to report. The separating direction itself reads 10.5646; the projected fourth power 5.0800, the design's own leverage 3.8362, the modelled active set 1.9529, the counted active set 2.0170 and a random direction in the same subspace 0.9422. The two active-set probes beat the random direction and lose to both of the earlier field's, which is the field's answer to the question that opened it.

A quantity that loses to a heuristic

Leverage is a heuristic about which units a balancing rule has most to say about. The constraint's active set is the thing the rule actually does. As a probe, the heuristic wins by 4.4 paired standard errors.

blocked · Randomisation One of them is mostly leverage. How much of the design's own leverage direction each active-set probe carries, once both are standardised and projected off the rule's span — which is what a probe is, so it is the comparison that matters. Over 189 designs the modelled active set agrees with leverage at |r| = 0.8359 ± 0.0114 and the counted one at 0.4239 ± 0.0216. So the modelled probe is largely leverage under another name and the counted one is genuinely a different direction — and the counted one is the worse probe, at 0.3854 of alignment against 0.4272. What the active set contains beyond leverage points away from where the set splits.

Counting it exactly does not help

If a modelled active set lost because the model was crude, the exact one would win. It is computed at a cost no trial can pay, and it is worse — so the approximation was never what was costing the probe.

blocked · Randomisation A fit takes the low frequencies out of what it leaves behind. The autocorrelation of the errors, of the residuals of a fitted benchmark, and of those residuals rescaled by their own leverage. (I − H) removes the component of the errors lying in a column space that is itself slow-moving, so the residuals are less persistent at every lag — by 5.9% at the first and 26.6% by the fourth. The leverage correction is the standard repair for what a fit does to a residual's size; drawn here against what it does to a residual's dependence, it does nothing.

The residuals are not the errors

A fit removes the part of the errors lying in its own column space, and a persistent design's column space is itself slow — so what is left behind is smoother than what went in, at every lag, by an amount that grows with the lag.

effective · Bootstrap What each correction is worth, exactly. Each variance estimate's expectation under a constant error variance, divided by the variance the slope actually has, at 20 rows on an even design, by two routes: the closed form E[eᵢ²] = σ²(1 − hᵢᵢ) carried through each correction's own weight, and the mean of 20000 counted estimates. The maximum leverage here is 0.1857 and the design's fourth-moment share Σu⁴/(Σu²)² is 0.0897, which is the only thing the closed form reads. HC0 comes out at 0.8603 — short by construction, since its factor is exactly 1 − 1/n − Σu⁴/(Σu²)². HC1 reaches 0.9559, HC2 is exactly 1.0000 at every design and every sample size, and HC3 overshoots to 1.1647. On an even design the four are within a fifth of each other and the choice barely matters.

Three corrections and a leverage

On an even design of twenty rows the four robust corrections read 0.8603, 0.9559, 1.0000 and 1.1647 of the truth and the choice barely matters. Add one point at x = 8 and they read 0.3191, 0.3419, 1.0000 and 5.1127.

sandwich · Misspecification Two far rows, and the line with one of them deleted. Twenty clean points and two rows near x = 9. The slope is −0.511 with every row, −0.376 with one far row deleted, and 0.495 with both deleted. Deleting one of them barely moves the line, because the other is still there.

Two points that hide each other

One far observation among twenty-one has a Cook's distance of 24.1. Put a second beside it and the two read 0.966 and 0.772, neither crossing 1, while together they reverse the slope and deleting both moves the fit by 53.3.

regression · Leverage How many of 100 points a fitted slope rests on, by the distribution the covariate was drawn from. The effective number of points behind a slope is n divided by the design's kurtosis. Over designs of 100, its median share is two points, half at each: 1.000; uniform: 0.551; normal: 0.349; exponential: 0.160; Student's t, 3 df: 0.148; lognormal, σ = 1: 0.076 — 34.9 of 100 points for a normal covariate and 7.6 for a lognormal one.

The points a slope rests on

Before any outcome is measured, a regression's design already says how many of its points the slope will rest on: n divided by the covariate's kurtosis, exactly. A hundred points of a normal covariate carry a slope on about 35 of them; a hundred of a lognormal one, on 7.6. The largest leverage follows the same tail — above one half in 15.7% of lognormal designs of a hundred and in none of the normal ones — and on a correct straight line, deleting the single highest-leverage point moves the slope by more than its standard error in 16.3% of lognormal datasets of a hundred and 0.1% of normal ones.

spread · Summary How often a quantile plot's shape test sees a skewed error distribution, on the errors and on residuals of fits of more columns. On the forty errors themselves the test flags 82.9%. On the residuals of fits with 2, 5, 10, 20 columns: 79.1%, 71.4%, 51.2%, 23.0%. On 40 − p errors, which is what losing the fitted columns' worth of observations alone would give: 80.6%, 76.1%, 69.1%, 49.5%.

Residuals that look too normal

The quantile plot most often read is of a regression's residuals, and residuals look more normal than the errors behind them. Each residual is a mix of all the errors, and the mix moves towards a bell as the fit takes more columns. On forty rows with skewed errors, the straightness test flags 82.9% of samples of the errors themselves, 79.1% of the residuals of a line, 71.4% with five columns, 51.2% with ten and 23.0% with twenty — far below the 49.5% that twenty errors would give, so the fit costs more than the observations it uses. Light tails vanish faster: 73.0% on the errors, 4.7% on twenty-column residuals. The band's level holds at every design, and one point with leverage 0.73 costs almost nothing; what costs is the number of columns.

lineup · Qq A point that decides a coefficient, on two covariates correlated at 0.9. A hundred points on two normal covariates with correlation 0.9. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the first coefficient's information, 14.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 26.1 effective points, with a variance inflation of 6.27.

A point ordinary on every axis

In a multiple regression each coefficient rests on its own number of points, n divided by the kurtosis of that covariate's residual after the others, exactly. Correlation between normal covariates inflates a coefficient's variance fivefold and leaves it resting on the same third of the points. What correlation does instead is hide the point that decides the coefficient: at a correlation of 0.9 it stands out on no single covariate's plot in 77.3% of designs of a hundred, and the largest leverage a normal design reaches depends on the number of points and covariates and on nothing else.

spread · Summary Three charges and two counts. The optimism of an in-sample fit of the widest candidate on 120 rows, in units of its own error variance, over 1500 draws, by persistence. Counted on the 40 rows straight after the sample: 11.1, 21.5, 35.6, 51.5, 65.5, 87.8, 119.1. Counted on rows 600 further on: 9.3, 18.3, 33.0, 50.6, 67.0, 96.1, 159.0. Charged by the long-series inflation: 10.0, 19.3, 34.7, 54.4, 74.3, 114.2, 234.1. Charged by the same inflation summed over the sample's rows: 10.0, 19.2, 34.2, 53.1, 71.8, 108.2, 208.8. Charged by twice the trace of the hat matrix against the errors' correlation on the sample's own design: 10.0, 18.7, 32.0, 47.5, 61.5, 85.2, 132.2.

A charge the sample can afford

The inflation charge for dependent rows runs to twice the counted optimism at persistence 0.95, and the reason offered was that a series of n rows cannot be optimistic about movements slower than n rows. Summing the inflation over n rows does almost nothing: 234 becomes 209, against 119 counted. Evaluating the same variance on the sample's own design gives 132, and moving the fresh rows away from the sample raises the count to 159. So the charge can be made nearly right, and when it is, the regret doubles from 0.032 to 0.061. A selection wants a much larger charge than the optimism, and past about fifty per predictor it is no longer selecting.

proxy · Forecast The excess over the Gaussian displacement, by window. By how much the expected leverage per parameter exceeds the Gaussian closed form, with the fresh row integrated out, at windows 20, 40, 80, 160, 320, 640. normal: -0.90%, -0.61%, -0.04%, -0.06%, -0.02%, 0.00%. t on eight: 4.14%, 2.78%, 1.79%, 0.73%, 0.28%, 0.18%. t on five: 10.70%, 7.50%, 3.93%, 2.01%, 1.26%, 0.70%. t on three: 44.44%, 32.67%, 22.62%, 16.43%, 11.70%, 7.63%. exponential: 27.91%, 14.01%, 7.21%, 3.53%, 1.93%, 0.85%. On normal predictors the closed form is exact from a window of 80.

An excess and its window

The parameter-count displacement is a Gaussian fact, and on heavy-tailed predictors it runs a few per cent over. Swept across windows from 20 to 640 rows, the excess is about the predictors' excess kurtosis divided by the window: a law with a fourth moment of nine on 80 rows is 7.5% over, and on 640 rows under one. At three degrees of freedom it falls only as the root of the window, from 44% to 7.6%. And the instability once blamed on that law was not there: with the fresh row integrated out, the batches agree to a point and a half, and the mean exists.

select · Multiplicity Whether residuals built to be independent see a skewed error better than ordinary ones. The straightness test, each version read at its own 5% point. On all forty errors: 82.9%. At 2, 5, 10, 20 columns: ordinary residuals 79.1%, 71.4%, 51.2%, 23.0%; BLUS residuals 76.4%, 64.9%, 38.3%, 16.2%; recursive residuals 64.4%, 49.5%, 25.9%, 9.1%; 40 − p errors 80.6%, 76.1%, 69.1%, 49.5%.

Residuals made independent

Recursive residuals — each row predicted from the fit to the rows before it — are exactly independent normals under the null for any design, so a quantile plot of them has an exact band. They see a skewed error worse than the ordinary residuals do, not better: 25.9% against 51.2% at ten columns on forty rows, 9.1% against 23.0% at twenty. BLUS residuals, the independent set closest to the errors, reach 38.3% and 16.2%. Every such set is a rotation of the ordinary residuals, so none holds information they lack, and independence is bought by mixing some residuals almost entirely out of other rows' errors — the least pure recursive residual at twenty columns is 0.4% its own.

lineup · Qq Coverage of the cluster-robust slope interval at six, ten and twenty clusters of unequal size, by reference. 5,000 studies of 300 rows at each setting, within-cluster correlation 0.2. 6 clusters, equal sizes: 89.1%, 92.6%, 96.0%; 6 clusters, 5:1, big at the edges: 80.1%, 88.1%, 95.2%; 6 clusters, 5:1, big at the centre: 93.8%, 95.4%, 96.8%; 6 clusters, 15:1, big at the edges: 76.2%, 86.8%, 94.5%; 10 clusters, equal sizes: 91.1%, 93.0%, 95.8%; 10 clusters, 5:1, big at the edges: 86.3%, 90.3%, 94.7%; 10 clusters, 5:1, big at the centre: 94.5%, 95.7%, 96.4%; 10 clusters, 15:1, big at the edges: 84.9%, 89.6%, 94.8%; 20 clusters, equal sizes: 92.9%, 93.8%, 95.1%; 20 clusters, 5:1, big at the edges: 90.9%, 92.7%, 94.8%; 20 clusters, 5:1, big at the centre: 94.5%, 95.0%, 95.6%; 20 clusters, 15:1, big at the edges: 90.6%, 92.4%, 94.7% — for the usual variance on G − 2, the bias-reduced variance on G − 2, and the bias-reduced variance on its effective count.

An effective number of clusters

Ten clusters of thirty rows, a covariate spread evenly across them, and a cluster-robust interval for the slope read against a t on G − 2 = 8 covers 91.1%. The slope needed about five degrees of freedom, not eight, because the two outermost clusters carry 49% of its variance and the middle two 0.6%. A count computed from the sizes and positions before any outcome is seen — 4.45 here, 3.32 when the big clusters sit at the edges — with the bias-reduced variance covers between 94.5% and 96.8% in all twelve layouts measured, where the usual interval on G − 2 covers between 76.2% and 94.5%.

sandwich · Misspecification

Named alongside it

The objects these essays reach for when they reach for this one.

LeverageLeast squaresClosed formModel diagnosticsAutocorrelationEffective sample sizeExperimental designModel selectionResidualAssignment mechanismConnected componentCovariate balance

All concepts