Normal Distribution
nothing is called 95% until it has been counted
Concept

Generalised least squares — where it appears

Fitting by minimising a residual sum weighted by the inverse of the error covariance, which is the efficient estimator when the errors are correlated. Applied to a first-order autoregression it is a one-line transform of the sample, and in the transformed sample an information criterion's ordinary penalty is correct again.

Named by 20 essays across 8 fields — each of them below, with the objects they name alongside it.

The price of each thing the rule is not told. What each rule gives up against the best model available, at a persistence of 0.85 on a fifteen-candidate table, over 400 draws. Reading down: least squares with the ordinary penalty; the whitening at the true ρ; the same at a ρ̂ estimated per candidate; that rule with the term the Gaussian likelihood carries and it omits; a Bartlett-tapered Ω̂ estimated once from the fullest candidate at L = 8; the same estimated per candidate; and the truncated Ω̂, which exists on only 45.0% of draws and is averaged over those. Knowing ρ recovers 89.9% of what counting rows gives up, estimating it 83.4%, and estimating a whole covariance 75.6%.

A covariance with no parameter in it

The whitening that repairs a criterion is told the dependence is a first-order autoregression and left to find one number. A real dependence is not one number, and the obvious estimate of it is not a covariance matrix.

banded · Dependence The dependence, at four removes. Under AR(1) at 0.8, four different sequences all called the dependence. The top line is the law. The middle line is what a sample of 120 errors reports on average — computable exactly, because the expectation of a sample autocovariance is arithmetic once the covariance is known. The lower line is what a candidate's residuals report, which is what every two-step rule in this collection actually reads: a fit removes variance, and it removes more of the persistent part than of the rest. At the first lag the three are 0.800, 0.7773 and 0.7338. The dots are counted from draws and share no arithmetic with the line they sit on; the worst departure is 1.2 standard errors.

A dependence fitted with the line

Every whitening in this collection reads the dependence off a set of residuals, and residuals are not errors. Fitting the two together recovers most of what that costs, and changes almost nothing about the decision it feeds.

together · Dependence Four dependences a single parameter cannot tell apart. Every law here is standardised to a lag-one autocorrelation of 0.8, so a rule told the errors are a first-order autoregression finds the same number in all four and has no way of seeing what separates them. The geometric decay is the world in which estimating a covariance rather than naming it was priced, and found to cost. The five-period moving average has 0.200 at the fourth lag and exactly nothing past it, where the geometric law says 0.328 at the fifth. Long memory at d = 4/9 is still at 0.576 by the twentieth lag, where the geometric law has reached 0.012. The break has no autocorrelation function at all: what is drawn for it is the average over the pairs at each gap, which is what a stationary estimate converges to.

A dependence with a shape

Four ways for errors to repeat, all with the same first lag and nothing else in common. A rule told the errors are a first-order autoregression finds the same number in all four, and is right about one of them.

general · Dependence Where the general fit becomes the parametric one. The band family's objective at the autoregression's own geometric sequence, cut off at each width, on one sample of 60 rows. The horizontal line is the profile likelihood the parametric fit maximises, written independently through a different whitening. At the full width the two are the same number to 3e-14, which is what says the general construction contains the parametric one rather than resembling it. Below 10 lags there is no line at all: the geometric sequence cut off short is not a covariance matrix, so the objective has nothing to evaluate. Between the two the truncation is briefly above the parametric likelihood — a wrong covariance can fit one sample better than the right one, which is the whole reason a width has to be charged for rather than chosen.

A family before a fit

A regression's coefficients and one correlation can be maximised together. Replace the correlation with an estimated covariance and there is nothing left for "jointly" to mean — until a set of covariances is named, and the set turns out not to contain the truth.

family · Dependence The one candidate an effective sample size is right about. n/n_eff with the finite-sample inflation Σ(1 − |k|/n)ρ^|k| is not an approximation to tr(HΩ) for a fit with only an intercept — it is that trace, to machine precision, because the hat matrix of a constant column is 1/n everywhere and its trace against Ω is the mean of Ω. The quoted limit form n(1 − ρ)/(1 + ρ) is not even right about that one. And the average correction the table's fifteen candidates actually need is 3.318 per parameter, well below the scalar, so applying it to all of them over-charges every one.

One number for a table of candidates

An effective sample size is a real quantity, it is exactly right about one thing, and that thing is a mean. Substituted into Akaike's criterion it changes nothing at all, because the penalty it is meant to fix has no sample size in it.

effective · Dependence A window with two wrong ends. The regret of a rule whitened by a Bartlett-tapered Ω̂, as the window widens, against three rules that need no window at all. At L = 0 the estimate is the identity and the rule is exactly least squares — 0.08556, the same number to five places. It falls to 0.01754 at L = 20 and rises again by L = 30, because a quarter of the sample's lags are then being estimated from it. The automatic bandwidth a practitioner would reach for, 4(n/100) to the power 2/9, which at this sample size is 4, gives 0.02963 — 69% above the best available window. The rule told the dependence is an AR(1) sits at 0.01440 throughout, which is the price of not knowing the form.

The window that has to be chosen, and the term that was dropped

An estimated covariance has a bandwidth in it, and both ends of the dial are wrong for different reasons. The rule a practitioner would reach for is two thirds worse than the best window there is.

banded · Order selection Generality in the wrong direction buys nothing. Regret on a sample whose persistence changes from 0.95 to 0.65 at row 60, over 200 draws. The three stationary rules — told one number, told a window, told an order — are within 0.4 standard errors of each other, and all three stop in the same place: they are general in the lag direction, and the departure is in the other one. Letting the model change once, at a point estimated from the same residuals, is worth 0.05021 more at 4.5 paired standard errors — about as much again as the whole of the first repair. Being told where the break is adds 0.01926, and being told the entire covariance adds 0.02465.

Where the generality runs out

A covariance that changes half way through a sample is not one a window can estimate. One number, a window and an order are worth the same as each other on it — and letting the model change once, at a point nobody can locate, is worth as much again as all three.

general · Dependence One likelihood, three answers. The concentrated Gaussian log-likelihood of one sample of 120 rows under AR(1) at 0.8, as a function of the correlation the errors are whitened at. Three rules put three different numbers on this curve. The two-step rule reads the least-squares residuals and lands at 0.7616, giving up 0.304 of log-likelihood. Iterating moves it to 0.8080 and gives up 0.002. The maximum is at 0.8044. The curve is not flat between them: what a fixed point of the residual update finds is a solution of a different equation, and the difference is the Jacobian term ½log(1 − ρ²), which grows as the correlation does.

Iterating is not maximising

Re-reading a correlation from the generalised residuals and refitting converges in seven steps. What it converges to solves the first-order condition of a sum of squares, and the likelihood has one term more than that.

together · Dependence The row count entered twice, and a penalty is one place. Regret against the best available model as the errors are made persistent. Counting rows more than quadruples; both penalty repairs — the trace, and the scalar effective sample size — are worse than it at every persistence measured; and whitening the sample and keeping the ordinary penalty falls, recovering 86.9% of what counting rows gives up at ρ = 0.85. Doing it at an estimated ρ recovers 80.6%, so having to estimate the dependence from the rows being selected on costs 7.3% of what knowing it is worth. Mallows' forms are drawn beside the logarithmic ones and behave the same, which is what rules the linearisation out.

The repair that was exact and made it worse

A penalty computed from the trace is exactly the optimism it estimates, and selecting with it gives up a fifth more than not correcting anything. The row count entered the criterion twice, and a penalty is the second place.

effective · Forecast The window a whitening wants is not the memory of the errors. Regret under a five-period moving average as the tapered estimate is given more lags, over 120 draws at n = 120. The best window is L = 30; the automatic bandwidth is 4 and the error model's own likelihood chooses 9.7 on average. Both land in the same place and both are short, and the reason is the taper: a Bartlett weight at lag k is 1 − k/(L + 1), so a window of 8 keeps 0.556 of whatever the fourth lag carries and a window of 30 keeps 0.871. A window has to be several times the memory before it stops removing the memory. The dashed line is the rule told the errors are a first-order autoregression, which needs no window at all.

The window a whitening wants

Every law here is best whitened by a window several times longer than its own memory, including the one whose memory ends at the fourth lag. The three ways of choosing it from the sample all land in the same place, and it is the wrong one.

general · Order selection Four sequences, and the rule only ever sees the last one. Under long memory at d = 4/9, four things that are all called the dependence. The law itself is the top line. What a sample of 120 rows reports on average is the second, computed exactly: subtracting a sample mean takes the first lag from 0.800 to 0.538. What a candidate's residuals report is the third, lower again at 0.472, because a fit removes dependence along with signal. The autoregressions are fitted to that third sequence and reproduce it exactly out to their own order — the Yule–Walker equations are solved to make it so — so everything they say past that is extrapolation. At the twentieth lag the law has 0.576, the residuals report 0.006, and an AR(8) extrapolates 0.028.

The order the tail is drawn at

A fitted autoregression reproduces the sample exactly at the lags it was fitted on, so everything it says past them is extrapolation — and the order is the dial that decides how much of it there is.

general · Order selection Fitting them together is worth something under one law. Four fits of the same regression under four dependences: least squares, the two-step plug-in every whitened rule in this collection runs, the coefficients and the band maximised together, and a whitening at the law's own covariance that nobody has. Under the moving average — the one law the band family contains — the joint fit beats the two-step by 0.0077 at 3.3 paired standard errors. Under the autoregression, long memory and the break it is a tie: 0.4, 1.0, 0.3 standard errors. That is the same ordering the likelihood gap gave, arrived at through the coefficients rather than through the objective.

What fitting them together buys

Maximising over the coefficients and the covariance together beats the two-step under one of four dependences and ties under the other three. It is the one the band family contains, and the likelihood said so before any coefficient was compared.

family · Dependence What a lag of band costs, as the sample grows. The optimism one more lag of a Bartlett band adds, measured by fitting on one sample and scoring on an independent one, 2,400 pairs at each size, with two standard errors: 0.3745 ± 0.0176 at 120 rows, 0.3842 ± 0.0266 at 240 rows, 0.4523 ± 0.0392 at 480 rows, 0.4022 ± 0.0563 at 960 rows. The window's summed weights give one half a lag and the convention charges one.

The charge at four times the sample

A band of lags is charged a full unit apiece and is worth about three eighths of one at a hundred and twenty rows. The field's last essay said the error in that convention was harmless only because the error surface is flat, and that a steeper surface would give it a cost. A longer sample was the obvious place for one. It is not there: from 120 rows to 480 the error's spread across band widths falls by three quarters, and the derived charge's advantage over the convention falls from 5.5 standard errors to 1.7. What the sample does move is the charge itself, from 0.37 of a unit a lag towards the half a unit the window's weights predict.

dimension · Criterion What the two-regime residue was made of, persistent first. Under a break in the persistence at row 60 of 120 (0.95 then 0.65), the selection regret of six whitenings over 200 draws: restart, unscaled, residual persistences 0.04043; restart, unscaled, persistences given 0.03488; restart, unscaled, fitted with the line 0.03466; the law's recursion, residual persistences 0.03856; the law's recursion, fitted with the line 0.01640; the law's own covariance 0.01579. The whitening that restarts at the break and does not rescale its two regimes gives up 0.01910 against the oracle even when it is given the true persistences; the law's recursion, which divides each row by its regime's innovation scale and continues across the break, is the oracle when given them and within 0.00061 of it when they are fitted with the regression.

The residue was the weights

A regression whose errors change persistence at a known row was whitened as two autoregressions and still gave up 0.0246 against the law's own covariance. That residue was put down to estimating the two persistences from residuals, and the repair named was to fit them with the line. Fitting them with the line moves the persistent regime's estimate from 0.81 to 0.89 and recovers a fifth of the residue — exactly what being told the true persistences recovers. The rest was the transform: it never divided the two regimes by their own innovation scales, which differ sixfold. The law's own recursion, told the persistences, is the oracle to machine precision; fitted with the line, it estimates the persistent one at 0.945 and comes within 0.0006 of the oracle.

general · Dependence Six intervals for the white predictor, a five-period moving average. Under a five-period moving average, over 200 regressions of 120 rows with an eight-lag band, how often each 95% interval for the white predictor's coefficient covers it: two-step band, as reported 95.5%; joint fit, as reported 72.0%; joint fit, White's sandwich 71.0%; joint fit, four-lag sandwich 71.0%; joint fit, band maximised again 85.5%; the law's own covariance 92.0%. The bootstrap redraws each regression 24 times from its fitted coefficients and fitted band, maximising the band afresh on every redraw.

An interval that knows the band was maximised

A regression whose coefficients and eight-lag error band are maximised together prints an interval that covers 72% to 83% where 95% was promised, because it reports nearly the precision of a known covariance and delivers less. Two repairs were named. A sandwich standard error read off the whitened residuals repairs nothing — 71% to 83% — because the maximisation has already made those residuals look independent. A parametric bootstrap that redraws from the fitted band and maximises the band again on every redraw repairs about half the gap, to 85.5% to 89%, by pricing the maximisation through repeating it. What it cannot price is that the band it redraws from is the maximised one.

family · Dependence Past the cap, long memory's surface is flat. The prediction error at every band width from four lags to eighty, above the lowest on the extended list, under long memory at d = 4/9 at three sample sizes, 600 draws each. At 120 rows the floor is at 12 lags and eighty is 0.0039 above it. At 480 the floor is at 30 lags and every width from twenty-four to eighty is within 0.0003 of it. At 960 it is at 48 and thirty to eighty lags are within 0.0000. The dashed line is the list's cap: the surface does not rise behind it, it levels off.

A cap that decides nothing

Every band-width list this field priced stopped at thirty lags, and long memory at a large sample is the one dependence whose best band wants to be wider. At 960 rows it does: the draw's own best width lies past the cap on 48.7% of draws. Lift the cap to eighty and the measured charge follows, from 28 lags to 46 — and the error moves by two ten-thousandths, in the wrong direction and inside its standard error, because the surface behind the cap is flat. The field's last conditional fails where it had its best chance.

dimension · Criterion What iterating the band is worth. How much lower the prediction error is when the Bartlett band is iterated to its fixed point than when it is read once off the least-squares residuals, paired on the draw, with two standard errors, 300 draws at 120 rows for each law and window. First-order autoregression: 4 lags 2.84 ± 0.36, 8 lags 3.43 ± 0.58, 20 lags 4.16 ± 0.80; five-period moving average: 4 lags 2.04 ± 0.28, 8 lags 2.69 ± 0.42, 20 lags 4.32 ± 0.68; long memory: 4 lags 0.83 ± 0.14, 8 lags 1.13 ± 0.25, 20 lags 1.33 ± 0.44 thousandths. Every one is below zero by more than two and a half standard errors; the gain is largest at the widest band under the two short-memory laws.

A fixed point with no likelihood under it

A tapered band of autocovariances has no likelihood to maximise, so the one rule that can fit its dependence together with the regression is to iterate: read the band off the residuals, refit, read it again. It converges on every draw, in seven to eleven fits at the median, gives back about half the memory the first fit took out of the residuals under a short-memory law, and lowers prediction error by two to four thousandths — five to eight paired standard errors. Under the law a first-order likelihood describes, its maximum still wins; under the two it does not, the band's fixed point beats it.

together · Dependence The floor a repair has to choose. Regret of the truncated covariance repaired by flooring its eigenvalues, at a persistence of 0.85 on 120 rows, 120 draws, by floor, on a log scale. Window 4: 1.5649 at 0.001, 0.0945 at 0.01, 0.0385 at 0.05, 0.0285 at 0.1, 0.0226 at 0.2, 0.0235 at 0.4, 0.0292 at 0.7; the taper at the same window 0.0275. Window 8: 1.1919 at 0.001, 0.0980 at 0.01, 0.0334 at 0.05, 0.0258 at 0.1, 0.0197 at 0.2, 0.0209 at 0.4, 0.0249 at 0.7; the taper at the same window 0.0178. Window 12: 0.8692 at 0.001, 0.0621 at 0.01, 0.0239 at 0.05, 0.0209 at 0.1, 0.0212 at 0.2, 0.0239 at 0.4, 0.0282 at 0.7; the taper at the same window 0.0150. Window 20: 1.7110 at 0.001, 0.2495 at 0.01, 0.0399 at 0.05, 0.0238 at 0.1, 0.0202 at 0.2, 0.0219 at 0.4, 0.0259 at 0.7; the taper at the same window 0.0135. Least squares gives up 0.0890 and the whitening told the truth 0.0075.

A covariance repaired

A truncated autocovariance sequence fails to be a covariance matrix on most draws of a persistent series, and this line of essays has always scored it on the draws where it survives. The textbook repair is to floor its eigenvalues. At a floor of 0.001 that moves the matrix by four per cent and gives up 1.56 of regret, seventeen times what least squares gives up, because a whitening reads an inverse and an inverse reads the smallest eigenvalues. At a floor of 0.2 it beats the taper at the shortest windows. So the repair is not a repair: it is a second dial.

banded · Dependence The shape one real modifier leaves on a table of eight characteristics, and one trial's table. Expected interaction z when characteristic 1 modifies the effect at h = 1: 2.80, 2.00, 2.00, 2.00, 0.00, 0.00, 0.00, 0.00 — the real row 2.80, its three nested neighbours 0.713 of that, the four unrelated characteristics nothing. One seeded trial of 400 reads 1.91, 1.90, 2.06, 1.95, 0.14, 0.33, 0.39, 2.08.

The pattern a cause leaves on its neighbours

A characteristic that modifies a treatment's effect raises every characteristic that overlaps it, in proportion to the overlap, and fitting that pattern was supposed to say which one is the cause. Fitted with the noise's own correlation it names the largest row on every trial, because the noise draws the same pattern — so the 54.4% at which the largest row is right among nested characteristics is the most any reading of the table can manage, and the honest answer is a set of about five.

alongside · Repetition Six intervals, one of them honest by right. How often each rule's reported 95% interval covers the true coefficient under a five-period moving average, over 1000 draws of 120 rows, for the most persistent predictor (its own lag-one correlation 0.9) and the white one. The law's own covariance covers 94.9% and 94.1%. Least squares covers 65.0% on the persistent predictor and 94.4% on the white one; a fitted ρ̂ 93.0% and 96.3%; the two-step band 86.5% and 97.3%. The joint fit covers 73.2% and 71.4% — short on both, and short on the white predictor where every other rule is honest — and counting its eight band coefficients gives 76.1% and 73.5%.

The interval the joint fit reports

Maximising a regression's coefficients and an eight-lag covariance band together gives a better estimate than plugging the band in. It also prints an interval that covers 71% to 85% of the time where 95% was promised — on the white predictor too, where every other rule is honest — because it reports almost the precision of the law's own covariance and delivers less than half of that advantage. Counting the eight lags it estimated buys one to two points.

timeseries · Dependence

Named alongside it

The objects these essays reach for when they reach for this one.

Model selectionAutocorrelationCovariance matrixInformation criterionNuisance parameterWhiteningLong memoryRegretTaperingDependenceMonte CarloProfile likelihood

All concepts