Normal Distribution
nothing is called 95% until it has been counted
Field

A search with no fixed point

The search field compares a benchmark with variants that all contain it. Take the fixed point away — let the benchmark be one candidate among sixteen, selected by the same data as its rivals — and the closed form for what every candidate is behind by survives intact, because it was never about nesting but about counting parameters. Three readings of one true null give 2.5%, 8.0% and 76.8%; Bonferroni is three times too strong on one table and not strong enough on another; and searching the benchmark makes the table harder to reject with rather than easier.
One true null, one table, five readings. every subset of four, fifteen models, at a null where nothing any candidate holds is worth anything, over 500 draws. Each bar is the share of draws on which that reading declares a difference at a nominal 5%. The reading is the whole of the difference between the bars: the data is identical. An open search over all 210 ordered pairs rejects 76.2%; the table's own 5% point is 3.163 against the 1.671 a single comparison uses. Bonferroni takes the open reading to 0.6% — and on the nested sequence the same correction does not reach the nominal level at all, because there the excess is a shift in the mean rather than a maximum over many.

When the benchmark is a candidate

A specification search with a benchmark nailed down is the case with a closed form. Take the nail out — let the model that would have been reported be one of sixteen, chosen by the same data as its rivals — and the same true null is read three ways, at 2.0%, 7.8% and 76.2%.

 Every candidate is behind by what its parameter count says. Each dot is one of the fifteen subsets of four predictors, fitted on a rolling window of 80 rows and scored against the benchmark out of sample over 60 origins, at a null where every one of them contains the truth. The line is σ²(q₀/(R − q₀ − 1) − q/(R − q − 1)), which is arithmetic on two integers and a window length. Most of these pairs are not nested — a subset of two predictors and a different subset of two share neither model — and the closed form does not care: the displacement is a statement about how many coefficients each side estimates. The candidates of the benchmark's own dimension sit at zero.

The displacement is a parameter count

A nested variant is behind its benchmark out of sample before anything is searched for. The closed form for how far turns out to have nothing about nesting in it — only two integers and a window length — and it prices a table where no candidate contains any other.

 Which repair goes with which defect. The share of true nulls rejected at a nominal 5% by a reference distribution generated from the fitted benchmark, over 120 draws with 59 resamples each. Where the errors are well behaved every resampling is fine and all four are conservative. Where the variance is a function of the design, the two that detach a residual from its own row reject 5.8% and 8.3% — and the block bootstrap, which is the resampling three earlier fields on this site reach for, repairs nothing at all, because the dependence it is built for is between origins and the rolling scheme reproduces that on its own. Where the errors are skewed the symmetric multiplier is the one that is wrong, and Mammen's two-point version is the only one of the four that is right in both columns.

Residuals that keep their own variance

A reference distribution for a search has to be generated from a fitted model, and the generator draws residuals. Four ways of drawing them keep four different things — and the one this site has reached for three times repairs nothing at all here.

 What the corner costs when the table is full of hopeless candidates. The benchmark holds two predictors, one of which is worth 1; a third predictor, worth the amount on the horizontal axis, is held only by candidates the benchmark does not contain. At the left the null is true and both procedures hold their level. To the right there is a genuinely better candidate, and the uncorrected reality check finds it 2.7% of the time while the same test with the clearly bad columns recentred finds it 51.0% of the time. The columns doing the damage are the ones nobody would have looked at twice: they are so far behind that they cannot win, and calibrating as though they might is what makes the test blind.

The corner the test is calibrated at

"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.

 The displacement per parameter, under five predictor laws. The out-of-sample differential one extra parameter costs a correct candidate, at a window of 80 rows, by two routes that share no arithmetic: counted from rolling forecasts over 1,500 draws of the fifteen-candidate table, and implied by the expected leverage of a fresh row over 80,000 draws of the predictors alone. The Gaussian closed form is 0.01386. normal: 0.01382 counted, 0.01386 from the leverage; Student's t, 8: 0.01379 counted, 0.01407 from the leverage; Student's t, 5: 0.01419 counted, 0.01448 from the leverage; Student's t, 3: 0.01751 counted, 0.01780 from the leverage; exponential: 0.01465 counted, 0.01487 from the leverage.

The displacement on a heavy tail

A correct candidate with one more parameter is behind out of sample by a closed-form amount, and the closed form rests on one Gaussian fact: the expected leverage of a fresh row is q/(R − q − 1). On predictors that are not Gaussian it becomes whatever the leverage is, and two routes that share no arithmetic agree on what that is. With a finite fourth moment it barely moves — 1.5% more per parameter at eight degrees of freedom, 4.5% at five, 7.3% on an exponential. At three degrees of freedom it is 28% more, and the estimate of it will not settle: four batches of twenty thousand draws disagree by more than a quarter.

 The excess over the Gaussian displacement, by window. By how much the expected leverage per parameter exceeds the Gaussian closed form, with the fresh row integrated out, at windows 20, 40, 80, 160, 320, 640. normal: -0.90%, -0.61%, -0.04%, -0.06%, -0.02%, 0.00%. t on eight: 4.14%, 2.78%, 1.79%, 0.73%, 0.28%, 0.18%. t on five: 10.70%, 7.50%, 3.93%, 2.01%, 1.26%, 0.70%. t on three: 44.44%, 32.67%, 22.62%, 16.43%, 11.70%, 7.63%. exponential: 27.91%, 14.01%, 7.21%, 3.53%, 1.93%, 0.85%. On normal predictors the closed form is exact from a window of 80.

An excess and its window

The parameter-count displacement is a Gaussian fact, and on heavy-tailed predictors it runs a few per cent over. Swept across windows from 20 to 640 rows, the excess is about the predictors' excess kurtosis divided by the window: a law with a fourth moment of nine on 80 rows is 7.5% over, and on 640 rows under one. At three degrees of freedom it falls only as the root of the window, from 44% to 7.6%. And the instability once blamed on that law was not there: with the fresh row integrated out, the batches agree to a point and a half, and the mean exists.

All essays