Normal Distribution
nothing is called 95% until it has been counted
Concept

Reality check — where it appears

A test of whether the best of a set of forecasters beats a benchmark, read against a bootstrap distribution of the largest statistic in the set. Its calibration assumes every candidate is exactly as good as the benchmark, and it loses size in proportion to how much of the table is strictly worse.

Named by 5 essays across 2 fields — each of them below, with the objects they name alongside it.

Eight candidates, one of them exactly as good as the benchmark. The candidate set: moving averages of the last 1, 2, 3, 5, 8, 13, 21 and 34 observations, each drawn as its expected squared error divided by the benchmark's — the mean of all 60. The persistence is not chosen, it is solved for: at φ = 0.4895 the best candidate in the set, the average of 2, has exactly the benchmark's expected squared error, and every other candidate is worse by between 0.5% and 5.1%. So the null that no candidate beats the benchmark is true, with one candidate on its boundary. Everything a set comparison claims about its own error rate has to be measured here, because anywhere further inside the null every procedure flatters itself.

Eight forecasters and one benchmark

A set of forecasters is a multiplicity problem on top of a dependence problem, and the two do not separate. Eight windows of one series carry the multiplicity of two and a half independent comparisons; eight separate problems carry eight.

ranking · Multiplicity Which repair goes with which defect. The share of true nulls rejected at a nominal 5% by a reference distribution generated from the fitted benchmark, over 120 draws with 59 resamples each. Where the errors are well behaved every resampling is fine and all four are conservative. Where the variance is a function of the design, the two that detach a residual from its own row reject 5.8% and 8.3% — and the block bootstrap, which is the resampling three earlier fields on this site reach for, repairs nothing at all, because the dependence it is built for is between origins and the rolling scheme reproduces that on its own. Where the errors are skewed the symmetric multiplier is the one that is wrong, and Mammen's two-point version is the only one of the four that is right in both columns.

Residuals that keep their own variance

A reference distribution for a search has to be generated from a fitted model, and the generator draws residuals. Four ways of drawing them keep four different things — and the one this site has reached for three times repairs nothing at all here.

select · Bootstrap Sixteen candidates nobody would have run, and what they cost. At φ = 0.65 one candidate in the original set is genuinely better than the benchmark, and the question is how often each procedure finds it. The added candidates are stale copies of the last value — read two, four, six … steps late — every one of them worse than the benchmark by at least 43%, and not one of them is ever the best candidate in a sample. The reality check goes from 35.5% to 0.0% as they are added, because its reference distribution has to assume every candidate is exactly as good as the benchmark and sixteen such assumptions is a critical value nothing reaches. The recentred version, which drops from the recentring the candidates the data has already ruled out — 15.1 of 24 of them — goes from 25.5% to 24.5%. The top line never moves: reporting the winner's own p-value cannot notice a change to a set it never looks at.

The models that were never in the running

A reference distribution for a set has to assume something about every candidate in it. Assuming that all of them are as good as the benchmark is what makes the reality check honest, and it is what sixteen hopeless candidates use to destroy it.

ranking · Multiplicity What the corner costs when the table is full of hopeless candidates. The benchmark holds two predictors, one of which is worth 1; a third predictor, worth the amount on the horizontal axis, is held only by candidates the benchmark does not contain. At the left the null is true and both procedures hold their level. To the right there is a genuinely better candidate, and the uncorrected reality check finds it 2.7% of the time while the same test with the clearly bad columns recentred finds it 51.0% of the time. The columns doing the damage are the ones nobody would have looked at twice: they are so far behind that they cannot win, and calibrating as though they might is what makes the test blind.

The corner the test is calibrated at

"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.

select · Reference Power as the recentring's threshold moves. The recentred set test's power at a persistence of 0.65, where one forecaster genuinely beats the benchmark, over 300 draws of 100 origins, as its threshold runs from zero to three times Hansen's. With the eight windows alone: 30.3%, 28.3%, 26.0%, 26.0%, 26.0%, 26.0%. With sixteen stale candidates added: 30.3%, 28.3%, 24.3%, 21.0%, 19.3%, 17.7%. The studentised reality check, which sets nothing aside, has 26.0% and 17.7%.

The threshold the recentring reads

A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.

ranking · Multiplicity

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloBenchmark forecastError rateBlock bootstrapComposite nullLeast favourable configurationMultiple comparisonsReference distributionStatistical powerBootstrapData snoopingFamilywise error rate

All concepts