Field

The best of a set, and what the search costs

Comparing two forecasters is a test. Comparing eight is a multiplicity problem on top of a dependence problem, and the two do not separate: eight windows of one series carry the multiplicity of about two independent comparisons and eight separate problems carry eight, so a correction that charges for the number of models is wrong in both directions. The reference distribution has to be over the whole set — and once it is, the models nobody would have run turn out to cost more than the ones that were close.

Three quantities, and only one of them crosses zero. Two forecasts of an AR(1) — the last value carried forward and the mean of the last 60 observations — at 1 step ahead. The curve through zero is σ₁² − σ₂², the difference in expected squared error that a comparison of accuracy tests; it changes sign at φ = 0.4922. The two curves above it are σ₁² − σ₁₂ and σ₂² − σ₁₂, the quantities the two encompassing tests are about, and neither of them comes near zero anywhere: the smallest value either takes across the range is 0.008 times the variance of the series. All three are closed forms in φ, R and h with no simulation in them. Equal accuracy is one hypothesis about this picture and encompassing is another, and a set of numbers can satisfy either without the other.

What the other forecast adds

Two forecasters, one series, and two different questions about them. Which is more accurate has an answer that changes with the persistence of the series; whether either is redundant has an answer that never changes at all.

Eight candidates, one of them exactly as good as the benchmark. The candidate set: moving averages of the last 1, 2, 3, 5, 8, 13, 21 and 34 observations, each drawn as its expected squared error divided by the benchmark's — the mean of all 60. The persistence is not chosen, it is solved for: at φ = 0.4895 the best candidate in the set, the average of 2, has exactly the benchmark's expected squared error, and every other candidate is worse by between 0.5% and 5.1%. So the null that no candidate beats the benchmark is true, with one candidate on its boundary. Everything a set comparison claims about its own error rate has to be measured here, because anywhere further inside the null every procedure flatters itself.

Eight forecasters and one benchmark

A set of forecasters is a multiplicity problem on top of a dependence problem, and the two do not separate. Eight windows of one series carry the multiplicity of two and a half independent comparisons; eight separate problems carry eight.

Sixteen candidates nobody would have run, and what they cost. At φ = 0.65 one candidate in the original set is genuinely better than the benchmark, and the question is how often each procedure finds it. The added candidates are stale copies of the last value — read two, four, six … steps late — every one of them worse than the benchmark by at least 43%, and not one of them is ever the best candidate in a sample. The reality check goes from 35.5% to 0.0% as they are added, because its reference distribution has to assume every candidate is exactly as good as the benchmark and sixteen such assumptions is a critical value nothing reaches. The recentred version, which drops from the recentring the candidates the data has already ruled out — 15.1 of 24 of them — goes from 25.5% to 24.5%. The top line never moves: reporting the winner's own p-value cannot notice a change to a set it never looks at.

The models that were never in the running

A reference distribution for a set has to assume something about every candidate in it. Assuming that all of them are as good as the benchmark is what makes the reality check honest, and it is what sixteen hopeless candidates use to destroy it.

A distribution drawn from the null

Between nested models the ordinary comparison statistic has a null distribution centred at minus one and a 95% point of a quarter. A correction to its mean repairs the centre and leaves the shape; simulating the null repairs both.

Power as the recentring's threshold moves. The recentred set test's power at a persistence of 0.65, where one forecaster genuinely beats the benchmark, over 300 draws of 100 origins, as its threshold runs from zero to three times Hansen's. With the eight windows alone: 30.3%, 28.3%, 26.0%, 26.0%, 26.0%, 26.0%. With sixteen stale candidates added: 30.3%, 28.3%, 24.3%, 21.0%, 19.3%, 17.7%. The studentised reality check, which sets nothing aside, has 26.0% and 17.7%.

The threshold the recentring reads

A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.

What Hansen's threshold costs, by the number of origins. The power a threshold of zero keeps over Hansen's, on the same draws. with sixteen stale candidates: 7.5%, 7.3%, 6.8%, 5.3%, 3.7%, 6.0%. the eight windows alone: 2.7%, 3.8%, 4.7%, 5.3%, 3.7%, 6.0%, at 25, 50, 100, 200, 400, 800 origins. The price does not fall as the sample grows.

A threshold against its sample

A recentred test's threshold exists so that the test is right in the limit, and at a hundred origins it costs a fifth of the power against a stale set. The natural defence is that the price falls as the sample grows and the guarantee starts to earn it. Across twenty-five to eight hundred origins it does not: a threshold of zero keeps four to seven points more power at every size, and its size at the corner stays at the level from fifty origins on. What does change with the sample is the stale set, which stops costing anything by two hundred origins under Hansen's rule and by eight hundred under the most cautious.

All essays