Normal Distribution
nothing is called 95% until it has been counted
Concept

Forecast combination — where it appears

Averaging several forecasts into one, whose error variance is a quadratic form in the weights and is usually below every one of its parts. The weights have to be estimated, and estimating them on the same errors they combine is what turns a theoretical gain into a sometimes-negative one.

Named by 4 essays across 2 fields — each of them below, with the objects they name alongside it.

Also named here as forecast encompassing — the same set of essays touches all of them, so they are one junction rather than several.

Three quantities, and only one of them crosses zero. Two forecasts of an AR(1) — the last value carried forward and the mean of the last 60 observations — at 1 step ahead. The curve through zero is σ₁² − σ₂², the difference in expected squared error that a comparison of accuracy tests; it changes sign at φ = 0.4922. The two curves above it are σ₁² − σ₁₂ and σ₂² − σ₁₂, the quantities the two encompassing tests are about, and neither of them comes near zero anywhere: the smallest value either takes across the range is 0.008 times the variance of the series. All three are closed forms in φ, R and h with no simulation in them. Equal accuracy is one hypothesis about this picture and encompassing is another, and a set of numbers can satisfy either without the other.

What the other forecast adds

Two forecasters, one series, and two different questions about them. Which is more accurate has an answer that changes with the persistence of the series; whether either is redundant has an answer that never changes at all.

ranking · Forecast The ranking on the left, the weights on the right. Eight moving-average forecasts of an AR(1) at φ = 0.4895, the persistence at which the best of them exactly ties the 60-observation benchmark. On the left, each candidate's expected squared error in units of the series' own variance: the smallest belongs to L = 2, at 1.0156. On the right, the weight each carries in the variance-minimising combination of all eight — and the best of them carries 0.00000. The two ends of the family carry 1.0172 of the weight between them, and the combination they make is worth 0.7817, which is 23.0% below the best single forecast. Both columns are closed forms in φ. Which forecast to keep and which forecasts to use are different questions, and this is a set where the answers share nothing.

The weight that is a vector

Two forecasts have a best combination and one number describes it. Eight have a best combination too, and the vector describing it puts nothing at all on the forecast with the smallest mean squared error.

search · Rank Eight encompassing nulls, all of them true at once. The forecast under test is the variance-minimising combination of the eight candidates, and the first-order condition that defines its weights is cov(e_c, e_j) = var(e_c) for every j — so every one of the eight nulls is exactly true simultaneously and every rejection counted here is false. 800 draws of 60 origins. The largest of the eight statistics rejects 25.0% of the time at a nominal 5% and one comparison stated in advance rejects 3.1%, which makes the set worth about 9.1 independent comparisons — nearly the eight it has. Bonferroni, which is far inside its level on a search over accuracy comparisons of the same eight forecasters, is at 4.8% here.

When every null is true

A reality check assumes that every candidate in the set is exactly as good as the benchmark, which is a configuration nobody's data is ever in. Test a combination against its own parts and that configuration is not assumed — it is what the arithmetic makes true.

search · Multiplicity How often each reading rejects when every null is true. At the optimal combination of eight forecasts, where all eight encompassing nulls are exactly true, the share of 1,200 draws on which each reading rejects at 5%, at 60, 120, 240, 480 origins. The joint Wald test against a chi-square on seven degrees of freedom: 26.6%, 16.4%, 9.8%, 8.6%; with Hotelling's finite-sample F: 15.1%, 12.4%, 8.3%, 8.0%; the largest of the eight read as a single comparison: 25.0%, 25.9%, 24.7%, 23.4%; the largest with Bonferroni over eight: 5.2%, 4.9%, 4.7%, 4.4%.

One test for eight nulls

A combination of eight forecasts can be tested against its parts one comparison at a time or all at once. All at once is a Wald test on the vector of eight encompassing means — which has seven degrees of freedom, not eight, because the means sum to zero on every origin when weighted by the combination. On sixty origins it rejects a true null 26.6% of the time, Hotelling's correction brings that to 15.1%, and handed its own true critical value it is less powerful than Bonferroni over the largest of the eight against both wrong combinations measured: 12% against 18%, and 47% against 96%.

search · Multiplicity

Named alongside it

The objects these essays reach for when they reach for this one.

Forecast encompassingBenchmark forecastCombination weightMean squared errorAutocorrelationBonferroniError rateLong-run varianceMonte CarloNull hypothesisReference distributionBlock bootstrap

All concepts