One test for eight nulls

When every null is true tested a combination of eight forecasts against each of its parts. At the optimal weights, the first-order condition makes every one of the eight encompassing nulls exactly true at once — not approximately, by an identity — so the configuration a reality check has to assume is the one the arithmetic produces. On that configuration it measured the readings of the largest of the eight comparisons: the winner’s own p-value rejects several times too often, Bonferroni over eight holds its level, and a set-level bootstrap holds it too.

It named one reading it did not run. A combination’s claim to encompass its parts is one claim about a vector, and the natural test of a claim about a vector is a joint one: a Wald test that every encompassing mean is zero together, one comparison rather than eight. It is a different object from the largest of eight, it asks a different question of the same numbers, and the essay declined it as a decision rather than an oversight.

This essay runs it. The joint test turns out to have one degree of freedom fewer than it looks, to be badly oversized at the sample sizes a forecaster has, and — given the reference distribution it ought to have — to be less powerful than the crude correction it was supposed to improve on.

Eight nulls, seven degrees of freedom #

The eight statistics are the means of $c_{j,t} = e_t(e_t - e_{j,t})$ over the origins, where $e_t$ is the combination’s forecast error and $e_{j,t}$ is forecast $j$’s. A combination is a weighted sum with weights summing to one, so $e_t = \sum_j w_j e_{j,t}$, and

$$\sum_j w_j c_{j,t} = e_t\Big(e_t - \sum_j w_j e_{j,t}\Big) = 0$$

on every origin. That is not a property of the optimal weights or of the population; it is arithmetic, and it holds for any combination on any sample. The eight columns are linearly dependent, their covariance has rank seven, and a Wald statistic built on all eight would be inverting a singular matrix.

So the test drops one column — the one with the largest weight, which the others determine — and reads the remaining seven: $W = P\,\bar c^\top \hat S^{-1} \bar c$, with $\hat S$ the Bartlett long-run covariance at the same lag the single-column statistics use. Against the best single forecast the dropped column is not a choice: with all the weight on one forecast its own column is $e_t(e_t - e_t)$, identically zero, and any other choice of column to drop leaves a singular matrix.

The weight that is a vector found the optimal combination of these eight moving averages putting no weight on the forecast with the smallest error. The identity is the other half of that picture: the vector has eight entries and one constraint, and every statement about it — including a test — lives in seven dimensions.

How often it rejects a true null #

On sixty origins the joint test against a chi-square on seven degrees of freedom rejects 26.6% of true nulls — more than five times its level. Hotelling’s finite-sample correction, which rescales the statistic and reads it against an F on seven and fifty-three, brings that to 15.1%: three times its level. The largest of the eight read as a single comparison rejects 25.0%, which is the earlier essay’s finding, and Bonferroni over the eight rejects 5.2%.

So the joint test is no better behaved than the naive reading of the maximum, and the corrected joint test is worse than the crude correction of the maximum by a factor of three. As the origins grow it improves — 16.4%, 9.8% and 8.6% at 120, 240 and 480 origins against a chi-square, 12.4%, 8.3% and 8.0% with Hotelling’s — and at four hundred and eighty origins, eight years of weekly forecasts, it is still rejecting at more than one and a half times its level. Bonferroni sits between 4.4% and 5.2% throughout.

The cause is the denominator. A Wald test is only as good as its covariance estimate, and this one is a seven-by-seven long-run covariance estimated from sixty dependent rows: twenty-eight distinct entries, each a weighted sum of autocovariances, inverted. Its smallest eigenvalues are underestimated, and the inverse amplifies exactly the directions where they are. The mean of $W$ across draws where every null is true is 11.0 on sixty origins, against the chi-square’s seven; the statistic’s real 95th percentile is 23.8, against the 14.1 the test reads it against.

Hotelling’s correction is the exact answer for a covariance estimated from independent rows and it does what it can. These rows are not independent — consecutive origins share most of their estimation window and their errors are serially correlated by construction — and that part of the distortion it cannot see.

A test handed its own critical value #

A size distortion has a repair if it can be measured, and here it can: the statistic’s own distribution where every null is true is the distribution of $W$ across the draws at the optimal combination, and its 95th percentile is the critical value a test that knew its reference distribution would use. Read against that, the joint test rejects exactly 5% of true nulls, by construction. The question then is the one the earlier essay asked of every reading of the maximum: when the combination is wrong, how often does it notice?

The two wrong combinations are the two the set field used: equal weights, which forecast 9.8% worse than the optimum in squared error, and the best single forecast, 29.9% worse. On sixty origins the joint test rejects equal weights 12.0% of the time and Bonferroni 18.1%; it rejects the best single forecast 46.6% of the time and Bonferroni 95.6%.

So with its size repaired the joint test is the weaker of the two against both alternatives, and against the larger one by a factor of two. That is the opposite of what a joint test is usually taken to buy, and it has a reason.

Where the alternatives point #

A Wald statistic whitens the vector of means by their covariance before it adds them up, so a shift along a direction in which the statistics share their noise is discounted, and a shift across directions in which they do not is counted in full. That is the right weighting when the covariance describes the null and the alternative alike. Here it does not.

At the optimal combination the seven columns are weakly related — their average pairwise correlation over the origins is 0.20 — which is the earlier essay’s observation that the eight statistics are nearly independent. Against the best single forecast they are not. Every column is then $e_t(e_t - e_{j,t})$ with the same $e_t$, the single forecast’s own error, multiplying it, and the average pairwise correlation rises to 0.54. The departure is in every column at once — the mean t-statistics against the other seven forecasts run from 1.1 to 3.6, six of them above two — and it is in the direction they share, because it is the one forecast’s own error, large on the same origins for every comparison, that both makes them all positive and makes them move together.

So the joint test sees a large, consistent departure along the one direction its covariance says is noisiest, and discounts it. The maximum reads each column on its own scale and finds six of them past two. Bonferroni’s eightfold penalty costs the maximum less than one standard error at these levels; the joint test’s whitening costs it most of the signal.

Equal weights are the other kind of alternative — each forecast’s weight is wrong a little, in both directions, and the mean t-statistics run from −1.4 to 1.8 with mixed signs — and there the gap is narrower: 12.0% against 18.1% at sixty origins, 34.6% against 47.2% at a hundred and twenty, 84.4% against 89.3% at two hundred and forty. Even on the alternative with no shared direction to discount, the joint test trails, because on sixty origins its covariance is still being inverted from too little data.

What the joint test is the right answer to #

None of this makes the joint test wrong. It answers a precise question — is the combination’s error orthogonal to every one of its parts at once — and for that question it is the efficient test when the covariance is known and the departures are spread. Neither condition holds here. The covariance is estimated from sixty dependent rows, and the departures that matter in a set of forecasts are concentrated in whichever forecasts the combination underweights.

The comparison with eight forecasters and one benchmark is instructive. There, the statistics being maximised over are nearly identical — eight forecasters each compared with one benchmark, sharing most of their noise — and Bonferroni is badly conservative because it treats eight nearly identical tests as eight independent ones. Here the eight statistics are nearly independent apart from the one identity, Bonferroni is close to exact, and the joint test’s ability to use their correlation buys nothing because there is almost no correlation to use and one exact dependence it has to remove by hand.

So the case for a joint test is strongest exactly where the case for Bonferroni is weakest, and in this set it is the other way round.

Reading the two together #

The joint test and the maximum are not rivals so much as two summaries of one vector, and a forecaster who computes both learns something neither says alone. On a combination where the maximum is large and $W$ is unremarkable, the failure is concentrated in one or two parts — a forecast the combination should lean on more. On one where $W$ is large and the maximum is not, the failure is spread — weights wrong a little everywhere, the equal-weights case. The power figures above say which of those two shapes a test of this size can detect on sixty origins: the first, by the maximum, almost always; the second, by either test, rarely.

A null with a model in it found the same thing one level down: a reference distribution is worth exactly as much as its fit to the configuration under test, and the configuration here — every null true, by construction — is one where the maximum’s reference distribution is simple and the joint test’s depends on a matrix nobody can estimate well.

The seventh degree of freedom is not a technicality #

The identity that removes one degree of freedom is easy to miss, and missing it is not a small error. A joint test written naively on all eight columns inverts a covariance that is singular in every sample, exactly — the identity holds origin by origin, so it holds for the sample covariance too — and the computation either fails or returns a number made of rounding error. A practitioner who reaches for a generalised inverse to make it go through gets the right statistic, because the vector of means lies in the seven dimensions the inverse keeps; but if they then read it against a chi-square on eight, as eight nulls suggest, the test is conservative by the difference between the two distributions’ tails — a 95th percentile of 15.5 where the right one is 14.1.

The same arithmetic constrains every set procedure built on these columns. The set-level bootstrap the earlier essay measured takes the maximum of eight statistics that are really seven, and is right not to care, because a maximum never inverts a covariance. The search over a table of nested models has no such identity — its comparisons are against one benchmark, not against the parts of a combination — and a joint test there would have its full count. A procedure that standardises these columns jointly — studentising the vector rather than each statistic — inherits the singularity and has to remove it by hand, exactly as the joint test does here. The rule is general: a test of a combination against its parts is a test in one dimension fewer than there are parts, and only the procedures that never invert a covariance are allowed to forget it.

What it costs to repair the size in practice #

The repair used above — the statistic’s own 95th percentile where every null is true — is not available to a forecaster, who has one sample and no draws at the optimum. Two practical versions exist and neither is measured here. A bootstrap of the origins in blocks reproduces the dependence that defeats Hotelling’s correction, at the cost of a seven-by-seven inverse on every resample. And a shrunken covariance — the diagonal plus a fraction of the off-diagonal — trades the inverse’s instability for a bias towards treating the seven means as independent, which on this set is nearly true.

On this evidence the second would bring the joint test close to a sum of seven squared t-statistics, and its power against the best single forecast would still be bounded by the shared-direction argument above, since shrinking the covariance towards its diagonal throws away exactly the correlation that hides the departure — and with it the protection the correlation gives under the null. So the size can be repaired and the power cannot be, by any construction that remains a quadratic form.

Twelve hundred draws on the set field’s eight moving averages #

Twelve hundred draws at each number of origins, on the configuration the set field used: eight moving-average forecasts of a first-order autoregression at the persistence where the equal and optimal combinations first diverge, a rolling estimation window of sixty rows, and one-step forecasts. Every reading — the joint test under both references, the largest of the eight read as one comparison and with Bonferroni, and one comparison stated in advance — is computed on the same draws, so a difference between them is a difference in how one vector is read. The single-column statistics are the Diebold–Mariano form with Harvey, Leybourne and Newbold’s correction, as in the earlier essay; the joint covariance uses the same Bartlett weights and lag.

The comparison stated in advance rejects equal weights 58.5% of the time on sixty origins, more than either test — because the forecast it was stated against, the shortest average, happens to be the one equal weights fail against most, with a mean t-statistic of 1.8. That is the familiar fact that a comparison chosen well in advance beats any correction for not choosing, and it is a statement about the luck of the choice rather than about the test.

Seven degrees of freedom, a critical value of 23.8, and the power after it #

The claims are that the eight encompassing statistics satisfy an exact linear identity on every origin, so the joint test has seven degrees of freedom; that it rejects 26.6% of true nulls on sixty origins against a chi-square and 15.1% with Hotelling’s correction, and still 8.6% and 8.0% on four hundred and eighty; that its own 95th percentile on sixty origins is 23.8 where the chi-square’s is 14.1; and that read against that true critical value it rejects equal weights and the best single forecast less often than Bonferroni over the maximum, at sixty, a hundred and twenty and two hundred and forty origins.

Not claimed: anything about a joint test with a bootstrapped or shrunken covariance, which is named above as the repair and not built; anything about combinations whose weights were estimated, where the weights’ own noise is inside the statistic; and anything about sets whose forecasts are more alike than eight moving averages of different lengths.

Still open: the joint test with a covariance the sample can afford #

A shrunken covariance is the obvious repair to the size, and it has a parameter — how far to shrink — that is itself a choice the sample must make. At full shrinkage the joint test becomes a sum of squared single-column statistics, which ignores the identity and so has to be read on eight degrees of freedom with one of them spurious; at none it is the test measured here. Whether some intermediate shrinkage holds the level on sixty origins without giving up the joint test’s advantage on spread departures — and whether there is any such advantage left once the level is held — is the measurement that would say whether a test of the whole weight vector is ever worth running on a forecaster’s sample.