A threshold against its sample

A test comparing a set of forecasters against a benchmark has a composite null: every candidate may be no better, and many of them may be far worse. The models that were never in the running found that stale candidates, far below the benchmark, wreck a reality check’s power by raising its critical value through noise they could not turn into a real lead. Hansen’s repair recentres: a candidate the sample places far enough below the benchmark is set aside at its own negative mean. How far is a threshold, $\sqrt{2\log\log P}$ in units of the candidate’s standard error. That is about 1.75 on a hundred origins.

The threshold the recentring reads swept that number at one sample size, and it found a lopsided dial. At a hundred origins, setting the threshold to zero, so that every candidate the sample places below the benchmark is set aside, kept 30.3% of the power against a better forecaster with or without sixteen stale candidates, where Hansen’s threshold kept 24.3% and 26.0%, and the corresponding excess size at the worst configuration was seven tenths of a point. But Hansen’s threshold is a rate. It is chosen to make the test consistent as $P$ grows, and a threshold of zero has no such guarantee. So a fair objection is that a hundred origins is not where the threshold is supposed to pay. The essay closed on the prediction that its price falls slowly as the sample grows, that the size a threshold of zero gives up should rise towards its asymptotic excess, and that a forecaster with a few years of monthly data never reaches the sample size at which the standard threshold earns its price.

Here are six sizes to settle that, from twenty-five origins to eight hundred.

The prediction is right about the forecaster and wrong about the reason.

A price that does not fall #

A size comparison needs a fixed null and a power comparison a shrinking alternative, or every test reads one at eight hundred origins. So the better forecaster’s advantage shrinks with the root of the sample, keeping the tests on the steep part of their power curves throughout. That means power at different sizes can be compared directly, and the same 600 draws at each size are read at each threshold, so that a gap between thresholds is a paired number.

The figure at the head of this essay is the gap between a threshold of zero and Hansen’s.

Against a stale set the price runs at five to seven points through the whole range, and there is no downward trend once the noise is allowed for: 3.7 at 400 is followed by 6.0 at 800. Without the stale set, the price starts lower and rises, to the same values as the upper row by 200 origins. There is not a size in this range at which the standard threshold has stopped costing power.

That is less surprising than it first seems. Hansen’s threshold grows only as $\sqrt{\log\log P}$: 1.53 at 25 origins, 1.75 at 100, 1.95 at 800. In studentised terms it is barely moving, across a range of samples that is a factor of thirty-two. And the candidates at the boundary of a better forecaster’s alternative stay at the same studentised distance from the threshold at every size, because the alternative is shrinking at the same rate as the standard error. So the comparison does not change with $P$, and there was never much reason to expect that it would. The price is a price for a fixed number of standard errors, and a sample of any size buys the same thing at that price.

The size it was buying #

The guarantee protects against a threshold set too low at the corner, where every candidate is exactly tied with its own benchmark, so the sample’s placing half of them below the line is pure noise, and a reference distribution built on those noisy negative means is too small.

At a threshold of zero the rejection rate runs 17.5%, 6.8%, 5.3%, 4.0%, 4.0% and 5.7%. At Hansen’s, 13.3%, 4.8%, 3.5%, 2.5%, 2.7% and 3.2%. The standard error is 0.9 of a point.

So from fifty origins on, a threshold of zero holds its level within two standard errors, on the configuration most hostile to it. Nothing in that row is rising towards an excess, as the earlier essay’s prediction said it should. If there is an asymptotic excess, it is not visible by eight hundred origins, at this precision. In practice, the thing the threshold’s guarantee protects against is not observed at any size that a comparison of forecasters is likely to have.

Hansen’s threshold, meanwhile, sits below the level throughout, at two and a half to three and a half per cent. So it is not only that it has a price. It is also that it has been holding the test too conservative by about the same margin as Bonferroni over eight, at 2.5% to 3.8%, which it was supposed to improve on. At these sizes the recentring buys no size protection that a threshold of zero does not buy, and it is paying for that protection in power.

What twenty-five origins does #

The left of that figure has a different story, and it is more serious than anything about a threshold. At twenty-five origins, every test on the page rejects at two or three times its level: 17.5% and 13.3% for the two recentred tests, 13.0% for the unstudentised check, and 10.7% for Bonferroni, which is supposed to be conservative. Bonferroni is only as good as the p-values it is fed, and these read a normal tail for a studentised mean of differentials. On twenty-five origins the differentials’ own variance is badly estimated, and all of these tests are reading the same number. It is nothing to do with multiplicity. It is a failure already documented for any studentised statistic built on a short dependent sample, and the 7.3% at the fixed boundary at 25 origins is the same thing in a milder configuration.

So a rule for a forecaster comparison should begin earlier than the threshold: below about fifty origins, none of these numbers mean what they say, and the dial being argued over is much the smaller problem.

The whole of the power #

The gap is the part worth comparing between thresholds, but the levels themselves are also worth a look.

Reading across, a threshold of zero has power of 48.8%, 34.8%, 36.3%, 35.0%, 32.7% and 34.8%. Hansen’s has 41.3%, 27.5%, 29.5%, 29.7%, 29.0% and 28.8%. Setting aside the top of the row at twenty-five origins, where the test is too liberal for its power to mean anything, both lines are flat, which is what a shrinking alternative was built to give. So the curves really are comparing the same difficulty at each size, and the distance between them does not close.

The third line is the informative one. Three times Hansen’s threshold, about 5.3 standard errors at a hundred origins, is close to doing no recentring. It has power of 20.3%, 19.0% and 19.5% from fifty origins to two hundred, a third lower than the other two, and it is the stale candidates that are doing that. At 400 it has risen to 25.8%, and by 800 to 28.3%, meeting Hansen’s. So what this line shows is a test that is not using its threshold at all being rescued by nothing but the sample, as the stale candidates fall out of contention on their own. Eventually any threshold does the job a threshold is for. After that all the thresholds above zero have the same power, 28 or 29%, and a threshold of zero alone stays above them.

The essay on what the other forecast adds makes a separate point worth keeping beside this one. That a set test cannot reject does not mean that nothing in the set is any good, and at thirty-odd per cent power it is going to fail to reject most of the time. The difference between 29% and 35% is real and it is modest. Neither makes this a test that settles anything alone at effects of this size.

What does fall with the sample #

Something does get better as the sample grows, and it is what the threshold was built for in the first place.

The power lost to sixteen stale candidates, at each threshold:

There is the sample size doing what it was supposed to. A stale candidate’s mean does not shrink as the sample grows, while the standard error does, so its studentised distance from the benchmark grows as $\sqrt{P}$, and at some size any fixed threshold rules it out with near certainty. Hansen’s threshold has done that by two hundred. Even a threshold nearly three times the size, at about 5.8 standard errors, has done it by eight hundred.

So there are two things happening, and they had been read as one. The cost of a stale set falls with the sample, for every threshold. That is the threshold’s job and it does it. The cost of the threshold itself does not fall, because that is a cost at the boundary, where nothing has moved. Hansen’s threshold’s defence, that it is a rate, is about the first thing, and against that first thing it is perfectly adequate. The second thing is what a forecaster is paying.

There is also a more direct way to read the upper rows of that table. A threshold of zero has essentially no stale-set cost after fifty origins. It is what the essay on what a set is worth would expect: a sample that places a candidate below the benchmark by any margin has shown it is not a threat to the maximum.

A further thing follows from that table, and it is about what a stale set means. In a real comparison stale candidates are not a nuisance to be tolerated, they are a choice. Somebody decided to include sixteen forecasters that were never going to win. At twenty-five or fifty origins, under any threshold above zero, that decision costs real power, so the honest move at those sizes is to take them out and to say so, since the essays on data snooping have already made the case that a set should be declared in advance. At eight hundred origins it no longer matters: stale candidates cost nothing under any threshold, and there is no need to argue about which ones to drop. The size at which the argument stops mattering is itself a useful number, and on this table it is about two hundred.

Which threshold a forecaster should use #

The earlier essay’s advice was a little hedged. This table sharpens it.

Under fifty origins, the threshold does not matter yet. The test is not valid at any threshold. Reduce the set, shrink the horizon, or say that the comparison cannot support a claim.

From fifty origins up, a threshold of zero holds its level at the corner and keeps four to seven points of power at every size measured. On this evidence it is the better choice, and the only cost is the guarantee at a sample size nobody has.

A forecaster who has to report the standard version should expect a modest loss, and report it. Hansen’s threshold is defensible, widely used and conservative. It is not wrong, it has just been calibrated against a danger that does not materialise in this range.

One caveat matters. This whole sweep is on one construction, of eight moving-average windows with sixteen stale candidates of a stated shape, at one persistence. A distribution drawn from the null makes the general point that a reference distribution is a claim about where the null sits, and the corner here is the hostile configuration among those the construction can reach. It is not necessarily the worst point of the composite null, which is a statement no finite sweep can make. A threshold of zero is exactly where a pathological design would do most harm, and a forecaster with reason to think their candidates are nearly tied should look at the row for the corner and at their own sample size before relying on it.

How the numbers were made #

Each size runs eight rolling mean forecasts on an autoregression, sixty rows of fit and the stated number of origins out of sample, with 199 block-bootstrap resamples per trial and 600 trials per cell. The corner is the field’s configuration with every candidate facing its own independent benchmark. Stale candidates are the earlier essay’s sixteen, each reading a single observation a growing distance back. The better forecaster’s edge is set by the true persistence, moving towards the boundary as $\sqrt{100/P}$ from the 0.65 used at a hundred origins, to about 0.546 at eight hundred.

At this many trials a rejection rate near 5% has a standard error of 0.9 of a point, and a power near 30% of 1.9. Differences between thresholds are paired, on the same draws and the same resamples, which is why a gap of three or four points in the first table can be read when a difference between two sizes of the same magnitude cannot.

Four things must hold. At every size from fifty to eight hundred, a threshold of zero must keep more than two points of power against a stale set. It must stay within three standard errors of the level at the corner at those sizes. A stale set must cost even three times Hansen’s threshold under two points by eight hundred origins, and at a hundred it must cost it over five. And the reading that the standard threshold has stopped costing power by eight hundred origins is refused: the gap there is 6.0 points.

Still open: when a corner is not the worst case, and how far below zero #

Going below zero. The dial has been read from zero upward, and nothing prevents reading it further: a negative threshold sets aside candidates that lead the benchmark a little, which is clearly unsafe in principle and, on the size numbers here, might not be at modest values. The measurement that would find its edge is an extension of this one, and it is the honest test of whether zero is a natural stopping point or merely the end of the sweep that was tried.

A worse corner. The candidates here are moving-average windows, which are strongly correlated with one another. A corner of nearly independent candidates, where the maximum of many tied statistics has the most room to fall below what a recentred reference expects, might produce the excess that this sweep does not find. The corner can be built from any set, and one built to be least favourable would tell a forecaster more than one built for convenience.

What a rate buys and what it does not #

This essay takes the recentred set test’s threshold across samples of 25 to 800 origins. Its claims are that the power a threshold of zero keeps over Hansen’s runs between 3.7 and 7.5 points against a stale set with no downward trend; that its size at the corner is 6.8%, 5.3%, 4.0%, 4.0% and 5.7% from fifty origins up, at a standard error of 0.9; that Hansen’s threshold sits at 2.5% to 4.8% over the same range; that the cost of sixteen stale candidates falls to nothing by two hundred origins under Hansen’s threshold and by eight hundred under a threshold three times the size; and that at twenty-five origins every test here rejects at more than twice its level, for reasons that have nothing to do with a threshold.

What stays out, as a decision: other candidate sets and other persistences; negative thresholds, named above; and any claim about the far limit, since a threshold of zero’s asymptotic excess may exist and is not visible at eight hundred origins. The boundary against the threshold the recentring reads is that it found the dial at one size, and this one checks whether the sample moves it, and finds that the sample moves something else.