The threshold the recentring reads

The models that were never in the running measured what a set of forecasters does to a test of whether any of them beats a benchmark. The reality check builds its reference distribution as if every candidate were exactly as good as the benchmark, which keeps it honest and makes it hostage to the set: add sixteen stale copies of the shortest window, each between 43% and 97% worse than the benchmark, and its power against a genuinely better forecaster falls from 34.3% to 0.0%. The recentred test — Hansen’s — studentises each candidate and keeps the ones the sample has placed far below the benchmark at their own means, so they cannot inflate the reference distribution’s maximum, and its power went from 26.0% to 24.3%.

How far below is a threshold. Hansen’s is $-\sqrt{2\log\log P}$ in units of the candidate’s own standard error, which on a hundred origins is 1.75. The essay noted that the threshold is a rate rather than a tuning constant, treated it as given, and left its choice out as a decision. This essay sweeps it, from zero — set aside every candidate the sample places below the benchmark at all — to three times Hansen’s value, where almost nothing is set aside and the test is the studentised reality check.

What the threshold does #

The recentred test’s statistic is the largest studentised candidate mean, floored at zero. Its reference distribution is the same maximum over block-bootstrap resamples, each candidate’s resampled mean centred at its own sample mean — so that the resample describes the null — and then shifted back down by that sample mean for the candidates the threshold sets aside. A candidate set aside contributes its resampled noise around a negative mean; a candidate not set aside contributes its resampled noise around zero.

So the threshold decides which candidates are treated as possibly on the boundary of the null and which as certainly inside it. A candidate treated as on the boundary can produce a large maximum in the reference distribution and raise the critical value; a candidate treated as inside cannot. The reality check treats every candidate as on the boundary, which is safe and is why stale candidates destroy it.

Setting aside too few candidates makes the test conservative and costs power. Setting aside too many — candidates whose true mean is zero but whose sample happened to fall below — makes the reference distribution too small and the test liberal. A threshold shrinking slowly with the number of origins is what makes the test valid asymptotically, and the question here is what it costs at a hundred origins.

Power, with and without the stale candidates #

The configuration is the earlier essay’s: eight moving averages of a first-order autoregression against a long-window benchmark, a hundred origins, at a persistence of 0.65 where one window genuinely beats the benchmark. Three hundred draws and 199 resamples, on that essay’s seed, so the row at Hansen’s threshold is its number.

With the eight windows alone, the recentred test’s power is 30.3% at a threshold of zero, 28.3% at half of Hansen’s, and 26.0% from Hansen’s threshold upwards. With sixteen stale candidates added it is 30.3% at zero, 28.3% at half, 24.3% at Hansen’s, 21.0% at one and a half times, 19.3% at twice, and 17.7% at three times — which is the studentised reality check’s own 17.7%, because at that threshold almost nothing is set aside.

Two things stand out. At a threshold of zero the stale candidates cost nothing at all: 30.3% with them and without them. And even with the eight windows alone, a threshold of zero gains four points over Hansen’s, because some of the windows are themselves clearly worse than the benchmark and the standard threshold keeps them in the reference distribution as though they might not be.

So the dial has a direction in which power rises steadily, and the standard setting sits partway along it. Whether that direction is free depends on what it does to the size.

Size, where it could go wrong #

At the boundary the earlier essay measured — the persistence at which the best window exactly ties the benchmark and the rest are worse — the recentred test rejects 1.0% of the time at every threshold with the eight windows alone, and between 0.3% and 1.0% with the stale candidates added. It is conservative throughout, and a threshold of zero changes nothing.

That configuration is a gentle one for a liberal threshold, because only one candidate is on the boundary and every other one is genuinely inside the null, where setting it aside is correct. The configuration that punishes a threshold is the corner: every candidate exactly on the boundary, each against its own benchmark, so that every candidate the threshold sets aside is set aside by noise. That is the setting eight forecasters and one benchmark built to give Bonferroni its best case, with the columns independent by construction.

At the corner a threshold of zero sets aside 4.4 of the eight candidates on average — half of them, as a sample mean of a candidate at zero is below zero half the time — and the test turns down the null 3.0% of the time. From half of Hansen’s threshold upwards it rejects 2.3%. The liberal end of the dial costs seven tenths of a point of size in the one configuration designed to make it pay, and leaves the test below its level.

Why setting aside half the boundary costs so little #

That is the result that needs an explanation, because the textbook objection to a threshold of zero is exactly the corner: recentre the candidates that happened to fall below zero, and the reference distribution is built on the luckier half, so its maximum is too small.

Two things blunt it. The test’s statistic is the largest studentised candidate mean, and the candidates the threshold sets aside are the ones with negative sample means — the ones least likely to be the observed maximum. The reference distribution needs to reproduce how large the largest of the other candidates’ noise can be, and the candidates set aside were never going to supply it. And the floor at zero means the test can only reject when some candidate’s sample mean is positive, which at the corner happens on most draws but is driven by the candidates the threshold did not touch.

What is left is the residual case in which a set-aside candidate’s resample would have been the reference distribution’s maximum, and at a hundred origins with eight nearly independent candidates that case moves the size by under a point.

How many each threshold sets aside #

With the eight windows alone, a threshold of zero sets aside 4.8 of them at the null and Hansen’s 0.7. With sixteen stale candidates added, zero sets aside 20.8 of twenty-four, Hansen’s 16.4, and three times Hansen’s 0.8. The stale candidates are between 43% and 97% worse than the benchmark, so a threshold that sets aside sixteen of twenty-four is mostly setting aside exactly the sixteen nobody should have run — and the one that sets aside 0.8 is building its reference distribution as though all of them might win.

The earlier essay read the recentring as a classifier, sorting candidates into the boundary and the interior, and found Hansen’s threshold setting aside 98.1% of the stale candidates at the null and 8.8% of the genuine ones. The sweep says what moving that classification is worth where a better forecaster exists: at Hansen’s threshold, 15.1 of the twenty-four are set aside where a better forecaster exists, and power is 24.3%; at zero, 18.4, and power is 30.3%.

Two dials, not one #

The recentred test differs from the reality check in two ways at once, and the earlier essay separated them: it studentises each candidate, and it sets some aside. The sweep holds the first fixed and moves the second, and its top end is the studentised reality check — studentised, nothing set aside — so the two dials can be read off one figure.

With the eight windows alone, the unstudentised reality check finds the better forecaster 34.3% of the time and the studentised one 26.0%: studentising costs eight points, because the windows differ in how variable their loss differentials are, and dividing by a candidate’s own standard error promotes the steadier ones at the expense of the one that is genuinely better and noisier. What studentising costs found the same price in an interval. Setting candidates aside then recovers four of the eight points at a threshold of zero and none at Hansen’s.

With the sixteen stale candidates, the order of the two dials reverses in importance. The unstudentised reality check falls to 0.0%, the studentised one to 17.7%, and setting candidates aside takes it back to 30.3%. Studentising is worth something here only because it puts the stale candidates’ enormous variances on the same scale as everyone else’s; setting them aside is worth more, because it removes them.

So a forecaster choosing a set test is choosing two things, and the evidence on each is different. Studentising helps when the set is polluted and hurts when it is clean. Setting aside helps in both, and helps most at the low end of the threshold.

The corner, read by every test #

The corner is worth reading for the other tests as well, because it is the configuration their names were designed for. With every window tied with its own benchmark and the columns independent, the winner’s own p-value rejects 24.7% of the time — the price of looking at eight and reporting one — Bonferroni over eight rejects 2.7%, and the unstudentised reality check 6.3%: above its level, by more than a standard error. The studentised version rejects 2.3%, and the recentred one 2.3% to 3.0% depending on the threshold.

That ordering is the one eight forecasters and one benchmark found for independent columns, and it is a reminder that the configuration which tests a threshold’s liberality is also the one in which the oldest correction is close to right. A threshold of zero at the corner rejects three tenths of a point more often than Bonferroni and seven tenths more than the studentised check, which is to say it sits among the conservative tests rather than the liberal ones — closer to the correction everybody already trusts than to the reading nobody should.

A threshold is a rate, and a hundred origins is a sample #

None of this says Hansen’s threshold is wrong. It is the choice that makes the test consistent: as the origins grow, a candidate on the boundary has its studentised mean near zero, the threshold $\sqrt{2\log\log P}$ grows without limit, and the probability of setting a boundary candidate aside goes to zero, so the reference distribution is asymptotically right. A threshold of zero has no such guarantee. At the corner it sets aside half the boundary at every sample size, and its size there tends to something above the nominal level.

What the measurement says is that the asymptotic guarantee is bought with power at the sample sizes forecasters have, and that the price is not small: a fifth of the test’s power against a better forecaster when the set contains stale candidates, a seventh when it does not. And the size the guarantee protects against, at a hundred origins, is seven tenths of a point at the worst configuration measured.

A distribution drawn from the null made the general point that a reference distribution is a claim about where the null is, and that a claim made too cautiously costs power in the same way a claim made too boldly costs size. The threshold is the dial on which those two costs trade, and at this sample size the trade is lopsided.

What a forecaster comparing a set should do with it #

Three readings, in order of how far the evidence supports them.

The stale candidates are a reason to set aside more, not fewer. The threshold that protects a test from candidates nobody should have run is a low one. At zero the test is immune to them here; at Hansen’s sixteen of them cost it about a fifteenth of its power, 26.0% to 24.3%; at three times Hansen’s they cost it nearly a third, 26.0% to 17.7%.

A threshold below Hansen’s costs under a point of size at a hundred origins. That is a statement about this set — eight windows and up to sixteen stale copies, nearly independent at the corner — and about a hundred origins. Under strong dependence between candidates, where several near the boundary move together, setting aside the luckier half might cost more, and nothing here measures it.

The threshold is the dial that matters most. On a clean set the choice of threshold moves power by four points and on a polluted one by thirteen, so of the two dials it is the one whose setting a reader most needs to know.

Report the threshold. A recentred test’s p-value depends on a choice the procedure’s name does not state, by as much as the difference between finding a better forecaster one time in three and one time in six. The rule is part of the data in a trial; here the threshold is part of the test, and a p-value quoted without it has the same defect as one quoted without its reference distribution.

Three hundred draws at each threshold, boundary and corner #

Three hundred draws at each threshold and configuration, 199 block-bootstrap resamples per draw with blocks of four, and a hundred origins with a rolling window of sixty rows. The power draws are the earlier essay’s at a persistence of 0.65, and the boundary draws are at the persistence where the best window exactly ties the benchmark, about 0.49. The corner draws one forecasting problem per window, each at the persistence where that window ties its own benchmark, so that the eight columns are independent and every one is on the boundary. The thresholds are 0, 0.5, 1, 1.5, 2 and 3 times $\sqrt{2\log\log P}$; at three times the threshold is 5.2 standard errors, past which almost nothing falls.

A rejection rate on three hundred draws has a standard error of about 1.3 points near 5% and about 0.6 near 1%, which is why the sizes at the boundary are read as equal across thresholds and the corner’s 3.0% against 2.3% as a small difference rather than a measured one.

A lower threshold buys power at no measured cost in size #

The claims are that lowering the recentring threshold from Hansen’s to zero raises the test’s power against a better forecaster from 24.3% to 30.3% with sixteen stale candidates in the set and from 26.0% to 30.3% without them; that at zero the stale candidates cost nothing; that raising it to three times Hansen’s lowers the power to the studentised reality check’s 17.7%; and that its size stays at or below 1.0% at the boundary and 3.0% at the corner at every threshold measured.

Not claimed: anything at other numbers of origins, where the threshold’s rate is the whole point; anything about sets of candidates that are strongly dependent near the boundary; and nothing about the unstudentised reality check, whose reference distribution has no threshold at all.

Still open: the threshold at a different sample size #

Hansen’s threshold grows with $\log\log P$ — from 1.75 at a hundred origins to 1.97 at a thousand — so its cost in power should fall slowly as the sample grows, and the size a threshold of zero gives up at the corner should rise towards its asymptotic excess. Sweeping the threshold at twenty-five, a hundred and four hundred origins, at the corner and against the stale set, would place the sample size at which the standard threshold starts to earn its price; the prediction from this table is that a forecaster with a few years of monthly data never reaches it.