The power a calibration buys

When two searches run on one sample — where a regression breaks, and which column from a dictionary enters it — the honest test of whether either found anything is a test of their joint maximum. Two searches that share nothing built the scale this field measures it on: the doubled log-likelihood gain of each search alone, of both searched jointly, and the excess, which is how much a rule that charges the two separately over-charges. The overlap at the critical value read the same quantities at the 95% point a test actually uses and found that a threshold summing the two searches’ own critical values rejects between none and 4.8% of true nulls across the field’s pairs: charging separately is conservative, at the tail as well as on average.

That essay named what it left out. Conservative at a null is a cost only if it costs power somewhere, and power needs an alternative the null draws do not contain. This essay adds three — a real break, a real column, both — and asks what a correctly calibrated joint test buys over the rules a practitioner would otherwise use.

Four rules, one set of suprema #

The setting is the field’s: a regression of a response on four persistent predictors, errors from a first-order autoregression at 0.8, 120 rows, and three pairs of searches. A break and a column: where the regression’s five coefficients change, among seventy-three candidate rows, and which of six independent noise columns enters it. Two independent dictionaries: which column of one set of six enters, and which of another. A break and a window: where the regression changes and how wide a band of autocovariances whitens it, the pair the earlier fields began from.

Each sample yields three suprema per pair — each search’s gain alone and the joint search’s gain — and four rules read them. The joint rule rejects when the joint gain passes its own 95% point, simulated on twelve hundred null draws of the same law. The summed rule rejects when the joint gain passes the sum of the two searches’ own 95% points, which is what charging them separately amounts to. Bonferroni rejects when either search’s own gain passes its 97.5% point. And either, uncorrected, rejects when either passes its own 95% point; it is here for its size. Every threshold is simulated, so every rule’s size is measured rather than assumed.

The alternatives are added to the null sample after it is drawn, so the dependence is identical to the null’s: a shift of 1.5 in the response from row 60, a coefficient of 0.3 on the first column of the first dictionary, or both. Four hundred draws at each.

The first block is each rule’s size, and it is where the rules should agree. The joint and Bonferroni rules reject 6.3% of true nulls, the summed rule 6.3%, the uncorrected one 12.0% — twice its nominal level, as two looks at 5% should give. With twelve hundred simulated nulls and four hundred test draws the standard error on a size is about a point, so the calibrated rules are at their level.

Against a real break the joint rule finds it on 65.3% of samples, the summed rule on 64.0% and Bonferroni on 55.0%. That is the ordering the field’s argument predicts: the joint statistic sees a break as well as the break search alone does, and Bonferroni pays for its split by asking the break search to pass a 97.5% point instead of a 95% one.

Against a real column the ordering turns over. The joint rule finds it on 11.8% of samples. Bonferroni finds it on 73.0%. The summed rule finds it on 10.8%. A column entering the response with a coefficient of 0.3 is a large effect — the column search alone sees it more than four times in five — and the calibrated joint test, the rule the field’s whole construction argued for, misses it nearly nine times in ten. With both signals present the joint test is at 70.5% and Bonferroni at 75.8%.

Why the joint test is blind to the column #

The thresholds explain it before any power is computed. On the null draws, the break search alone has a 95% point of 66.4 on the doubled log-likelihood scale; the column search alone, 7.3; the two searched jointly, 72.5. A break in a regression of five coefficients, searched over seventy-three rows of persistent errors, finds a great deal of nothing: the errors’ own wandering looks like a change in the regression somewhere, and the search reports the best-looking somewhere. A column search over six independent noise columns finds a little.

The joint statistic is the gain of searching both at once, and on a null it is dominated by the break: the joint search’s threshold sits six units above the break search’s own. A real column adds about what it adds to the column search — some eleven units at a coefficient of 0.3 on 120 rows — to a joint statistic whose null spread is measured in tens. It moves the joint gain past its threshold only on the samples where the break search’s luck was already near the edge. Bonferroni reads the column search on its own scale, where eleven units is most of the way past a threshold of 8.7.

So a joint test calibrated on its own null is the right test for a signal the joint statistic’s dominant search is for, and a weak test for a signal its quieter search is for. That is a property of how the joint maximum is formed and it holds whatever the alternative’s size.

At half the break and half the column, the joint test finds the break on 24.3% of samples and Bonferroni on 20.0%; it finds the column on 7.2% — level with its size — and Bonferroni on 16.8%. At one and a half times, the break is found on 94.5% and 91.5%, and the column on 30.0% against 98.0%. A column so large that the uncorrected reading finds it 98.3% of the time is missed by the calibrated joint test seven times in ten. The joint test’s advantage on the break is between three and ten points at every strength; its deficit on the column grows with the column.

Where calibration does buy the power #

The pair the earlier fields began from is where calibrating the joint statistic pays, and pays heavily. On the break and the window, the summed rule’s size is 0.3% — the window search’s own 95% point is 125, because the likelihood rises with every lag of band whatever the data say, and the sum of the two thresholds sits far past where the joint statistic ever goes on a null. Its power against a real break is 14.5%. The calibrated joint rule holds a size of 7.2% and finds the break on 62.0%, forty-seven points more. Bonferroni finds it on 59.8%.

Here the joint statistic is the right instrument because the window search is not looking for a signal at all: the window is a nuisance, chosen to whiten the errors, and nothing in the alternatives is a window effect. The pair’s joint gain is the break search done properly — with its whitening chosen alongside it — and its own threshold is the threshold for a break found that way. Summing thresholds charges the break search for the window search’s uninformative rise, which is the over-charge the charge that is not a sum measured at a null, and it costs forty-seven points of power when there is something to find.

On the two independent dictionaries the four rules are close. Three quarters of the way to one search placed this pair at the zero of the field’s overlap scale, the control whose two searches share nothing but the response, and nothing about its power contradicts that placing. A column in the first set is found by the joint rule on 70.8% of samples, by Bonferroni on 73.3% and by the summed rule on 56.3%; the two searches have nearly the same scale — 95% points of 7.3 and 7.4 — so neither dominates the joint maximum, and the joint rule loses only what any rule loses by looking in two places.

What decides which rule to use #

Two properties of a pair, both visible before any data arrive.

Whether each search is looking for something. A nuisance search — a window chosen to whiten, a tuning parameter chosen to fit — belongs inside the joint statistic, because its rise on a null is not evidence of anything and the joint statistic’s own calibration absorbs it. Charging it separately is the error the field has been pricing, and here its price is most of the test’s power.

Whether the searches that are looking for something have comparable null scales. When one search’s null 95% point is nine times the other’s, a joint maximum is in effect the noisier search, and a signal for the quieter one has to overcome the noisier one’s luck. Bonferroni gives each search its own scale and its own share of the error rate, and on such a pair it is the better test for one alternative by a margin the joint test never wins back on the other.

The two properties separate the pairs here exactly. The break and the window: one real search and one nuisance — calibrate jointly. The two dictionaries: two real searches on one scale — either rule. The break and the column: two real searches on scales nine to one — split.

The same lesson from a different direction #

This is not the first place in the collection where a joint test has lost to a split. One test for eight nulls tested a forecast combination against its eight parts all at once, by a Wald test on the vector of encompassing statistics, and found that even read at its own simulated critical value it was less powerful than Bonferroni over the largest of the eight. The reason there was different in detail and the same in shape: a joint statistic sums evidence across components, so a signal concentrated in one component is diluted by the noise in all the others, and a rule that looks at each component on its own scale finds the concentrated signal sooner.

The break-and-column pair is the extreme case of that dilution. The joint search is not a sum of the two gains — it is the gain of a search over their product — but on a null it behaves like the break search with a small addition, so a column’s signal is diluted by the break search’s whole null spread. And the field’s earlier finding that the two searches share almost nothing — the excess on this pair is small and its overlap near zero — is exactly the condition under which Bonferroni’s split is least wasteful: two searches that do not overlap cost nothing to test separately beyond the halving of each one’s level.

It is worth saying what that does not mean. It does not mean the field’s construction was wrong to insist on the joint statistic. On the break-and-window pair it is the only calibrated rule that is also sensible, because a window is not a hypothesis anyone is testing and Bonferroni would spend half the error rate on it; there the joint test and Bonferroni nearly tie, 62.0% and 59.8%, and the joint test wins on size, 7.2% against 6.8%, only by the noise. What it means is that “calibrate the joint maximum” is a rule for searches whose maximum is the object of interest. Two different promises makes the same point about familywise and false-discovery corrections: which correction is right is a statement about which error the analysis exists to avoid, and the answer here is a statement about which signal the analysis exists to find.

How the numbers were made #

Twelve hundred null draws give each pair’s thresholds — the 95% points of each search alone and of the joint search, and the 97.5% points for Bonferroni — on the same dictionaries’ construction the field uses, a fresh set of noise columns drawn per sample from a stream of its own. Four hundred draws at each alternative, with the signals added to a null sample after it is drawn: the shift lands on every row from 60, the column enters with its coefficient on every row. Every supremum is the field’s: the break searched over the middle sixty per cent of rows with the regression’s five coefficients changing, the column chosen from six, the window from eight widths with its determinant and a unit charge a lag. Rates are reported with binomial standard errors of about two and a half points at four hundred draws.

Checked on the measurement: on the break and the window, the calibrated joint test finds a real break more than forty points more often than the summed thresholds; on the break and the column, it finds a real column less than a third as often as Bonferroni; and both calibrated rules hold their size within three points of 5% on every pair. The refusal is the claim the field’s construction invites — that the calibrated joint test is at least as powerful as Bonferroni at every alternative — and the column on the break-and-column pair breaks it by sixty points.

Still open: a joint statistic on a common scale #

The finding suggests a construction rather than a choice between two existing ones. The joint test is blind to the column because the break search’s gain is on a scale nine times the column’s. A joint statistic built from each search’s gain standardised by its own null — each gain’s null tail probability, say, with the joint statistic the smaller of the two and calibrated as a pair — would give each search its own scale, as Bonferroni does, and keep the joint calibration’s exactness, as the joint test does. Whether it beats Bonferroni anywhere is the measurement: it should on pairs whose searches are correlated, where Bonferroni’s split is wasteful, and match it on pairs like these, whose searches share almost nothing. It has not been built.

What is claimed here, and what is not #

This essay takes the power of four rules for one decision about two searches, under a first-order autoregression at 120 rows. The claims are that every calibrated rule holds its size within a point or two of 5% and the uncorrected reading is near 12%; that on the break-and-column pair the calibrated joint test finds a real break on 65.3% of samples against Bonferroni’s 55.0% and a real column on 11.8% against 73.0%, and at one and a half times the column’s strength on 30.0% against 98.0%; that on the break-and-window pair calibration raises the power against a real break from the summed rule’s 14.5% to 62.0%; that on two independent dictionaries the joint rule and Bonferroni are within three points; and that the null 95% points behind these are 66.4 for the break search, 7.3 for a column search and 125.0 for the window search.

What stays out, named as a decision: other laws, under which every threshold here would move; other alternatives, such as a break in one coefficient rather than all five, which would put a break search’s signal on a smaller scale; and the standardised joint statistic, deferred above.

The boundary against the overlap at the critical value is that it measured the rules’ sizes and this essay their power. Against the charge that is not a sum, which priced the break-and-window pair’s over-charge at a null, this essay finds what the over-charge costs when there is a break to find — and finds the joint test’s own weakness on a pair the earlier field did not run.