The overlap at the critical value

Two searches that share nothing built a sequence of five pairs of searches on one sample — two dictionaries of independent columns at the bottom, a break and a whitening window near the top, a break and a step column it contains at the very top — and measured for each how much less the two searches find together than the sum of what each finds alone. The field’s third essay put the break-and-window pair at three quarters of the way to being one search, and ended by naming what the whole sequence left out. Every number on it is an average. A rule that charges a pair of searches does not use the average of the joint statistic; it uses the point the joint statistic passes one time in twenty when nothing is there. An overlap of 0.762, the field’s highest measured pair, does not say what critical value to use.

This essay reads the same suprema at their quantiles. The answer it gets is that the overlap a rule needs is larger than the one the field measured, on every pair, and that the reason has nothing to do with what the searches share.

The same draws, read at a quantile #

Each draw gives three numbers on the doubled log-likelihood scale a chi-square critical value is quoted on: what search A finds on its own, what search B finds, and what the two find together. Averaged, they give the field’s overlap. Ranked, they give each statistic’s quantiles, and the overlap at a level $p$ is the same arithmetic on quantiles:

$$\frac{q_p(A) + q_p(B) - q_p(C)}{\min\bigl(q_p(A),\,q_p(B)\bigr)},$$

which is what a rule adding the two searches’ own critical values over-charges, as a share of the smaller. At $p = 0.95$ it is exactly the correction a practitioner would want from the field.

On every pair but the containment the overlap climbs with the level. The two independent dictionaries — the field’s control, whose overlap on average is −0.024 on this scale — read −0.176 at the median, zero at the 75% point, 0.238 at the 90%, 0.320 at the 95% and 0.538 at the 99%. The break and the whitening window read 0.512 at the median and 0.732 at the 95% point. The break and an independent column, which the field found complementary on average, read −1.211 at the median and 0.205 at the 95% point. The containment — a break search and a step column the break search already includes — reads exactly 1 at every level, because its joint search is its first search and so are all of its quantiles.

The scale here is the log-likelihood rather than the share of the residual sum the field mostly used, because a critical value is a log-likelihood. On the share scale the break-and-window pair’s average overlap is the field’s 0.762; on this scale it is 0.564. Either way the 95% point is above it.

The control, where nothing is shared #

The control is the clearest case, because there is nothing to explain except arithmetic. Its two dictionaries are six noise columns each, drawn independently of the response and of each other, and each search picks the one column that fits best. On average the joint search finds 6.57 units of log-likelihood and the two separately find 3.29 and 3.20: additive to the second decimal. Their draw-by-draw correlation is 0.025.

At the 95% point the two separate searches’ critical values are 7.50 and 7.45, summing to 14.96, and the joint search’s is 12.57. A rule adding the critical values over-charges by 2.38 units — the 0.320 overlap — on a pair whose searches have nothing in common.

The reason is that a quantile of a sum is not the sum of quantiles, even for independent quantities. For the joint statistic to pass the sum of the two 95% points, both searches have to do unusually well on the same draw, or one has to do far better than its own 95% point; the first happens on about one draw in four hundred, and neither statistic’s tail is long enough for the second to make up the difference. So the sum of the 95% points sits well out past the joint statistic’s own 95% point. Below the median the same skew runs the other way, and the sum of the medians falls short of the median of the sum, which is why the control’s overlap is negative there.

That account can be checked without the joint search at all. Pair every value of one search with every third value of the other, at random rather than on the same draw, and the 480,000 sums are the joint statistic two searches would have if they shared nothing — not even the response. Its 95% point is 12.16, beside the joint search’s 12.57 and the summed 14.96. The convolution follows the joint search and not the sum: almost all of the over-charge at the critical value is the arithmetic of adding tails. The joint search sits a few tenths above the convolution, which is the same small excess it shows on average — on average the joint search finds a little more than the two separately, 6.57 against 6.49, from pairs of columns that fit better together than either does alone.

For a normal statistic the same arithmetic has a closed form. Two independent normals with spreads $\sigma_A$ and $\sigma_B$ have a sum whose upper point sits $z\sqrt{\sigma_A^2 + \sigma_B^2}$ above its mean, while the sum of their upper points sits $z(\sigma_A + \sigma_B)$ above; the gap is positive at every upper level and grows with $z$. Supremum statistics are skewed rather than normal, which moves where the overlap crosses zero — near the 75% point here rather than at the median — but not the direction.

The same pairs, one level up #

Read at the 95% point, the field’s sequence keeps its order and moves up. The break and an independent column go from −0.758 to 0.205; the two independent dictionaries from −0.024 to 0.320; the two step dictionaries, cut a few rows apart, from 0.009 to 0.541; the break and the window from 0.564 to 0.732; the containment stays at 1.

The step dictionaries are worth a sentence. On average they are as additive as the control, and draw by draw their two searches are correlated at 0.915, because a step at row 40 and a step at row 43 pick up the same feature of the sample. Correlation of that size moves nothing on average and a great deal in the tail: the two searches have their good draws together, so their critical values sum to far more than the joint statistic’s. Here the convolution, which assumes independence, overshoots the joint search — 80.53 against 73.15 — and the excess over the convolution is the correlation’s share of the overlap.

So the field’s sequence was a sequence of averages, and a rule lives on a different sequence whose values are all higher. The overlap a rule needs is never smaller than the one measured on average, on any pair here, and on the two pairs closest to additive it is more than a third of a search’s whole critical value larger.

The over-charge in units a reader knows #

The over-charge is easier to judge against a familiar yardstick. A single free parameter’s 95% critical value on this scale is the chi-square point on one degree of freedom, 3.841. At the 95% point the two independent dictionaries’ summed critical value over-charges by 2.38 units — 0.62 of a parameter’s worth, for two searches that share nothing. The break and an independent column over-charge by 1.54, 0.40 of a parameter. The two step dictionaries over-charge by 26.27, which is 6.84 parameters’ worth, and the break and the window by 50.57, 13.16 parameters’ worth.

Put the other way, the joint statistic’s 95% point is a fraction of the summed one: 0.841 of it for the control, 0.980 for the break and a column, 0.736 for the step dictionaries, 0.744 for the break and the window, and 0.576 for the containment, whose joint critical value is simply the break search’s own. A rule that adds the parts sets its threshold between two per cent and seventy-four per cent too high depending on the pair, and on no pair here is it too low.

The control’s 0.62 of a parameter is the cleanest number in the essay to carry away. It is the price of adding two critical values rather than simulating one, paid by a pair of searches with nothing in common, and it is the floor under every pair a practitioner would actually run: whatever the searches share is charged on top of it.

What a threshold built from the parts rejects #

The practical test is what a joint threshold built from the two searches’ own critical values does at a true null.

Summing the two critical values rejects 1.4% of true nulls for the control, 0.7% for the step dictionaries, 0.1% for the break and the window, and 0.0% for the containment. The only pair near its level is the break and an independent column, at 4.8%, and it gets there by accident: it is complementary on average, so its joint search finds more than the sum, and that pushes against the tail arithmetic and nearly cancels it.

Taking the field’s mean overlap off the sum does not repair any of them except the containment. It gives 1.3%, 3.2%, 0.8%, 5.0% and 2.3%. The containment comes out exact because its overlap is one at every level, mean included, so the correction removes the redundant search entirely. Every other pair’s overlap at the critical value is larger than its mean overlap, so the corrected threshold is still too high.

That is the same failure the charge that is not a sum found for one pair — a test that charges two searches what each costs alone barely fires — now located. It was never mainly that the searches overlap. On the control they do not, and the summed critical value still holds the test at under a third of its level.

A conservative test does not announce itself. A practitioner running the summed threshold on the break and the window sees a statistic below its critical value on almost every sample, and reads that as a sample with nothing in it, which is what a test at its nominal level would license. Here it licenses nothing: the test would say the same of a sample with a real break of moderate size, because the threshold has been put fifty units past where the joint statistic’s null lives. The error is invisible from inside one analysis and is visible only to a simulation of the procedure as a whole, which is the argument these essays keep returning to — the level of a procedure is a measured property of the procedure, searches included.

The convolution as a calibration, and where it stops #

The convolution that explained the control can also be used as a threshold. It needs only the two searches’ own null laws, each of which a practitioner would have simulated anyway to calibrate the searches separately, and pairing them at random costs nothing. Its 95% point is the critical value the pair would have if the two searches shared nothing at all.

On the control it is close: the joint statistic passes the convolved threshold on 6.1% of null draws, a point above nominal, because the joint search reaches slightly further than two independent searches would. On the break and an independent column it gives 5.7%. On the step dictionaries, whose searches are correlated at 0.915, it gives 3.0%, and on the break and the window 0.6% — because those two searches share ground, the independent convolution’s tail is much longer than the joint statistic’s, and the threshold it gives is far too high.

So the convolution is a calibration for searches that share nothing, and only for those. It repairs the arithmetic of adding tails, which is the whole of the control’s problem, and it says nothing about shared ground, which is most of the problem for any pair a practitioner would actually run. For those the joint statistic has to be simulated as a joint statistic.

At the stricter level the same pattern is sharper. Summing the two searches’ own 99% points holds the control at 0.17% against a nominal 1%; the convolved 99% point holds it at 1.00%. For the break and the window both are almost nothing — 0.00% summed and 0.08% convolved.

Why no single overlap number transports #

The overlap at the 99% point is not the overlap at the 95% point either. The control reads 0.538 at 99% against 0.320 at 95%, the step dictionaries 0.682 against 0.541, and the break and the window 0.754 against 0.732. A rule at a stricter level needs a larger correction, and how much larger depends on the shapes of both statistics’ tails, which is to say on the pair.

That closes a door the field had left open. It would have been convenient if each pair’s overlap were a single property — a measure of how much two searches share, to be looked up and applied at any level. On average it is close to that. At the critical values it is a function of the level and of the joint shape of two skewed statistics, and the only general statement it supports is a direction: add critical values and the test is conservative, by more the further out the level is.

The repair is the one the field on two searches used, and nothing in the sequence replaces it: simulate the joint statistic under the null and read its quantile directly. A pair of searches has one critical value and it belongs to the pair. What a zero is made of separated the field’s averages into overlap and interaction; this essay says that whatever they are made of, the averages are the wrong summary for a test.

Still open: the separation, at the critical value #

The separation field split each pair’s average excess into ground both searches find and configurations only the joint search reaches, using one extra supremum per draw. The same split can be read at quantiles, and it is the obvious next measurement: whether the tail over-charge on the control is all arithmetic of adding tails, as the convolution suggests, or whether the interaction term — which is never negative — has a tail of its own that the convolution cannot see. The convolution’s 12.16 against the joint search’s 12.57 leaves room for the second at a few tenths of a unit. The split at the critical value makes that measurement.

A second open question is the law. Everything here is under the first-order autoregression. The tail arithmetic does not depend on the law, but the break-and-window pair’s shared ground does, and the earlier field’s rule-dependent charge moved by half between laws.

The claim, and where it stops #

This essay takes the field’s overlap between two searches, read at quantiles of the joint statistic rather than at its mean. The claims are that on the doubled log-likelihood scale the two independent dictionaries overlap by −0.024 on average and by 0.320 at the 95% point, with draw-by-draw correlation 0.025; that their summed 95% points are 14.96 against the joint search’s 12.57, and the two marginal laws convolved give 12.16; that every pair but the containment overlaps more at the 95% point than at the median, and the sequence keeps its order at the critical value; that a joint threshold summing the two searches’ own critical values rejects between 0.0% and 4.8% of true nulls across the five pairs, and one corrected by the mean overlap between 0.8% and 5.0%, exact only for the containment; and that the overlap at 99% is larger again.

What stays out, named as a decision: other laws, deferred above; other levels of the individual tests, since every critical value here is a 95% point; and the power of a correctly calibrated joint test, which is the charge field’s subject and needs alternatives this essay does not draw.