Normal Distribution
nothing is called 95% until it has been counted
Concept

Data snooping — where it appears

Reporting the best of many comparisons at the significance level of one, so that the search itself is what produced the result. The reading is not visible in the output: a table of models and their errors looks the same whether one comparison was made or two hundred and ten.

Named by 13 essays across 6 fields — each of them below, with the objects they name alongside it.

The eighth was not a constant. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations between the candidates, over 800 draws apiece. The earlier field reports this flat at about an eighth across list length, on a table and a world it never varies. Vary how much the omitted coefficients are worth — one multiplier, with the table, the list, the law and the sample size all held — and it runs from 17.6% to 1.5%, a factor of 11.75. The world in which every candidate is true is the world in which the tuning list decides most; the world in which one candidate dominates is the world in which it decides nothing.

The eighth that was not a constant

How often a per-candidate tuning list changes which candidate wins is reported flat at about an eighth across list length. Vary how far apart the candidates are instead and it runs from 17.6% to 1.5%.

turnover · Order selection What each pair is made of. Each pair of searches, over 300 draws, split into the two effects its excess is the difference of. The overlap is what the second search loses by having the first already run at its own answer; the interaction is what the joint search finds by moving the first off it. They subtract to the excess exactly, on every draw, because the pinned supremum cancels. Two disjoint dictionaries of independent columns read an excess of 0.000011 and are made of 0.000514 and 0.000503. A break paired with a dictionary of step columns has an interaction of exactly 0 and is all overlap. And a break paired with an independent column has an overlap of -0.004395 against an interaction of 0.002364, which is what puts its excess below zero.

Two effects in one number

How much two searches over one sample share is measured as the net of two things — ground both of them find, and configurations only the joint search reaches. One extra supremum per draw separates them exactly.

separate · Break point Eight candidates, one of them exactly as good as the benchmark. The candidate set: moving averages of the last 1, 2, 3, 5, 8, 13, 21 and 34 observations, each drawn as its expected squared error divided by the benchmark's — the mean of all 60. The persistence is not chosen, it is solved for: at φ = 0.4895 the best candidate in the set, the average of 2, has exactly the benchmark's expected squared error, and every other candidate is worse by between 0.5% and 5.1%. So the null that no candidate beats the benchmark is true, with one candidate on its boundary. Everything a set comparison claims about its own error rate has to be measured here, because anywhere further inside the null every procedure flatters itself.

Eight forecasters and one benchmark

A set of forecasters is a multiplicity problem on top of a dependence problem, and the two do not separate. Eight windows of one series carry the multiplicity of two and a half independent comparisons; eight separate problems carry eight.

ranking · Multiplicity Both halves grow; the difference does not. The control pair's two components and their difference, against how much each of its two searches can find, over 1200 draws at each dictionary size. Two disjoint sets of independent columns are additive at every size — the excess stays inside a standard error or two of zero throughout — and it is not because there is nothing there. The overlap grows from 0.000112 at two columns to 0.000870 at ten, a factor of 7.76, and the interaction grows with it, staying within a factor of two of the overlap at every size. Two searches competing for one residual sum share ground and find configurations neither has alone, in almost equal measure, and their difference is what the earlier field's scale calls zero.

What a zero is made of

Two disjoint dictionaries of independent columns read an excess of 0.000116 and are made of an overlap of 0.000583 and an interaction of 0.000467. The control the whole scale is anchored on reads zero because two effects cancel.

separate · Break point The split depends on the order. How much two searches share, measured both ways round, over 300 draws. Pinning the first search at its own answer and searching the second gives one overlap; pinning the second and searching the first gives another. A break paired with an independent column reads -0.004395 one way and 0.002163 the other, at 8.70 paired standard errors and on opposite sides of zero. The excess the two components subtract to is the same in both orders by construction, so what changes is only how it is attributed. There is no order-free way to say which of two searches found ground both can reach, and the two orders bracket it.

A split that depends on the order

Run the second search first and pin that instead, and the same draw gives a different overlap and a different interaction — with the same difference. And one pair has no second order at all.

separate · Break point Sixteen candidates nobody would have run, and what they cost. At φ = 0.65 one candidate in the original set is genuinely better than the benchmark, and the question is how often each procedure finds it. The added candidates are stale copies of the last value — read two, four, six … steps late — every one of them worse than the benchmark by at least 43%, and not one of them is ever the best candidate in a sample. The reality check goes from 35.5% to 0.0% as they are added, because its reference distribution has to assume every candidate is exactly as good as the benchmark and sixteen such assumptions is a critical value nothing reaches. The recentred version, which drops from the recentring the candidates the data has already ruled out — 15.1 of 24 of them — goes from 25.5% to 24.5%. The top line never moves: reporting the winner's own p-value cannot notice a change to a set it never looks at.

The models that were never in the running

A reference distribution for a set has to assume something about every candidate in it. Assuming that all of them are as good as the benchmark is what makes the reality check honest, and it is what sixteen hopeless candidates use to destroy it.

ranking · Multiplicity One window for the table, or one each. The regret of the same fifteen-candidate table under AR(1) at 0.8 over 150 draws, with the window attached three ways. Chosen once from the fullest candidate's residuals it gives up 0.02518. Chosen from each candidate's own residuals, with the covariance estimate still shared, it gives up 0.02799 — a paired cost of 0.00281 at 2.0 standard errors for the tuning parameter alone. Estimating the covariance per candidate as well costs 0.01087, so the objection already on record is about 3.9 times the size of the one that was not.

A window for every candidate

The window and the order a whitening needs are chosen once, from the fullest candidate, on an argument that was made about an estimated covariance. A tuning parameter is not a covariance, and the two cost different amounts.

together · Order selection An overlap that depends on where it is read. Each pair's overlap on the doubled log-likelihood scale — the sum of the two searches' own quantiles less the joint search's quantile, as a share of the smaller — at five quantile levels, over 1200 draws under the first-order autoregression. two dictionaries of independent columns: -0.176 at 0.5, -0.000 at 0.75, 0.238 at 0.9, 0.320 at 0.95, 0.538 at 0.99; a break, and an independent column: -1.211 at 0.5, -0.406 at 0.75, -0.029 at 0.9, 0.205 at 0.95, 0.161 at 0.99; two dictionaries of step columns: -0.397 at 0.5, 0.145 at 0.75, 0.438 at 0.9, 0.541 at 0.95, 0.682 at 0.99; a break, and a step column it contains: 1.000 at 0.5, 1.000 at 0.75, 1.000 at 0.9, 1.000 at 0.95, 1.000 at 0.99; a break, and a whitening window: 0.512 at 0.5, 0.640 at 0.75, 0.716 at 0.9, 0.732 at 0.95, 0.754 at 0.99. Every pair but the containment overlaps more at the 95% point than at the median, the control included.

The overlap at the critical value

Two searches over independent noise columns are additive on average: the joint search finds what the two find separately, to within a fortieth of the smaller. At the point a test actually uses — the joint statistic's 95% quantile — the same pair overlaps by 0.320, and a rule that adds the two searches' own critical values rejects 1.4% of true nulls instead of 5%. Nothing about the searches causes it. Quantiles of a sum do not add, and on every pair in the field the overlap at the critical value is larger than the overlap the field measured.

apart · Criterion The overlap climbs and the interaction does not. The over-charge at each quantile level split into overlap — the two searches' own quantiles less the sequential run's — and interaction — the joint search's quantile less the sequential run's — for three pairs over 1200 draws. two independent dictionaries: overlap -0.40, 0.01, 1.55, 2.44, 5.72; interaction 0.07, 0.01, 0.11, 0.06, 0.00. two step dictionaries: overlap -0.73, 8.19, 22.53, 31.02, 47.75; interaction 4.87, 4.55, 4.90, 4.75, 1.10. a break and a window: overlap 17.88, 29.52, 43.01, 53.79, 71.61; interaction 1.57, 1.29, 1.19, 3.22, 2.75, at the 50%, 75%, 90%, 95% and 99% points.

The split at the critical value

Split the over-charge of two searches at the joint statistic's 95% point rather than at its mean, and the two components behave differently. The interaction — what only the joint search can reach — stays within two units of its average at every level up to the 95% point. The overlap climbs from nothing to most of the over-charge, on two searches over independent columns as much as on searches that share ground. At the critical value the split's overlap has stopped meaning shared ground: it is mostly the arithmetic of adding tails, and the identity that defines it hides that.

separate · Break point What four adjusted analyses report about their own precision and what they have, 50 patients an arm, baseline correlated 0.7. Reported against actual variance of the treatment estimate, as shares of the unadjusted analysis's, over 20,000 trials of 50 patients an arm. Adjusted for the baseline as measured: 0.529 reported, 0.518 actual. Median cut fixed in advance: 0.716 and 0.700. The decile cut with the best fit: 0.671 and 0.736. The decile cut with the smallest p-value: 0.762 and 1.615.

The cut that fitted best

A trial that adjusts for its baseline cut at whichever of the sample's nine deciles fits the outcomes best — without ever looking at the treatment difference — reports a variance 0.671 of the unadjusted analysis's where its estimates actually have 0.736, which is worse than the 0.700 a median fixed in advance delivers. The sample says the chosen cut kept 72.4% of what the baseline could remove; the population says it kept less than a median. A true null is rejected 5.87% of the time at fifty patients an arm and 9.31% at ten, and a cut chosen for its p-value rejects 14.22%.

planned · Power Four rules for a break and a column. How often each of four rules says something is there, for the pair of searches over a break and a column, under four truths, 400 draws each under a first-order autoregression at 0.8 with every threshold simulated on 1200 null draws. nothing there: joint, calibrated 6.3%, summed thresholds 6.3%, Bonferroni 6.3%, either, uncorrected 12.0%. a shift from row 60: joint, calibrated 65.3%, summed thresholds 64.0%, Bonferroni 55.0%, either, uncorrected 69.3%. a column enters: joint, calibrated 11.8%, summed thresholds 10.8%, Bonferroni 73.0%, either, uncorrected 82.5%. both: joint, calibrated 70.5%, summed thresholds 69.0%, Bonferroni 75.8%, either, uncorrected 84.8%. The first block is each rule's size.

The power a calibration buys

Charging two searches separately makes a joint test conservative, and calibrating the joint statistic on its own null repairs that. Against a real break, on the pair of a break search and a window search, the repair is worth 47 points of power — 62.0% against 14.5%. On the pair of a break search and a column search it is worth nothing, and worse: a real column is found by the calibrated joint test on 11.8% of samples and by a plain Bonferroni split on 73.0%, because a joint statistic takes its threshold from the noisier of its two searches.

apart · Criterion One order's interaction climbs under long memory. The break-and-window pair's interaction — the joint search's quantile less the sequential run's — at five quantile levels, in the field's order (break first, solid) and the other (window first, dashed), under three laws, 1200 draws each. first-order autoregression: break first 1.57, 1.29, 1.19, 3.22, 2.75; window first 0.98, 1.26, 1.53, 2.55, 2.14. five-period moving average: break first 2.00, 1.88, 2.29, 2.63, 0.60; window first 0.16, 0.21, 0.26, 0.29, 0.00. long memory: break first 0.53, 0.80, 1.05, 0.46, 1.35; window first 2.01, 1.62, 4.75, 6.07, 7.85, at the 50%, 75%, 90%, 95% and 99% points. Under two of the laws, and in the field's order under all three, the interaction stays within a few units of its median. Under long memory with the window searched first it climbs from 2.01 to 7.85, against an average of 1.99; the same order's averages under the other two laws are 0.94 and 0.17.

A tail the law gives the pair

Split the over-charge of two searches at their critical value and the interaction — the configurations only the joint search reaches — stays near its average while the overlap climbs. That held under one law. Under a five-period moving average and under long memory it holds for every pair in the order the field reads, to within three units. In the other order, one pair acquires a tail of its own: with the window searched first and the break second, the interaction under long memory climbs from 2.0 at the median to 7.9 at the 99th percentile, four times its average.

separate · Break point What Hansen's threshold costs, by the number of origins. The power a threshold of zero keeps over Hansen's, on the same draws. with sixteen stale candidates: 7.5%, 7.3%, 6.8%, 5.3%, 3.7%, 6.0%. the eight windows alone: 2.7%, 3.8%, 4.7%, 5.3%, 3.7%, 6.0%, at 25, 50, 100, 200, 400, 800 origins. The price does not fall as the sample grows.

A threshold against its sample

A recentred test's threshold exists so that the test is right in the limit, and at a hundred origins it costs a fifth of the power against a stale set. The natural defence is that the price falls as the sample grows and the guarantee starts to earn it. Across twenty-five to eight hundred origins it does not: a threshold of zero keeps four to seven points more power at every size, and its size at the corner stays at the level from fifty origins on. What does change with the sample is the stale set, which stops costing anything by two hundred origins under Hansen's rule and by eight hundred under the most cautious.

ranking · Multiplicity

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloCombinatorial searchLikelihood ratioSelection effectChange pointSpecification searchOverfittingSupremum statisticBenchmark forecastCritical valueDependenceVariance decomposition

All concepts