Normal Distribution
nothing is called 95% until it has been counted
Concept

Composite null — where it appears

A hypothesis satisfied by a whole region of parameter values rather than by a single point, so that one statistic has many null distributions. A test calibrated at one corner of that region is calibrated for a configuration the data may not be in, and loses size as the data moves away from it.

Named by 8 essays across 5 fields — each of them below, with the objects they name alongside it.

Sixteen candidates nobody would have run, and what they cost. At φ = 0.65 one candidate in the original set is genuinely better than the benchmark, and the question is how often each procedure finds it. The added candidates are stale copies of the last value — read two, four, six … steps late — every one of them worse than the benchmark by at least 43%, and not one of them is ever the best candidate in a sample. The reality check goes from 35.5% to 0.0% as they are added, because its reference distribution has to assume every candidate is exactly as good as the benchmark and sixteen such assumptions is a critical value nothing reaches. The recentred version, which drops from the recentring the candidates the data has already ruled out — 15.1 of 24 of them — goes from 25.5% to 24.5%. The top line never moves: reporting the winner's own p-value cannot notice a change to a set it never looks at.

The models that were never in the running

A reference distribution for a set has to assume something about every candidate in it. Assuming that all of them are as good as the benchmark is what makes the reality check honest, and it is what sixteen hopeless candidates use to destroy it.

ranking · Multiplicity What the corner costs when the table is full of hopeless candidates. The benchmark holds two predictors, one of which is worth 1; a third predictor, worth the amount on the horizontal axis, is held only by candidates the benchmark does not contain. At the left the null is true and both procedures hold their level. To the right there is a genuinely better candidate, and the uncorrected reality check finds it 2.7% of the time while the same test with the clearly bad columns recentred finds it 51.0% of the time. The columns doing the damage are the ones nobody would have looked at twice: they are so far behind that they cannot win, and calibrating as though they might is what makes the test blind.

The corner the test is calibrated at

"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.

select · Reference An exact test rejecting a true hypothesis a fifth of the time. How often each analysis reports an effect when the average treatment effect is exactly zero and the effect varies between units, at 150 units with 25% treated. The permutation test on the difference in means reads 4.20% where the effect is constant — where the two nulls coincide and its exactness applies — and 22.93% where the effect varies with a standard deviation of 3. The same test on the studentised difference reads 6.27% there, and the ordinary large-sample t, which makes no exactness claim at all, reads 6.60%.

The null the exactness is for

A permutation test is exact under the hypothesis that the treatment changed nothing for anybody. Under the hypothesis it changed nothing on average, with a quarter of the units treated and the effect varying between them, it rejects a true null 22.93% of the time.

exact · Nuisance Power as the recentring's threshold moves. The recentred set test's power at a persistence of 0.65, where one forecaster genuinely beats the benchmark, over 300 draws of 100 origins, as its threshold runs from zero to three times Hansen's. With the eight windows alone: 30.3%, 28.3%, 26.0%, 26.0%, 26.0%, 26.0%. With sixteen stale candidates added: 30.3%, 28.3%, 24.3%, 21.0%, 19.3%, 17.7%. The studentised reality check, which sets nothing aside, has 26.0% and 17.7%.

The threshold the recentring reads

A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.

ranking · Multiplicity How slow a shared return a panel of pairs can see at 200 observations each. The average of the pairs' Engle–Granger statistics, against its own 5% point. One pair reaches 80% power up to a half-life of 5.1 steps, four pairs 9.2, sixteen 15.0. At a half-life of 12 the three find the relation 20.0%, 54.2% and 99.8% of the time.

A slow return across many pairs

One pair of two hundred observations is found tied four times in five only if its gap halves within 5.1 steps. Sixteen independent pairs sharing the same half-life, tested by the average of their Engle–Granger statistics, are found four times in five up to a half-life of 15.0 — and at a half-life of twelve, 99.8% of the time against one pair's 20.0%. What the widened test rejects is a statement about the panel: with half the pairs tied it rejects 55.4% of the time, with a quarter 21.2%, and a pair-by-pair reading of a panel in which all sixteen are tied finds 2.65 of them, against 0.78 when none are.

cointegration · Spurious What Hansen's threshold costs, by the number of origins. The power a threshold of zero keeps over Hansen's, on the same draws. with sixteen stale candidates: 7.5%, 7.3%, 6.8%, 5.3%, 3.7%, 6.0%. the eight windows alone: 2.7%, 3.8%, 4.7%, 5.3%, 3.7%, 6.0%, at 25, 50, 100, 200, 400, 800 origins. The price does not fall as the sample grows.

A threshold against its sample

A recentred test's threshold exists so that the test is right in the limit, and at a hundred origins it costs a fifth of the power against a stale set. The natural defence is that the price falls as the sample grows and the guarantee starts to earn it. Across twenty-five to eight hundred origins it does not: a threshold of zero keeps four to seven points more power at every size, and its size at the corner stays at the level from fifty origins on. What does change with the sample is the stale set, which stops costing anything by two hundred origins under Hansen's rule and by eight hundred under the most cautious.

ranking · Multiplicity How fast a gap has to close before a sample can see it close. The power of the test against the half-life of a disagreement, at 100, 200, 400 observations, each read against its own simulated critical value. Every pair in every reading is genuinely tied together, so a non-rejection is a miss. At 200 observations a gap that halves in 3 steps is found 99.9% of the time and one that halves in 12 steps is found 15.3% of the time — and by 35 steps the reading is 6.1%, which is the test's own size. Beyond that the curves are flat because there is nothing left to detect with.

How slow a return a sample can see

At two hundred observations the test finds a gap that halves in five steps four times in five, one that halves in eight 37.3% of the time, and one that halves in fifty 4.95% of the time — which is the rate at which it finds pairs with no mechanism at all. The boundary moves with the sample, not with its square root.

timeseries · Spurious How often the pooled test finds a relation in a panel with none, when the pairs share a shock. Against the 5% point computed for independent pairs, over 1,200 panels of pairs of 200 observations with no pair tied. Sixteen pairs, raw: 4.6% at a common share of 0, 7.2% at a common share of 0.25, 15.5% at a common share of 0.5, 27.3% at a common share of 0.75, 31.6% at a common share of 0.9; after removing the cross-sectional averages: 5.3%, 5.1%, 5.8%, 5.3%, 4.8%. Four pairs, raw: 5.4%, 6.5%, 6.8%, 12.6%, 16.9%.

A shock every pair shares

Sixteen pairs of series whose shocks are half common are worth 5.37 independent pairs to the panel's average statistic, and sixteen that share nine tenths are worth 1.44. Read against the 5% point computed for independent pairs, the pooled test then calls a panel with no relation tied 15.5% and 31.6% of the time; read against its own point, it finds a relation shared by every pair 22.3% of the time at nine tenths common, against one pair's 20.0%. Removing each date's cross-sectional average gives back the level and all sixteen pairs when every pair feels the shock equally, and leaves 13.3% false alarms when the loadings run from a half to one and a half.

cointegration · Spurious

Named alongside it

The objects these essays reach for when they reach for this one.

Statistical powerMonte CarloLeast favourable configurationBenchmark forecastCointegrationCritical valueEngle–GrangerError rateHalf-lifeMultiple comparisonsReality checkReference distribution

All concepts