Selection effect — where it appears
Named by 49 essays across 25 fields — each of them below, with the objects they name alongside it.
A break that was looked for
A two-regime whitening finds its change point by maximising a profile, and then reads a criterion that counts parameters. Under no break there is no parameter to count, because every position describes the same model.
A criterion is a prediction of the hold-out
A rolling hold-out spends half the sample measuring what a criterion computes from all of it. Against an oracle that is arithmetic rather than an estimate, the criterion gives up 0.01701 and the hold-out 0.03200 — and the number the hold-out reports for its own winner is optimistic by more than either.
A rate times a size
A sweep reported what it costs to let every candidate choose its own tuning parameter and found it flat across the list. It was reporting a product, and the two things multiplied together do not behave the same way at all.
The comparison that was not made
Choosing a whitening's window separately for every candidate costs 0.00401 of regret. The same question about an order was named and left, because the two lists are different lengths. The order's answer is 0.00360, and matching the lists changes almost nothing.
The eighth that was not a constant
How often a per-candidate tuning list changes which candidate wins is reported flat at about an eighth across list length. Vary how far apart the candidates are instead and it runs from 17.6% to 1.5%.
The length nobody has
Every comparison of block windows in this collection is made at each window's own best block length. That length has a standard deviation of sixteen across draws and averages twenty-five. No rule is aimed at it.
Two effects in one number
How much two searches over one sample share is measured as the net of two things — ground both of them find, and configurations only the joint search reaches. One extra supremum per draw separates them exactly.
Two searches, one sample
A searched break in a regression manufactures 34.7 of likelihood ratio where a count of coefficients says 11.1. A searched window manufactures 84.0. The two together manufacture 99.4, not 118.7.
Two searches that share nothing
Two searches over independent columns remove shares of the residual sum that add exactly. On the scale a chi-square point is quoted on they look super-additive by a fifth of a unit, and none of it is overlap.
A search that is already the other
A break search shifts every coefficient after a row, so a step column is one of the directions it can move in. Paired with a dictionary of them it reads exactly one, on every draw, and that fixes the top of the scale.
Eight forecasters and one benchmark
A set of forecasters is a multiplicity problem on top of a dependence problem, and the two do not separate. Eight windows of one series carry the multiplicity of two and a half independent comparisons; eight separate problems carry eight.
The charge that is not a sum
Charging two searches what each costs on its own is conservative, and conservative here means the test never fires. At the largest break measured it declares nothing, on every draw, while a calibrated threshold reaches 29%.
The quarrel that changes the winner
A disagreement about the tuning parameter costs 0.031 when it changes which candidate the table selects and −0.0007 when it does not. The distance between the values disagreed about has nothing to do with it.
Two factors pointing opposite ways
As the candidates on a table are pulled apart, they quarrel about the tuning parameter three times as often and the quarrel decides the winner thirty times less often. A sweep that reads the first factor has read the one pointing the wrong way.
What a zero is made of
Two disjoint dictionaries of independent columns read an excess of 0.000116 and are made of an overlap of 0.000583 and an interaction of 0.000467. The control the whole scale is anchored on reads zero because two effects cancel.
The zero that was a crossing
A cell that leaks 0.002% where adding its two halves gives 16.6% is a field's headline. On a finer grid it passes through zero at a rank correlation of 0.38 — two hundredths from where it was measured.
A charge that depends on the rule
The break search's charge is 34.7 on its own and 15.4 once a window has been chosen from the same sample. Most of what a break search finds under correlated errors is the correlation, and a whitening has taken it already.
A list is not a rule
How often fifteen candidates disagree about a tuning parameter runs from nothing at two values on the list to two draws in five at thirteen. What the disagreement costs does not move at all.
A split that depends on the order
Run the second search first and pin that instead, and the same draw gives a different overlap and a different interaction — with the same difference. And one pair has no second order at all.
A table and a list
A nested ladder of candidates differing by one coefficient was predicted to turn over more often at every list length. It turns over less at every one, and its list changes the winner half as often.
Choosing whether to break
Charging what the search manufactures takes a rule from splitting a stationary sample on 99% of draws to 16%. It also costs regret, because the two mistakes a rule can make are not the same size.
How often it matters
The disagreement rate rises by half across the list and the share of disagreements that decide anything falls by nearly the same factor. Their product — how often the tuning list changes which candidate wins — sits at an eighth and does not move.
The condition that cannot be dropped
The weights may not read the block they weight. Estimate the variance ratio inside each block rather than across the trial and the coverage falls to 83% — on an interval that is at the same time seventy per cent wider.
The repair that was exact and made it worse
A penalty computed from the trace is exactly the optimism it estimates, and selecting with it gives up a fifth more than not correcting anything. The row count entered the criterion twice, and a penalty is the second place.
Three quarters of the way to one search
The pair that started this reads 0.762 on a scale whose one is containment. And the pair that shares nothing but its response reads −0.306, so the sign the earlier field found does not transport at all.
What choosing the length costs
The gap between two block windows at the best available length is 2.12 points. What the best rule a practitioner could run gives up against that same length is 7.26. The argument is a third of the size of the thing it is inside.
A schedule that reads the mean
The block sizes may be anything at all provided they are functions of the contrasts. Two natural schedules break that, in opposite directions — and the most natural mistake of the three is not a schedule at all but a stopping rule, at 86.87% coverage and fewer observations.
A second break on a flat profile
Searching a hundred and twenty rows for one change point manufactures five units of likelihood. Searching for a second manufactures four more, on a series that has at most one — and on a profile whose whole range is under seven.
A window for every candidate
The window and the order a whitening needs are chosen once, from the fullest candidate, on an argument that was made about an estimated covariance. A tuning parameter is not a covariance, and the two cost different amounts.
The interval after a stop it chose
A rule that stops when the estimated precision is good enough stops on the samples whose estimate was small. Its interval covers 90% and claims 95%, and a fresh sample of the same random size covers 95.4%.
A step that is not a ratio
Run the separation sweep on a tuning list of integers rather than a geometric ladder and the two factors still point opposite ways. The invariant does not survive: along a row of integers the probability moves by 2.163 where along the geometric ladder it moves by 1.208.
A length read off the whole sequence
The automatic block length reads one number off the sample — the lag-one correlation — and assumes the rest of the dependence is geometric. Reading the whole autocovariance sequence instead removes the assumption and puts the length where the law wants it. At a hundred and twenty rows it buys nothing, even where the assumption is wrong; it costs 3.87 points where the assumption is right, and by 960 rows that premium is 1.15 and the payout under a law the plug-in misreads is 3.77. Under long memory neither rule, nor the best length on each draw, gets within ninety per cent.
Two values the criterion cannot tell apart
Fifteen candidates disagree about a whitening's order on two draws in five when the list holds all thirteen orders — and on more than one in five when it holds only one and two. Every adjacent pair of positive orders disagrees about as often as that pair does, and every pair holding an order the criterion never picks disagrees on none. A list's disagreement is set by the pairs of values on it that the criterion cannot tell apart, not by how many values it has, and one candidate's own doubt about its pick predicts all fifteen candidates' disagreement with a rank correlation of 0.96.
What a longer sample settles
At a hundred and twenty rows, letting each candidate choose its own whitening changes which of fifteen candidates wins on about an eighth of draws, whatever the list, the table or the law. At four hundred and eighty rows it changes the winner on 1.0% of draws for a window and 2.3% for an order, and at nine hundred and sixty on a quarter of one per cent. The sample settles it through one factor only: how often the candidates disagree about the tuning value hardly moves, and the share of their disagreements that change the winner collapses.
Right for the wrong reason
A robust standard error costs no coverage where the risk is absent — 95.52% against 95.06% at twenty rows. It costs a 6.89% wider interval and a variance estimate 2.572 times as variable, and the pre-test that would avoid paying recovers 15.9% of what the insurance is worth.
The cut that fitted best
A trial that adjusts for its baseline cut at whichever of the sample's nine deciles fits the outcomes best — without ever looking at the treatment difference — reports a variance 0.671 of the unadjusted analysis's where its estimates actually have 0.736, which is worse than the 0.700 a median fixed in advance delivers. The sample says the chosen cut kept 72.4% of what the baseline could remove; the population says it kept less than a median. A true null is rejected 5.87% of the time at fifty patients an arm and 9.31% at ten, and a cut chosen for its p-value rejects 14.22%.
How much of the forbidden spread a rule may read
A stopping rule that reads the spread its interval will report breaks the interval. Blend that spread into what the rule reads, a share at a time, and the cost is not proportional to the share — at a tight requirement a rule can read four fifths of it before anything is lost, and the last fifth costs nearly four points.
An interval that knows the band was maximised
A regression whose coefficients and eight-lag error band are maximised together prints an interval that covers 72% to 83% where 95% was promised, because it reports nearly the precision of a known covariance and delivers less. Two repairs were named. A sandwich standard error read off the whitened residuals repairs nothing — 71% to 83% — because the maximisation has already made those residuals look independent. A parametric bootstrap that redraws from the fitted band and maximises the band again on every redraw repairs about half the gap, to 85.5% to 89%, by pricing the maximisation through repeating it. What it cannot price is that the band it redraws from is the maximised one.
The same dial under a second law
On a first-order autoregression, separating the candidates moved the chance that a tuning list changes the winner 2.35 times as far on a whitening window as on a sieve order, and the difference was put down to the step: an integer step is worth a third of a geometric one. Rerun the whole sweep with errors from a five-period moving average and the difference is gone — the probability spans a factor of 5.53 on the window and 5.13 on the order, a ratio of 1.08 — and near the null both lists leave about five and a half candidates within reach, where the autoregression left 3.6 and 1.9. The two factors still point opposite ways under both laws. The step's size was a property of the law the lists were tuned on.
The table decides when the sample settles it
Letting each candidate regression choose its own whitening changes which candidate wins on about an eighth of draws at 120 rows, and a longer sample settles it — for fifteen nested-and-crossed subsets, at 0.25% by 960 rows. A nested sequence of four candidates gets there a sample size sooner, from half the rate. Six candidates that all hold two predictors never get there: at 960 rows their tuning list still changes the winner on 2.3% of draws, nine times the full table's, because a disagreement among candidates of one size goes on deciding the winner as often at 960 rows as at 240.
A quarrel's cost is its width
When fifteen candidate regressions disagree about a whitening's autoregressive order, the disagreement costs in proportion to how far apart the two orders are — 0.0027 of regret for every order of distance, a line that explains 96% of the cost across sixty-six pairs — and not at all by which orders they are: a quarrel between eleven and twelve costs what one between one and two costs, though one of those holds the truth. Fitted on pairs, the line predicts every longer list's cost. The window's quarrels follow no such line.
A charge that reads the draw
Three charges built to read the sample track the best band width on their own draw at −0.012, −0.019 and −0.041, deliver more error than the fixed rule they are calibrated to, and pick a width half again as variable. The statistic moves; the answer does not.
Intervals for the findings
Benjamini–Hochberg's findings usually go out each with its ordinary 95% interval. With ten real effects of two standard errors among twenty tests, 11.59% of those intervals miss their effect, every miss on the far side, and the interval around the most prominent finding covers 72.36% of the time — 2.38% when the effects are one standard error. Intervals widened for the number of findings hold the share that miss under 5%.
How slow a return a sample can see
At two hundred observations the test finds a gap that halves in five steps four times in five, one that halves in eight 37.3% of the time, and one that halves in fifty 4.95% of the time — which is the rate at which it finds pairs with no mechanism at all. The boundary moves with the sample, not with its square root.
The surface under every law
A draw-reading charge for a covariance band failed on one law, and the failure was deferred to a law with a steeper error surface. Run under all four laws in the collection, the surface is flat from six lags to thirty under every one, no draw-reading charge tracks anything, and long memory — the law expected to be steepest — is the flattest end to end. What the law moves is the level of the charge, not whether a draw can be read.
The height at the chosen setting
The height a response-surface fit predicts at the setting it recommends reads high, because the setting was chosen where the fit was highest. The obvious repair — choose on some runs, estimate on the rest — is impossible in the usual thirteen-run design, which has nine distinct settings where two separate fits need twelve. A parametric bootstrap of the optimism removes four fifths of it at no cost in runs, 0.884 down to 0.183 at twice the noise. With the design run twice, correcting the full fit beats splitting it: an error of 0.924 against 1.213, at a better setting.
Testing for a flat point first
Test whether the gradient is zero, and if the test does not reject, read the interval off the second-order law that holds at a stationary point. For a squared mean that two-stage rule covers about 50% at its worst whatever the pretest's level — 49.72% with a 5% test, where the tangent interval it was meant to rescue never falls below 85.98% — because its worst sits exactly where the true distance equals the test's critical value, half the samples are declared flat there, and the second-order interval cannot reach the truth from any of them.
A correction built on a retest
Tweedie's correction for regression to the mean needs the error variance, and a study gets it from a few people measured twice. Twenty of them are enough: with normal errors the estimated correction then beats the correlation's rule on 73% to 77% of studies of a thousand readings from a heavy-tailed population, though its variance is uncertain by a third. What twenty cannot reveal is the error's shape. Laplace errors of the same variance make the correction overstate the lead a top group keeps by about 0.17, the plain rule then wins on most studies, and a study needs about two hundred retest pairs before a check on its own differences notices.
An interval that knows its finding was found
An interval for one Benjamini–Hochberg finding can be read from the distribution of its estimate given that it cleared the threshold it had to clear. It covers the finding's effect 94.8% of the time with real effects of two standard errors, against the ordinary interval's 87.0%, and it covers the most prominent finding 91.3% of the time where the ordinary one covers 72.4%. The price is width — a median 5.06 standard errors against 3.92 — and something larger: for three findings in four at that effect size the interval reaches back past zero, so a finding that cleared the line is, once the line is accounted for, not distinguishable from nothing.
Named alongside it
The objects these essays reach for when they reach for this one.
Monte CarloInformation criterionModel selectionDependenceRegretTuning parameterSpecification searchWhiteningOverfittingStructural breakBandwidth selectionLikelihood ratio