Normal Distribution
nothing is called 95% until it has been counted
Concept

Selection bias — where it appears

The distortion introduced by having chosen what to look at from the same data being reported. It has the same shape whether the choosing was a model, a subgroup or an extreme observation, and its size is the amount that made the choice.

Named by 27 essays across 16 fields — each of them below, with the objects they name alongside it.

The same study read three ways. At time 2 the truth is 0.497. Kaplan–Meier gives 0.532; dropping the censored subjects gives 0.180; treating the censoring time as the event time gives 0.392. Both naive readings understate survival, because the subjects they mishandle are the ones doing well.

The data that stops early

A subject still event-free when a study ends is not missing and not observed. It is known to exceed something, which is a third state most tools have no slot for — and the two obvious ways of forcing it into one are wrong by 31 and 13 percentage points.

survival · Censoring Three mechanisms leave the slope alone; one does not. The bias of the complete-case slope under each of four missingness rules, counted over 4000 studies of 200 rows at 35.0% missing, with the closed form printed beside each count. Missingness that depends on nothing, on the regressor, or on the second covariate leaves the slope exactly where it was — the closed forms are zero to machine precision and the counts are -0.0005, -0.0005 and -0.0011 against standard errors of about 0.0018. Missingness that depends on the outcome moves it by -0.1635, which is 27.3% of the slope being estimated. The same share of rows is lost in every case.

Three mechanisms and one dataset

Four rules for which outcomes go missing, each calibrated to lose the same 35% of the rows and each leaning on what it reads with the same coefficient. Three leave the fitted slope exactly where it was, and the one that reads the outcome moves it by 0.163531.

missing · Missingness What a rule gives away by being predictable. Minimisation run at every probability from a coin to fully deterministic, over 500 cohorts of 120 at each. The upper line is the share of assignments an investigator who knows the rule and the enrolled patients can name in advance: 49.7% at p = 0.5, which is a coin and cannot be beaten, and 87.6% at p = 1 — short of everything only where the two arms tie and the rule falls back on a coin. The lower line is what that is worth: an investigator who enrols a patient 0.5 of a standard deviation better than average whenever they predict their favoured arm produces a treatment effect of 0.75 where the truth is zero. Nothing about the randomisation was broken; the bias entered through who was enrolled, which is the one thing an allocation rule cannot control. The dashed line is the closed form 2δ(2g − 1).

The rule that can be guessed

A balancing rule improves as it becomes more deterministic, and a deterministic rule can be worked out in advance from information the person enrolling the patient already has. At full determinism 87.6% of assignments are guessable, and an investigator who acts on the guess produces a treatment effect of three quarters of a standard deviation where the truth is zero.

covadapt · Assignment Unrepresentative in every respect but the one that matters. Three properties of the complete cases as the chance of being observed leans harder on the regressor, in closed form, at 35.0% of outcomes missing throughout. The mean of the regressor among the rows kept climbs from 0.0000 to 0.5528 against a population mean of zero, and the mean of the outcome from 0.0000 to 0.3980 above its own. The bias in the fitted slope is exactly zero at every one of the ten settings, because selection acting on the regressor alone leaves the conditional law of the outcome given the regressor untouched and least squares conditions on exactly that. The sample is wrong about almost everything and right about the one quantity being estimated.

Dropping the incomplete rows

Push the missingness until the rows that survive have a covariate mean of 0.543905 against a population zero and a variance of 0.5041 against one, and the fitted slope is still exactly right. Where the rule reads the outcome instead, the same sweep takes coverage to 2.42% at eight hundred rows.

missing · Missingness The best of 8 arms, tested as though it were the only one. 8,000 trials with no effect in any arm. Stage one runs 8 arms at 60 each, the best is carried forward, and stage two adds 30 more to it and to the control. The histogram is where the final statistic lands and the curve is the standard normal it is being read against — shifted right, because the arm was chosen for being ahead. 10.5% of these trials clear 1.96 against a claimed 5%, and the value that actually holds the rate for this design is 2.313.

Dropping the losers

Carrying the best of eight arms forward and testing it at 1.96 rejects a true null 10.3% of the time — the hypothesis was chosen by looking at the data, so the statistic is a maximum wearing a single comparison's clothes. The value that holds the rate is 2.313, and it has to be solved for.

adaptive · Stopping Two measurements of the same thing, correlated 0.60. Pick the worst 15% on the first measurement and their average rises by 0.78 on the second. Pick the best and theirs falls by 0.48. No treatment was given to anybody.

Regression to the mean

Select the worst performers, measure them again, and they improve. Select the best and they decline. No intervention is required for either, the size of the apparent effect is predictable from the correlation alone, and it is the reason so many things appear to work.

paradox · Rtm Forty O'Brien–Fleming trials at a true effect of 0.16, with the boundary written as an effect. The dashed line is the smallest effect a trial can report and still stop at each look: 0.510 at 80 observations, 0.255 at 160 observations, 0.170 at 240 observations, 0.128 at 320 observations, 0.102 at 400 observations. The true effect is 0.16, so at 3 of the five looks a trial cannot stop without reporting more than it. 29 of these forty trials stop before the last look, each marked where it stopped.

The effect a stopped trial reports

An O'Brien–Fleming trial at 88.45% power holds its error rate exactly and reports an effect 9.6% too large on average. The 11.39% of trials that stop at the second look report 1.83 times the truth, the ones that cross at the last look report 0.80 times it, and pooling every trial by its size gives the truth back to the last digit.

sequential · Stopping What the family-wise correction does to the effect it lets through. At two standard errors the estimate that clears an uncorrected 5% threshold averages 1.35 times the truth, and the one that clears the family-wise threshold averages 1.69 times it. The correction fixes the error rate by demanding a larger estimate, and a larger estimate is a more selected one.

The correction that makes the estimate worse

Correcting for twenty analyses repairs the p-value by demanding a larger statistic, and a larger statistic is a more selected one. At two standard errors the surviving estimate averages 1.35 times the truth before the correction and 1.69 times it after — so the honest error rate is bought with a more inflated effect.

paths · Forking What a guesser gets, and what a guesser gets for nothing. 600 trials of 150 patients under a fully deterministic rule, with an investigator who knows the rule, the factors and every assignment so far. The guess rate falls with the number of arms — 87.8% at 2, 86.2% at 3, 81.0% at 4 — which reads like a trial getting safer and is not: what a guesser can trade on is the excess over the 50%, 33%, 25% they would get by naming an arm at random, and that goes the other way, from 1.76× chance at two arms to 3.24× at 4. The gap a guesser manufactures between the best and worst arm under a true null is 0.759, 0.733, 0.711 standard deviations — nearly unchanged.

Guessing one arm in three

A balancing rule is guessable because it is balancing. With three arms the next assignment is worked out less often than with two — and by more, relative to what a guesser gets for nothing, and the damage they can do is almost unchanged.

multiarm · Assignment Three estimates of the same effect, and the honest one is the worst. 8,000 trials in which every arm has an effect of exactly 0. The arm carried forward is reported by its first stage at +0.187 above the truth — it was chosen for being ahead — and by its second stage at -0.0032, which is unbiased by construction because the selection could not see it. The combination that gets published is at 0.123, exactly the share of the first stage's bias the first stage contributes. Root mean squared error: 0.240, 0.261, 0.182 — the unbiased estimate is the least accurate of the three.

The estimate after the choice

An arm chosen for being ahead is ahead by more than it should be, and the trial then publishes the average of the stage that chose it and the stage that did not. The unbiased estimate is the one built from a third of the data — and it is the least accurate of the three.

adaptive · Curse What the interval covers once the order is chosen as well. 1200 series of 40 observations from an AR(1) with φ = 0.7, at each horizon, on one set of seeds. The upper line is the interval computed at the true parameters — it covers 95.5% on average, which is the check that σ²Σψ² is the right formula rather than a claim about anything a forecaster can do. The lower line is the same formula fed σ̂² and φ̂: 93.6% at one step and 90.8% at 6. The third line chooses the order by AIC from the same data before computing the interval, which costs a further 0.8 points at h = 6.

The interval after the choice

Estimating the coefficients of a known model costs a 95% forecast interval about two points of coverage. Choosing which coefficients to estimate, from the same forty observations, costs another four and a half — so the step nobody records in the output is the more expensive of the two.

forecast · Order selection What the interval covers, after a design that read the data. 800 experiments of 12 runs, all at the same truth. The first pair is a design fixed in advance; the second is one whose settings were chosen from the first stage's own outcomes. If choosing the design from the data broke the inference, the second pair would sit below the first, and it does not — 91.3% against 92.6%. What does move the coverage is the shape of the interval rather than the design: the Wald interval assumes the estimate is normal around its own standard error and falls short under both designs, and the profile interval, computed from the same residual sums of squares with no derivative in it, covers 94.3% and 94.9%.

What a design chosen from the data costs

Two earlier measurements found what happens when a rule reads the data, and the error rate broke both times. A design that reads the data to decide where to put its runs breaks nothing — and the control that proves it also finds what the real shortfall is.

robust · Local design Four numbers, and only one of them is the question. Four quantities in a population where the effect is not the same for everybody: 40.0% compliers, 25.0% always-takers and 35.0% never-takers, with the compliers carrying an effect 1.0000 larger than everybody else's. The population average effect is 0.5000. The compliers' average effect is 1.1000. A valid instrument converges on 1.1000 — the second of those, not the first, and the two differ by 0.6000. Comparing the treated with the untreated as they stand gives 1.4182, wrong by 0.9182 in the same direction, because always-takers start 1.5000 above never-takers before any treatment happens. The instrument removes the selection and changes the question at the same time, and only one of those is reported.

Whose effect it is

With a perfectly valid instrument and no violation of anything, the estimate converges on 1.1000 where the population average effect is 0.5000. The gap is exactly θ(1 − p_c), the always-takers and never-takers cancel out of both halves of the ratio, and five per cent defiers move the answer to 1.2667.

instrument · Exclusion The ratio of the arms' hazards over time, when treatment halves every patient's hazard. Every patient's hazard ratio is 0.5. With two risk groups — half at 0.1 a year, half at 1 — the ratio of the arms' hazards rises from 0.5 to 0.929 at 3.5 years and falls back towards 0.5. With a gamma frailty of variance 1 it rises steadily, to 0.857 at ten years.

Every patient at one half

A randomised trial in which the treatment halves every patient's hazard, and patients differ only in their baseline risk. The unadjusted Cox hazard ratio is 0.524 after six months of follow-up, 0.635 after three years and 0.679 after seven, and the ratio of the arms' hazards at a single moment climbs from 0.50 to 0.929 and falls back. Nothing is confounded and nothing is wrong. Survival removes high-risk patients from the control arm faster, and a hazard ratio compares whoever is left.

reversal · Simpson The line is the sample, and the sample is the finding. 900 draws of two independent standard normal causes, with the 453 of them past a threshold of 0.00 marked and the 447 that fall short left pale. In the population the two are independent by construction. Inside the selected sample the correlation is -0.4669 in closed form and -0.5050 counted on these 453 rows, and the least-squares line through them has a slope of -0.545. The mechanism is visible in the picture rather than argued: the threshold removes one corner of the cloud, and a cloud with a corner missing is a cloud whose two coordinates carry information about each other.

The sample is a condition

Two independent standard normals, selected on their sum exceeding its median, read a correlation of exactly −1/(π − 1) = −0.4669 inside the sample. Nothing is measured badly and nothing is missing — and both halves of that split read it, in the same direction, while the population containing both reads zero.

collider · Conditioning A cohort screened once, the top tenth enrolled, and followed up with nothing given — correlation 0.6. 2000 people read once at screening and once at follow-up, with a test–retest correlation of 0.6 and no treatment. The 215 above a cut at the top ten per cent of one reading (1.282 standard deviations) are enrolled. Their mean screening reading is 1.744 and their mean follow-up reading 0.982, a fall of 0.762 ± 0.055 with nothing done to anyone. The closed form for the fall is (1 − ρ) times the truncated-normal mean, (1 − 0.6) × 1.755 = 0.702.

The measurement that got them enrolled

Enrol the top tenth of one screening reading and give them nothing, and they fall by 0.702 standard deviations at follow-up. Measured from a fresh reading taken after enrolment they fall by nothing. Averaging ten screening readings still leaves 0.101, and it takes twenty-one to get under 0.05.

paradox · Rtm Twenty runs simulating an exactly 95% interval, checked every 250 replications. Each line is one run's running estimate; the dashed band is where the Wilson interval of the running estimate still contains 95%, and a run stops, marked, the first time it leaves the band. 8 of these twenty stop before 10,000 replications. The exact probability of stopping, from the recursion over the count, is 29.54%.

A simulation that stops when it looks settled

A simulation of an interval that covers exactly 95%, checked every 250 replications for a significant departure and stopped when it finds one, flags that correct interval on 29.54% of runs. Stopped instead as soon as its estimate reaches 95%, it reports an interval that covers 94% as meeting its level on 37.21% of runs. Stopped when the estimate stops moving, it reports the right number — and has quietly chosen to run about fifteen hundred replications.

method · Seeds How far three estimates of the chosen arm sit from its true effect, with all eight alike. Eight arms of sixty, the best carried forward with thirty more. Average error: both stages, as published +0.123; stage two alone −0.001; stage one shrunk, then both +0.020. The arms' spread estimates to zero on 57.1% of trials.

Shrinking the arm that was chosen

A trial that carries the best of eight arms forward reports it as better than it is — by +0.123 when the eight arms are alike — and its second stage alone is unbiased but the least accurate of the estimates. Pulling the chosen arm's first stage back towards the eight arms' mean before combining removes most of the bias when the arms are alike (+0.020) and gives the smallest error of any estimate, 0.133 against the published 0.182. When one arm really is ahead by half a standard deviation, the same shrinkage pulls the true winner back towards the losers and understates it by 0.103, with more error than the published estimate. The weight is estimated from eight numbers, and it is right when the selection was noise and wrong when the selection was right.

adaptive · Curse What a guesser gets from less than everything. 1,000 three-arm trials of 150 patients under minimisation on three factors with the preferred arm taken with probability 1, and five investigators guessing each next assignment from different information. Knowing the rule, the factors and every assignment so far: 86.3%. Knowing everything about the patients at their own centre and nothing about the others, and applying the rule's own score to that: 52.8%. Knowing only how many of their own centre's patients went to each arm: 55.5%. Knowing only the trial's arm totals, which a drug-supply system shows anyone who reorders: 54.5%. A guess at random gets 33.3%. The cruder the information, the less the guesser gets — except that the rule's own score, applied to a partial record, does worse than simply counting.

A guesser who sees the totals

The bound on how guessable a three-arm minimisation is assumes an investigator who sees everything. One who sees only the arm totals — what a drug-supply screen shows — names the next arm 54.5% of the time against a chance 33.3%, and manufactures 45% of the full guesser's bias.

multiarm · Assignment The top of a heavy-tailed population keeps its lead; the top of a light-tailed one gives it back. Select the top share on the first reading and read the group again: the share of its mean lead the second reading keeps, by integration over the true score (lines) and counted on 400,000 draws a parent in 20 batches (points, with two standard errors). The normal keeps exactly 0.6 at every selection. At the top half the Laplace keeps 0.541, the t 0.535 and the uniform 0.648 — the heavy tails keep LESS than the correlation. By the top one per cent the order has reversed: 0.761, 0.784 and 0.443. At one in ten thousand the t keeps 0.977 and the uniform 0.346.

A lead that a heavy tail keeps

Four populations whose readings all correlate at exactly 0.6, and whose least-squares slopes all read 0.6. Select the top one per cent on one reading and measure them again: they keep 60% of their lead if the true scores are normal, 76.1% if they are Laplace, 78.4% if they are a t on four degrees of freedom — and 44.3% if they are uniform. The correlation predicts the regression of the extremes for one shape of population only.

paradox · Rtm How far four estimates of the chosen arm sit from its true effect, with one arm ahead by 0.5. Eight arms of sixty, the best carried forward with thirty more, 20,000 trials. Average error: both stages, as published +0.006; shrunk under a normal prior −0.103; shrunk under a t prior, 3 df −0.101; normal prior with a fifth vague −0.073. Root-mean-square error: 0.151, 0.189, 0.192, 0.176.

A prior that lets one arm stand out

Shrinking the chosen arm of eight under a normal prior understates a genuine winner by 0.103. A t prior on three degrees of freedom with the same spread understates it by 0.101 — the tail is too far out to reach a lead of three and a half standard errors. A prior with a fifth of its weight on a vague component understates it by 0.073, costs 0.010 of bias when the arms are alike, and still has more error than the unshrunk estimate. With sixteen arms the same mixture recovers three fifths of the loss, because a bell of fifteen shows its outlier; with eight, the arms put more than half their posterior weight on 'one stands out' in 0.4% of trials where one does.

adaptive · Curse Sixty-three subsets of six functions. The largest share of a coin's imbalance left in any of four shapes, measured on fresh trials, for each of the 63 subsets of six functions at 100 units. Subsets of one size spread widely: at four functions from 12.1% to 72.9%. The designer's order of addition gives 1: 94.4%, 2: 41.2%, 3: 39.0%, 4: 31.5%, 5: 27.0%, 6: 22.7%, against the best chosen subset of each size at 1: 83.6%, 2: 41.2%, 3: 25.4%, 4: 12.1%, 5: 16.2%, 6: 22.7%.

The list it was chosen on

Six functions of one covariate looked like 720 designs, one per order of addition. The rule reads a span and not a sequence, so they are 63, and at a hundred units the best four leave 12.1% of a coin's imbalance against the designer's first four at 31.5%. Those best four contain no cube and are, function for function, the four shapes the worst case was scored on. Read on six shapes it was not told about, the same design leaves the cube 77.0% of a coin's imbalance, and the designer's four, which lost by a factor of two and a half, leave it 28.1%.

shape · Criterion The guess rate through a trial. Guess rates in each tenth of a trial of 150 patients on three arms with the rule taking its preferred arm with probability 0.8, over 1500 trials. knowing everything: 65.9%, 72.2%, 72.0%, 72.8%, 72.4%, 72.2%, 72.5%, 72.7%, 72.7%, 72.4%. a joint table learned in earlier trials: 53.4%, 55.4%, 54.8%, 55.1%, 54.3%, 53.8%, 54.9%, 54.8%, 54.1%, 54.8%. the centre's totals: 45.4%, 50.3%, 50.2%, 50.8%, 49.8%, 50.2%, 50.9%, 50.6%, 50.4%, 50.7%. following whichever has been right: 46.5%, 50.2%, 49.3%, 49.6%, 48.9%, 48.7%, 49.3%, 49.0%, 49.0%, 48.7%. a joint table learned in this trial: 36.4%, 43.8%, 45.0%, 46.2%, 46.7%, 46.8%, 48.1%, 48.5%, 47.9%, 48.9%. the centre's own factors, scored: 46.8%, 51.4%, 49.7%, 49.9%, 48.4%, 48.6%, 48.5%, 47.7%, 47.6%, 46.7%. Chance is 33.3%.

The guesser who learns

An investigator who watches a three-arm minimisation can tally which of several partial readings has been right so far and follow the leader. It does not help. Following the leader names 48.9% of assignments where the best single reading names 49.9%, and a table of what the three readings say together, learned inside the trial, starts at 36% and has not caught up by a hundred and fifty patients. Learned over earlier trials, that table reaches 54.5%. The full guesser's 71.8% stays out of reach of all of them, because the gap to it is information rather than practice.

multiarm · Assignment Twenty studies of a thousand readings estimate the regression of the extremes: the t, four degrees parent. Each thin line is one study of 1000 readings from the t, four degrees parent: Tweedie's formula with the log-density's slope estimated by a degree-5 log-spline, drawn up to that study's largest reading. The thick line is the share kept by integration over the true score, and the dashed line the correlation, 0.6. The top ten readings of the median study begin at 2.45. Over 400 studies the corrected share kept by the top one per cent averages 0.8190, with a spread of 0.0806, against 0.7842 by integration; a Gaussian kernel averages 0.7914 with a spread of 0.0853.

The slope of a density nobody can see

Tweedie's formula corrects a reading by the slope of the readings' own log-density, and a study has its readings. Estimated from a thousand of them, the correction for the top one per cent beats the correlation's linear rule on 84.0% to 98.0% of studies from heavy-tailed populations and on 75.0% to 81.5% from a bounded one — and costs an error of 0.09 to 0.12 where the population is normal and the rule was already exact. At 250 readings the log-spline loses to the rule it replaces, and at 16,000 the same log-spline gets worse on a power tail.

paradox · Rtm What a confirmation run at the chosen setting would find. The true optimum is worth 62.348. At σ = 2 the fit predicts 62.679 at the setting it recommends and the truth there is 61.821 — a gap of 0.858, which is 0.72 of the prediction's own standard error. The setting itself gives up 0.527 against the best available.

The run that confirms it

The setting a response-surface analysis recommends was chosen because the fitted surface was highest there, so the height the fit predicts at it is a maximum over a random field. At twice the noise the fit predicts 0.858 more than is there — 0.72 of the prediction's own standard error — and the gap is not noise, it is the selection.

surface · Optimum The error of the chosen arm's estimate against a fixed second stage's size, and a second stage sized by the interim, 8 arms, one ahead by half a standard deviation. Fixed second stages of 10, 25, 40, 55, 70 patients: both stages combined err by 0.172, 0.155, 0.143, 0.133, 0.126; the mixture-shrunk estimate by 0.208, 0.183, 0.164, 0.151, 0.140. The rule that sizes the second stage by the interim spends 41.4 on average and errs by 0.142 and 0.159; its second stage alone errs by 0.225.

A second stage sized by the first

A drop-the-losers trial can size its second stage by what the first stage says about the arms' shape — more patients when the chosen arm's lead looks like luck, fewer when it looks real. The rule does what it was built to do: at sixteen arms with a genuine winner it spends 51.0 patients a group when the interim chose the wrong arm and 24.2 when it chose the right one. And it buys nothing. Against a fixed second stage of the same average size, every estimate of the chosen arm errs by the same to within a few thousandths; the published combination's bias rises from 0.014 to 0.022, and the second stage alone, still unbiased, errs by 0.319 against 0.281.

adaptive · Curse How far the published estimate of the chosen arm sits from the best arm's effect, by how six hundred patients are split between choosing and estimating. Eight arms and a control, six hundred patients: a stated number an arm in the first stage and the rest split equally between the chosen arm and the control in the second. Root mean squared error of the published estimate about the best arm's true effect, over 5,000 trials a split. All eight alike: best at 16 an arm (0.092), 0.183 at the familiar sixty; One modestly ahead: best at 56 an arm (0.151), 0.153 at the familiar sixty; One clearly ahead: best at 48 an arm (0.146), 0.154 at the familiar sixty; Two clearly ahead: best at 40 an arm (0.128), 0.151 at the familiar sixty; Spread evenly: best at 40 an arm (0.131), 0.146 at the familiar sixty. The split that guards every world puts 32 an arm before the choice and costs at most 21.3% over each world's best.

Patients spent before the choice

A trial that picks the best of eight arms and then confirms it splits its patients between choosing and estimating, and the familiar shape — sixty an arm before the choice, thirty after — spends ninety per cent of a six-hundred-patient budget on the choice. Measured against the effect a reader is told about, the best arm's, that split is nearly right when one arm is modestly ahead and twice as wrong as it needs to be when the arms are alike. A split of thirty-two an arm before the choice is never more than 21.3% worse than the best split for whichever world holds; sixty an arm can be 98.8% worse. The two targets an estimate could serve — the chosen arm's effect and the best arm's — pull the split in opposite directions, and only one of them is what gets published.

adaptive · Curse

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloClosed formThe winner's curseAdaptive designMean squared errorShrinkageCovariate-adaptive randomisationInterim analysisMinimisationRegression to the meanBlindingConditional distribution

All concepts