Normal Distribution
nothing is called 95% until it has been counted
Concept

Measurement error — where it appears

The part of a recorded value that belongs to the occasion of measuring rather than to the thing measured. It averages away over many readings of one subject, and when subjects are selected on a single noisy reading it is the reason the group's next reading falls back towards the mean.

Named by 10 essays across 3 fields — each of them below, with the objects they name alongside it.

Two measurements of the same thing, correlated 0.60. Pick the worst 15% on the first measurement and their average rises by 0.78 on the second. Pick the best and theirs falls by 0.48. No treatment was given to anybody.

Regression to the mean

Select the worst performers, measure them again, and they improve. Select the best and they decline. No intervention is required for either, the size of the apparent effect is predictable from the correlation alone, and it is the reason so many things appear to work.

paradox · Rtm Two groups, a baseline and a follow-up, and nothing happening in between — baseline reliability 0.6. 600 units in two pre-existing groups whose true means are 1.00 apart, read once at baseline and once at follow-up, with no change for anybody. The two groups' mean changes are −0.075 and −0.032, so the change-score analysis reports a group difference of 0.043. The regression of follow-up on baseline and group reports 0.409, against a closed form of (1 − λ) × 1.00 = 0.400: at any one baseline reading the two groups' lines sit that far apart, because each group's units regress towards their own group's mean. The pooled slope in this sample is 0.614, the baseline's reliability.

Two analyses of one baseline

Two groups read at baseline and again at follow-up, with no change for anybody. Subtracting the baseline reports a group difference of −0.0014 and adjusting for it reports 0.4008 — and each analysis is exactly right about one reason the groups started apart and wrong by 0.40 about the other.

paradox · Rtm A cohort screened once, the top tenth enrolled, and followed up with nothing given — correlation 0.6. 2000 people read once at screening and once at follow-up, with a test–retest correlation of 0.6 and no treatment. The 215 above a cut at the top ten per cent of one reading (1.282 standard deviations) are enrolled. Their mean screening reading is 1.744 and their mean follow-up reading 0.982, a fall of 0.762 ± 0.055 with nothing done to anyone. The closed form for the fall is (1 − ρ) times the truncated-normal mean, (1 − 0.6) × 1.755 = 0.702.

The measurement that got them enrolled

Enrol the top tenth of one screening reading and give them nothing, and they fall by 0.702 standard deviations at follow-up. Measured from a fresh reading taken after enrolment they fall by nothing. Averaging ten screening readings still leaves 0.101, and it takes twenty-one to get under 0.05.

paradox · Rtm What balancing a recorded number does to the true one. The imbalance three rules leave in the true covariate X when what they read is a recording W of it, at six reliabilities and n = 200, 600 trials each, as a share of a coin's exact 2/√n. The lower curve is √(1 − λ), the closed-form floor for any rule that reads W: X = λW plus a residual of variance 1 − λ that W says nothing about, and a coin is all any rule can be on that residual. The rule that reads the number sits on it — 0.125 at 1, 0.345 at 0.9, 0.452 at 0.8, 0.612 at 0.64, 0.718 at 0.5, 0.834 at 0.3 — while balancing W itself to 0.125–0.133 of a coin's throughout. The upper curve is √(1 − 2λ/π), the same floor for minimisation on a median split of W, and minimisation sits on that one. At λ = 2/π the rule that reads the number is exactly as good as a median split of an error-free covariate: a cut is a measurement with reliability 0.637.

A number read with error

A rule that balances the recorded covariate balances the true one only as far as the recording reaches. At a reliability of 0.8 it leaves 45% of a coin's imbalance in the covariate that drives the outcome, back on the coin's 1/√n rate — and a median split turns out to be a recording with reliability 2/π.

continuous · Assignment The top of a heavy-tailed population keeps its lead; the top of a light-tailed one gives it back. Select the top share on the first reading and read the group again: the share of its mean lead the second reading keeps, by integration over the true score (lines) and counted on 400,000 draws a parent in 20 batches (points, with two standard errors). The normal keeps exactly 0.6 at every selection. At the top half the Laplace keeps 0.541, the t 0.535 and the uniform 0.648 — the heavy tails keep LESS than the correlation. By the top one per cent the order has reversed: 0.761, 0.784 and 0.443. At one in ten thousand the t keeps 0.977 and the uniform 0.346.

A lead that a heavy tail keeps

Four populations whose readings all correlate at exactly 0.6, and whose least-squares slopes all read 0.6. Select the top one per cent on one reading and measure them again: they keep 60% of their lead if the true scores are normal, 76.1% if they are Laplace, 78.4% if they are a t on four degrees of freedom — and 44.3% if they are uniform. The correlation predicts the regression of the extremes for one shape of population only.

paradox · Rtm Four rules and what each leaves. The share of a coin's imbalance each rule leaves in the true value of each covariate, at 200 units and a reliability of 0.8. one reading of X: 21.4% in X and 100.1% in Z; two readings of X: 12.3% in X and 105.0% in Z; one reading each of X and Z: 21.7% in X and 21.5% in Z; two of X and one of Z: 13.4% in X and 22.0% in Z. The floors in closed form are 20.0% for one reading and 11.1% for two.

A second reading or a second covariate

A rule that balances a recording balances the truth only as far as the recording reaches, and a second reading of the covariate raises that reach. So does a first reading of something else. At a reliability of 0.8 the first reading of a second covariate does more once that covariate matters a third as much as the first, and the threshold is exact: the square root of (1 − λ)/(1 + λ), with no trial size in it. A protocol's list of measurements turns out to be a small knapsack, and it is rarely filled by reading anything twice.

continuous · Assignment Twenty studies of a thousand readings estimate the regression of the extremes: the t, four degrees parent. Each thin line is one study of 1000 readings from the t, four degrees parent: Tweedie's formula with the log-density's slope estimated by a degree-5 log-spline, drawn up to that study's largest reading. The thick line is the share kept by integration over the true score, and the dashed line the correlation, 0.6. The top ten readings of the median study begin at 2.45. Over 400 studies the corrected share kept by the top one per cent averages 0.8190, with a spread of 0.0806, against 0.7842 by integration; a Gaussian kernel averages 0.7914 with a spread of 0.0853.

The slope of a density nobody can see

Tweedie's formula corrects a reading by the slope of the readings' own log-density, and a study has its readings. Estimated from a thousand of them, the correction for the top one per cent beats the correlation's linear rule on 84.0% to 98.0% of studies from heavy-tailed populations and on 75.0% to 81.5% from a bounded one — and costs an error of 0.09 to 0.12 where the population is normal and the rule was already exact. At 250 readings the log-spline loses to the rule it replaces, and at 16,000 the same log-spline gets worse on a power tail.

paradox · Rtm A proxy removes less than its reliability, always. The share of the confounding bias removed by adjusting for a proxy, against how well the proxy measures the confounder. The diagonal is the answer a reader would guess — a covariate that is 80% signal removes 80% of the problem. The curve is what the arithmetic gives: the reliability, times one minus the squared correlation between the treatment and the confounder, divided by one minus the product of those two. That squared correlation is 0.4475. A reliability of 0.8 removes 68.85% and one of 0.6 removes 45.32%. The two agree only at the ends, and the gap is widest where most applied covariates sit.

Adjusting for a shadow

A covariate that is 80% signal removes 68.85% of the confounding, not 80% — the share is λ(1 − ρ²)/(1 − λρ²) and it is below the reliability everywhere. The residual bias is 0.1084 against an effect of 0.5, and at 25,600 rows it is 17.6 standard errors wide.

collider · Conditioning Three adjustment rules on four thousand structures, as every covariate gets noisier. Four thousand random structures of six covariates, the treatment's effect fixed at 0.5, every covariate observed with the same reliability. Mean absolute bias of adjusting for nothing stays 0.190; adjusting for everything measured falls from 0.377 with exact covariates to 0.279 at reliability 0.8 and 0.203 at 0.5; adjusting for the common causes rises from 0 to 0.056 and 0.118. Adjusting for everything beats adjusting for nothing on 33.8% at 1, 38.3% at 0.9, 42.0% at 0.8, 45.3% at 0.7, 48.3% at 0.6, 50.8% at 0.5. The common causes never do worse than nothing at any common reliability.

A set of shadows

Measure every covariate of four thousand random structures with reliability 0.8 and adjusting for the common causes still never does worse than adjusting for nothing — the bias left is each confounder's part times one minus its proxy's reliability, scaled alike. Adjusting for everything measured gets better, beating nothing on 42.0% of structures against 33.8% with exact covariates, because noise blunts the bad adjustments too. Spread the reliabilities to 0.75 ± 0.25 and the right set does harm on 7.15% of structures, every one of them with confounders pushing opposite ways.

collider · Conditioning How many people a study has to retest before its estimated correction is worth having, normal errors. Root mean squared error, over 400 studies of a thousand readings at each retest size, of the share of the top one per cent's lead a second reading keeps: Tweedie's correction with a kernel log-slope and the error variance estimated from the retested people (solid), and the linear rule at the reliability those people estimate (dashed). Laplace parent: 0.121, 0.108, 0.084, 0.086, 0.084, 0.078 against 0.343, 0.259, 0.201, 0.183, 0.171, 0.164. T, four degrees parent: 0.133, 0.103, 0.086, 0.084, 0.080, 0.078 against 0.358, 0.278, 0.235, 0.212, 0.205, 0.187. Uniform parent: 0.247, 0.192, 0.156, 0.140, 0.118, 0.117 against 0.260, 0.208, 0.185, 0.166, 0.160, 0.157 — at 10, 20, 50, 100, 200, 1000 retested.

A correction built on a retest

Tweedie's correction for regression to the mean needs the error variance, and a study gets it from a few people measured twice. Twenty of them are enough: with normal errors the estimated correction then beats the correlation's rule on 73% to 77% of studies of a thousand readings from a heavy-tailed population, though its variance is uncertain by a third. What twenty cannot reveal is the error's shape. Laplace errors of the same variance make the correction overstate the lead a top group keeps by about 0.17, the plain rule then wins on most studies, and a study needs about two hundred retest pairs before a check on its own differences notices.

paradox · Rtm

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloClosed formRegression to the meanTest–retest reliabilityCovariate adjustmentSelection biasHeavy tailTweedie's formulaAdjustment setBiasCausal diagramConfounding

All concepts