Normal Distribution
nothing is called 95% until it has been counted
Series

Summary — the series

10 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. Four datasets, slope 0.50, R² 0.67. Every one of these fits reports the same slope to two decimals and the same R². Only the first is a linear relationship with noise: the second is a curve, the third is a line with one outlier, and the fourth has its slope set by a single point.

    Four datasets, one summary

    Four datasets agree on slope, intercept and R² to two decimals. One is a linear relationship, one is a curve, one is a line with an outlier, and one has its slope set by a single point. The summary cannot tell them apart and neither can any other summary.

    part 1 · regression
  2. R² against the number of useless predictors, n = 30. The response is pure noise and so is every predictor, so the true relationship is nothing at all. R² rises from 0.000 to 0.648 anyway, following k/(n − 1) — which is what a criterion that rewards higher R² is actually rewarding.

    R² is not a measure of fit

    Adding a predictor with no relationship to anything cannot reduce R², and in expectation raises it by 1/(n − 1). Twenty useless predictors on thirty points give an R² of 0.69 from pure noise.

    part 2 · regression
  3. One relationship at five designs, residual spread 1.00. Every panel has the same slope of 1, the same intercept of 0 and the same residual standard deviation of 1.00. Only the range of x differs. R-squared runs from 0.021 to 0.849, and the estimated residual spread is 0.9932 in all five.

    R² is a property of the design

    One line, one residual spread, five studies that differ only in how far apart they placed their x values. R² runs from 0.021 to 0.849 and the estimated residual spread is 0.993 in every one of them. Nothing about the relationship changed.

    part 3 · spread
  4. Two studies with an R-squared of 0.85. The left study's points sit 0.50 from the line and the right study's 2.00 — a factor of 4.0. Both report an R-squared of 0.85 and, at the same sample size, the same standard error for the slope. The design was chosen to make it so, and it can always be chosen.

    The t statistic wearing different clothes

    For a simple regression, t² = (n − 2)R²/(1 − R²), exactly, on every dataset — checked to sixteen significant figures over five hundred fits. So a paper reporting R² and a p-value has reported one number twice, and two studies with the same R² have points four times further from the line.

    part 4 · spread
  5. How far apart four summaries put the quartet. Each bar is the largest value minus the smallest across the four datasets. Pearson spans 0.0018, which is the construction working. Distance correlation — the measure that is zero if and only if the variables are independent — spans 0.101, and Spearman spans 0.491.

    The summary that was meant to work

    Distance correlation is zero if and only if two variables are independent, which is exactly the guarantee a correlation coefficient lacks. Run on the four datasets that share a correlation, it spreads them by 0.10 — and Spearman, which guarantees nothing, spreads them by 0.49.

    part 5 · spread
  6. Three summaries of a fitted line's geometry on Anscombe's four datasets, with each rule's threshold. Largest leverage: 0.318, 0.318, 0.318, 1.000 against a threshold of 0.364. Largest Cook's distance: 0.489, 0.808, 1.393, undefined, the point fits itself against 1 and against 0.364. Largest studentised deleted residual: 2.1, 2.2, 1192.2, undefined against a Bonferroni threshold of 3.90.

    Two numbers for the fit's geometry

    No summary of the dependence between x and y can flag Anscombe's third and fourth datasets. A summary of the fit's own geometry can, and it takes two numbers of different kinds: the largest leverage, which reads the design before any y is seen and flags the fourth with certainty, and the largest deleted residual against a Bonferroni threshold, which flags the third at 1,192 and raises a false alarm on 4.78% of honest datasets. The rule of thumb most often quoted, Cook's distance over 4/n, flags the honest line too — and 66.45% of honest datasets like it.

    part 6 · spread
  7. How many of 100 points a fitted slope rests on, by the distribution the covariate was drawn from. The effective number of points behind a slope is n divided by the design's kurtosis. Over designs of 100, its median share is two points, half at each: 1.000; uniform: 0.551; normal: 0.349; exponential: 0.160; Student's t, 3 df: 0.148; lognormal, σ = 1: 0.076 — 34.9 of 100 points for a normal covariate and 7.6 for a lognormal one.

    The points a slope rests on

    Before any outcome is measured, a regression's design already says how many of its points the slope will rest on: n divided by the covariate's kurtosis, exactly. A hundred points of a normal covariate carry a slope on about 35 of them; a hundred of a lognormal one, on 7.6. The largest leverage follows the same tail — above one half in 15.7% of lognormal designs of a hundred and in none of the normal ones — and on a correct straight line, deleting the single highest-leverage point moves the slope by more than its standard error in 16.3% of lognormal datasets of a hundred and 0.1% of normal ones.

    part 7 · spread
  8. A point that decides a coefficient, on two covariates correlated at 0.9. A hundred points on two normal covariates with correlation 0.9. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the first coefficient's information, 14.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 26.1 effective points, with a variance inflation of 6.27.

    A point ordinary on every axis

    In a multiple regression each coefficient rests on its own number of points, n divided by the kurtosis of that covariate's residual after the others, exactly. Correlation between normal covariates inflates a coefficient's variance fivefold and leaves it resting on the same third of the points. What correlation does instead is hide the point that decides the coefficient: at a correlation of 0.9 it stands out on no single covariate's plot in 77.3% of designs of a hundred, and the largest leverage a normal design reaches depends on the number of points and covariates and on nothing else.

    part 8 · spread
  9. A point that decides an interaction, on two covariates correlated at 0. A hundred points on two normal covariates with correlation 0. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the interaction's information, 10.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 19.0 effective points.

    A term built from the others

    A regression's squared terms and interactions are columns built from its covariates, and each coefficient on one rests on the number of points divided by the kurtosis of what the built column adds — fifteen for the square of a normal covariate, nine for the product of two uncorrelated ones, three for a treatment's interaction with a normal covariate, which is the same as the covariate's own coefficient. A design of a hundred rests on more than the population says, 12.2 points behind a square against 6.7, because a hundred draws rarely hold the tail that sets the kurtosis. Centring changes nothing about the square's coefficient, to the last digit. And correlation between two covariates hides the point that decides each one's own coefficient while it reveals the point that decides their interaction.

    part 9 · spread
  10. The weight each of a hundred outcomes carries in the fitted value at the 97.5th percentile of a lognormal covariate. One design of a hundred lognormal covariate values. The fitted value at a covariate value of 7.10 is a weighted sum of the outcomes, with weights fixed by the design. Under a line and its square it rests on 10.7 effective points, the largest carrying 19.1% of its variance. Under a natural spline with knots at 0.17, 0.57, 1.18, 3.95 it rests on 3.8, the largest carrying 41.1%.

    A value read off a curve

    No coefficient of a spline is read; what is read is the fitted curve's value at a covariate value of interest, and that value is a weighted sum of the outcomes with weights the design fixes before any outcome is seen. In the tail of a lognormal covariate a natural spline's value rests on 4.8 of a hundred points, fewer than the squared term's coefficient's 6.9, and the squared term's own value there rests on 10.2 — yet it is three times further from the truth, because its extra points are borrowed from where the curve has a different shape and buy no precision at all. At four hundred points, moving the last knot past the value raises its count threefold and halves its error; doubling the data does neither.

    part 10 · spread

All series