Normal Distribution
nothing is called 95% until it has been counted
Field

What a summary of a scatter is a property of

R-squared is read as a property of a relationship and it is a property of a design. Five studies differing only in how far apart they placed their readings report it from 0.021 to 0.849, with the same line and an estimated residual spread of 0.993 in every one. For a simple regression the squared t statistic is (n − 2) times R-squared over (1 − R-squared), exactly, checked to sixteen significant figures across five hundred fits — so a report carrying both has carried one number twice. A coefficient that is zero only under independence does exist: distance correlation separates the four datasets built to share a correlation by 0.10, and the rank coefficient that guarantees nothing separates them by 0.49.
One relationship at five designs, residual spread 1.00. Every panel has the same slope of 1, the same intercept of 0 and the same residual standard deviation of 1.00. Only the range of x differs. R-squared runs from 0.021 to 0.849, and the estimated residual spread is 0.9932 in all five.

R² is a property of the design

One line, one residual spread, five studies that differ only in how far apart they placed their x values. R² runs from 0.021 to 0.849 and the estimated residual spread is 0.993 in every one of them. Nothing about the relationship changed.

 Two studies with an R-squared of 0.85. The left study's points sit 0.50 from the line and the right study's 2.00 — a factor of 4.0. Both report an R-squared of 0.85 and, at the same sample size, the same standard error for the slope. The design was chosen to make it so, and it can always be chosen.

The t statistic wearing different clothes

For a simple regression, t² = (n − 2)R²/(1 − R²), exactly, on every dataset — checked to sixteen significant figures over five hundred fits. So a paper reporting R² and a p-value has reported one number twice, and two studies with the same R² have points four times further from the line.

 How far apart four summaries put the quartet. Each bar is the largest value minus the smallest across the four datasets. Pearson spans 0.0018, which is the construction working. Distance correlation — the measure that is zero if and only if the variables are independent — spans 0.101, and Spearman spans 0.491.

The summary that was meant to work

Distance correlation is zero if and only if two variables are independent, which is exactly the guarantee a correlation coefficient lacks. Run on the four datasets that share a correlation, it spreads them by 0.10 — and Spearman, which guarantees nothing, spreads them by 0.49.

 Three summaries of a fitted line's geometry on Anscombe's four datasets, with each rule's threshold. Largest leverage: 0.318, 0.318, 0.318, 1.000 against a threshold of 0.364. Largest Cook's distance: 0.489, 0.808, 1.393, undefined, the point fits itself against 1 and against 0.364. Largest studentised deleted residual: 2.1, 2.2, 1192.2, undefined against a Bonferroni threshold of 3.90.

Two numbers for the fit's geometry

No summary of the dependence between x and y can flag Anscombe's third and fourth datasets. A summary of the fit's own geometry can, and it takes two numbers of different kinds: the largest leverage, which reads the design before any y is seen and flags the fourth with certainty, and the largest deleted residual against a Bonferroni threshold, which flags the third at 1,192 and raises a false alarm on 4.78% of honest datasets. The rule of thumb most often quoted, Cook's distance over 4/n, flags the honest line too — and 66.45% of honest datasets like it.

 How many of 100 points a fitted slope rests on, by the distribution the covariate was drawn from. The effective number of points behind a slope is n divided by the design's kurtosis. Over designs of 100, its median share is two points, half at each: 1.000; uniform: 0.551; normal: 0.349; exponential: 0.160; Student's t, 3 df: 0.148; lognormal, σ = 1: 0.076 — 34.9 of 100 points for a normal covariate and 7.6 for a lognormal one.

The points a slope rests on

Before any outcome is measured, a regression's design already says how many of its points the slope will rest on: n divided by the covariate's kurtosis, exactly. A hundred points of a normal covariate carry a slope on about 35 of them; a hundred of a lognormal one, on 7.6. The largest leverage follows the same tail — above one half in 15.7% of lognormal designs of a hundred and in none of the normal ones — and on a correct straight line, deleting the single highest-leverage point moves the slope by more than its standard error in 16.3% of lognormal datasets of a hundred and 0.1% of normal ones.

 A point that decides a coefficient, on two covariates correlated at 0.9. A hundred points on two normal covariates with correlation 0.9. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the first coefficient's information, 14.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 26.1 effective points, with a variance inflation of 6.27.

A point ordinary on every axis

In a multiple regression each coefficient rests on its own number of points, n divided by the kurtosis of that covariate's residual after the others, exactly. Correlation between normal covariates inflates a coefficient's variance fivefold and leaves it resting on the same third of the points. What correlation does instead is hide the point that decides the coefficient: at a correlation of 0.9 it stands out on no single covariate's plot in 77.3% of designs of a hundred, and the largest leverage a normal design reaches depends on the number of points and covariates and on nothing else.

 A point that decides an interaction, on two covariates correlated at 0. A hundred points on two normal covariates with correlation 0. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the interaction's information, 10.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 19.0 effective points.

A term built from the others

A regression's squared terms and interactions are columns built from its covariates, and each coefficient on one rests on the number of points divided by the kurtosis of what the built column adds — fifteen for the square of a normal covariate, nine for the product of two uncorrelated ones, three for a treatment's interaction with a normal covariate, which is the same as the covariate's own coefficient. A design of a hundred rests on more than the population says, 12.2 points behind a square against 6.7, because a hundred draws rarely hold the tail that sets the kurtosis. Centring changes nothing about the square's coefficient, to the last digit. And correlation between two covariates hides the point that decides each one's own coefficient while it reveals the point that decides their interaction.

 The weight each of a hundred outcomes carries in the fitted value at the 97.5th percentile of a lognormal covariate. One design of a hundred lognormal covariate values. The fitted value at a covariate value of 7.10 is a weighted sum of the outcomes, with weights fixed by the design. Under a line and its square it rests on 10.7 effective points, the largest carrying 19.1% of its variance. Under a natural spline with knots at 0.17, 0.57, 1.18, 3.95 it rests on 3.8, the largest carrying 41.1%.

A value read off a curve

No coefficient of a spline is read; what is read is the fitted curve's value at a covariate value of interest, and that value is a weighted sum of the outcomes with weights the design fixes before any outcome is seen. In the tail of a lognormal covariate a natural spline's value rests on 4.8 of a hundred points, fewer than the squared term's coefficient's 6.9, and the squared term's own value there rests on 10.2 — yet it is three times further from the truth, because its extra points are borrowed from where the curve has a different shape and buy no precision at all. At four hundred points, moving the last knot past the value raises its count threefold and halves its error; doubling the data does neither.

All essays