The summary that was meant to work

Four datasets, one summary ends on a claim it supports by argument rather than by measurement: no summary would work. The argument is that a summary is a projection and a projection loses what it projects away.

It is worth testing, because there is a summary with a theorem attached that says it does not lose dependence.

The measure with a theorem #

Distance correlation is built from the pairwise distances between observations rather than from their products. Form the matrix of |x_i − x_j|, doubly centre it — subtract the row means and the column means and add the grand mean back — do the same for y, and average the elementwise product of the two centred matrices. Normalise by the geometric mean of the same quantity computed with each variable against itself.

The result is between 0 and 1, and it has the property no correlation coefficient has:

It is zero if and only if the two variables are independent.

A Pearson correlation of zero means uncorrelated, which is much weaker: $y = x^2$ on a symmetric x has a Pearson correlation of exactly zero and is a deterministic function. On constructed data the difference is stark — on two hundred uniform draws with $y = x^2$, distance correlation reads 0.494 and Pearson reads 0.0025. On two hundred independent draws they read 0.110 and 0.0724. The theorem does what it says.

What it does on the quartet #

Pearson agrees to three decimal places across the four, which is the construction working and is the point of the quartet. So does $R^2$, at 0.667 in every one of them — which is the same statement, since R-squared is the square of the correlation in a simple regression, and the quartet’s agreement on both is one agreement rather than two.

Distance correlation spreads them by 0.101, which is more than Pearson and is not much. The four values run from 0.807 to 0.908 — all high, all in the range a reader would call “strong dependence”, and the ordering is not the ordering of anything a reader cares about: the dataset with an outlier scores highest.

Spearman spreads them by 0.491 and Kendall by 0.436 — four to five times as far as the measure with the theorem.

The guarantee is about a limit and the question is about eleven points #

The apparent paradox dissolves once the theorem is read carefully.

Distance correlation is zero if and only if the variables are independent — in the population, and with the sample version converging to the population one as n grows. At eleven observations it is a noisy estimate of a population quantity, and the population quantities of the four datasets are not very different: all four are strongly dependent relationships, and distance correlation measures how much dependence there is rather than what shape it has.

That is the general point and it is more useful than the measurement. It is the same distinction a normality test draws and loses: a test that rejects normality says the data are not normal and says nothing about which departure, and the departure is what a reader needs. Detecting dependence and describing dependence are different jobs, and a measure optimised for the first is not thereby good at the second. A test with power against every alternative is a test that distinguishes “something” from “nothing”; telling a curve from a line with an outlier is distinguishing two somethings, and nothing in the theorem bears on it.

The rank correlations do better here for a reason that is not a virtue either. Spearman is the Pearson correlation of the ranks, so it responds to monotonicity, and the four datasets differ sharply in how monotone they are — the outlier dataset is almost perfectly monotone, the curve turns over at its right end, and the fourth has ten tied x values. That the ranks separate them is a coincidence of these four datasets rather than a property of rank correlations, and a different quartet could be constructed on which the ranks agree and something else separates them.

What the four numbers would have had to do #

It is worth being explicit about what would count as success here, because “spreads them further” is a weak criterion and the quartet deserves a stronger one.

A summary that worked would have to order the four by how much the linear model is wrong, or at least separate the one that is right from the three that are not. The first dataset is a linear relationship with noise and is the one a regression correctly describes; the other three are not.

No column does that. Pearson puts them within 0.002 of each other. Spearman puts the outlier dataset highest and the degenerate one lowest, which is an ordering by monotonicity and has nothing to do with whether a line is the right model. Kendall puts the degenerate one at exactly 1.000, the highest possible value, which is the most misleading single number in the table — it reads as a perfect relationship and is produced by ten tied x values and one point.

Distance correlation puts the outlier dataset highest and the honest one third of four. So on the criterion that matters, the best of the four summaries is the one that is flattest, because at least its ordering carries no false signal.

That is the honest reading of the table and it is a stronger statement than the quartet’s own. Not only does no summary separate them; the ones that do spread them, spread them in orders that a reader would act on and should not.

The two the measure cannot reach, and why #

The four datasets have four defects and only two of them are about the shape of the dependence.

The curve is a genuine shape difference: the relationship between x and y is not linear, and a measure sensitive to the form of the dependence has something to see. Distance correlation does see it — it scores the curve highest of the three non-outlier datasets, 0.869 against 0.824.

The line with an outlier is not a shape difference. Ten of its eleven points lie almost exactly on a line, and one does not. The relationship is a strong linear one; what is wrong is one observation. No summary of the dependence between two variables can flag that, because the dependence is fine — and distance correlation’s response is to score it 0.908, the highest of the four, because ten points on a line is very strong dependence and the eleventh barely dilutes it.

The one carrying the slope is a design defect, not a dependence defect. Ten of its x values are identical and the eleventh is far away, so the relationship is determined by a single point and almost no dependence has been measured at all. Distance correlation scores it 0.807 — the lowest of the four, but barely — because the configuration is strongly dependent in the technical sense.

So the quartet’s essay is right and the reason is sharper than the reason it gives. It is not that summaries lose information in general; it is that two of the four defects are not properties of the joint distribution of x and y, and no function of that joint distribution can report them. One is a property of a single observation and one is a property of the design.

What does reach them #

Each of the two has a diagnostic and neither is a summary of the dependence.

Leverage reaches the fourth. $h_{ii} = 1/n + (x_i - \bar{x})^2/S_{xx}$ depends only on the x values, so it is computable before y is looked at, and on that dataset the far point has a leverage of about 1 — it determines its own fitted value entirely. That is a property of the design, computed from the design.

Influence reaches the third. Cook’s distance asks how far every fitted value moves if one point is dropped, and the outlier’s is large. That is a property of one observation, computed by removing it — and two points can hide each other from it, which is its own limitation and a different one from the summaries’ above.

The pattern is worth stating because it generalises past the quartet. A summary is a function of the whole dataset and a defect in one observation survives averaging. The diagnostics that find such defects are always per-observation — leverage, influence, deleted residuals, studentised residuals — and the reason is arithmetic rather than tradition: a quantity computed over n observations dilutes anything that lives in one of them by a factor of n.

That is the same argument as the residual plot, which is a per- observation display rather than a summary, and it is why a plot remains the first diagnostic in a field that has had good summary statistics for a century. It is also why R-squared cannot be rescued by a better version of itself: every proposal in that direction is a different function of the same whole dataset, and the dilution is a property of the averaging rather than of the particular average chosen.

The cost of the measure with the theorem #

Distance correlation is not free, and the costs are worth stating beside what it does not buy here, because they decide when it is the right tool.

It is quadratic in the sample size. The construction is two n × n matrices of pairwise distances, so a hundred thousand observations is ten billion entries. There are faster algorithms for the univariate case; the general one is what the definition gives.

It has no sign. Distance correlation is non-negative by construction, so it cannot say whether the relationship is increasing or decreasing. On the quartet that is a small loss and on ordinary data it is a large one — the first thing anyone wants from a correlation is its direction.

It has no closed-form null distribution. Testing whether a distance correlation is significantly non-zero is done by permutation, because the sampling distribution under independence depends on the marginal distributions of both variables. That is not a difficulty — a permutation test is the right tool and costs what permutations cost — but it means the measure arrives without the table of critical values that makes a correlation coefficient usable by inspection.

Its scale is not interpretable. A Pearson correlation of 0.8 has a meaning: it is the cosine of an angle, its square is a share of variance, and it corresponds to a known amount of scatter about a line. A distance correlation of 0.8 corresponds to nothing a reader can picture, and the four values on the quartet — all between 0.807 and 0.908 — illustrate the problem: the scale is compressed near the top and the compression is where the interesting comparisons are.

There is one setting where it is clearly worth all three costs, and it is not a two-variable one. Distance correlation is defined between vectors of any dimensions, and it is one of very few measures of association that is. Asking whether a ten-dimensional measurement is independent of a three-dimensional one has no answer from any correlation coefficient, and it has one here — which is where the measure earns its place in practice, and is a long way from the scatter plot this page is about.

So the right use of it is the one the theorem supports and no other: as a test for independence, where the question is whether there is anything at all and the answer is wanted with power against every alternative. Used that way it is excellent and there is nothing else quite like it. Used as a summary to be compared across datasets, it is a number with a good theorem and a bad scale.

What is claimed here, and what is not #

Three statements, deliberately of decreasing strength.

Distance correlation does what it claims, established on constructed data before it is used on anything: on two hundred draws with $y = x^2$, it reads 0.494 against Pearson’s 0.0025, and on independent data it reads 0.110 against 0.0724. That is the property the whole page turns on, and establishing it on the quartet would have been circular — the quartet is the question, not the calibration.

The four datasets share a Pearson correlation to within 0.01. That is the quartet’s own claim, restated here so that a change to the stored values would show up on this page rather than quietly changing the comparison.

A rank correlation separates the four further than distance correlation does, which is the finding. It is an inequality between two spreads rather than a threshold, because the size of the gap is a fact about these four datasets and the ordering is the fact worth carrying.

A fourth statement is worth making because it is the one a reader is most likely to supply for themselves. Nothing here says distance correlation is a poor measure. It is the sharpest available instrument for the question it was built for, and the quartet is not that question — eleven points, four strongly dependent relationships, and a demand that the four be told apart by shape. Reporting that it does badly at a job it was not designed for would be the same category error the page is about, so the finding is stated as a comparison with a measure that has no guarantee at all, and left there.

The reading that does not survive is a dependence measure taken as a diagnostic. That reading needs distance correlation to separate the clean line from the line with an outlier, and the two score 0.824 and 0.908 — a gap of 0.084, in the wrong direction and too small either way.

What the quartet is evidence for, after all this #

The four datasets are usually presented as an argument for plotting the data, and the measurement on this page narrows what they establish.

They do not establish that summaries are inadequate in general. Two of the four defects are reachable by summaries — leverage and influence are both single numbers per observation and both have single-number summaries over a dataset — and the third, the curve, is reachable by a test for curvature, which is an ordinary thing to fit.

What they establish is narrower and harder to work around. The summaries a reader is given are not the summaries that would have caught these, and no amount of adding dependence measures to the output changes that, because the defects are not dependence defects. The output would have to grow a maximum leverage, a maximum Cook’s distance and a lack-of-fit test, and at that point the reader is reading four numbers rather than one and the plot has become cheaper.

The plot’s advantage over that list is not that it contains more information — it contains less, in a formal sense, since it is the same eleven points. The advantage is that a reader does not have to know in advance which defect to look for, and every list of summaries is a list of defects somebody anticipated. That is the whole argument for plotting, stated without any claim that summaries are weak, and it is the version that survives the measurement above.

Still open: whether a summary could reach the third and fourth #

The argument above says no summary of the joint distribution can flag a single bad observation or a degenerate design. That is a proof about summaries of the joint distribution, and it leaves a gap.

A summary of the fit rather than of the relationship is a different object. The maximum leverage, the maximum Cook’s distance, the ratio of the largest to the median leverage — each is a single number computed from the whole dataset that would flag two of the four immediately, and each is a summary in the ordinary sense.

So the correct statement is narrower than “no summary would work”: no summary of the dependence between x and y would work, and a summary of the regression’s own geometry might. What such a summary should be, whether one number can flag both defects or whether two are needed, and what it costs in false alarms on honest data, are questions that the twenty-panel device answers for a picture and nothing answers for a number.