A degrees of freedom that is not a count

The t interval’s own essay recommends Welch’s two-sample test by default, on the grounds that it “costs nothing, is barely less powerful when the variances are equal, and is substantially more reliable when they are not”. Every clause of that is a claim and none of them is measured there.

The region #

Forty units, split between two groups; the second group’s variance a stated multiple of the first’s; and the two population means equal throughout, so a 5% test should reject 5% of the time everywhere.

The pooled test does not. Its size runs from 0.55% to 18.91% across twenty-five cells, and the pattern in the grid is the whole result:

The middle row is flat. With the groups equal in size, the pooled test reads 5.3%, 5.1%, 5.3%, 5.3% and 5.3% across variance ratios from a quarter to four. Equal sizes make the assumption of equal variances irrelevant, which is the classical result and is visible as a row.

The top-left and bottom-right corners are anti-conservative. Eight units against thirty-two with the small group four times as variable gives 18.91% — a test labelled 5% rejecting a true null nearly one time in five.

The opposite corners are conservative to the point of uselessness. Eight against thirty-two with the large group four times as variable gives 0.55%, a ninth of its label, and a test that rejects at 0.55% has correspondingly little power.

The rule is one line and it is worth carrying: the pooled test is too liberal when the smaller group is the more variable one, and too conservative when it is the less variable one. The smaller group’s variance is under-weighted by the pooling, so its contribution to the standard error is too small when it should be large.

What Welch does instead #

Welch computes the standard error directly, without pooling:

$$\operatorname{se} = \sqrt{\frac{v_1}{n_1} + \frac{v_2}{n_2}},$$

which is the obvious thing and is not the difficulty. The difficulty is that the resulting statistic is not a t on any number of degrees of freedom, because the denominator is a sum of two scaled $\chi^2$ variables with different scales and that sum is not $\chi^2$.

Satterthwaite’s approximation replaces it with the $\chi^2$ that has the same mean and variance, which gives

$$\nu = \frac{(v_1/n_1 + v_2/n_2)^2}{\dfrac{(v_1/n_1)^2}{n_1 - 1} + \dfrac{(v_2/n_2)^2}{n_2 - 1}} .$$

Three things about that expression are unusual enough to cause discomfort, and all three are real.

It is not an integer. At eight and thirty-two units with a variance ratio of four it averages 24.8, and a degrees of freedom of 24.8 is not a count of anything.

It depends on the data. $v_1$ and $v_2$ are the estimated variances, so ν is a random variable — its standard deviation across samples is 7.36 at that configuration. The reference distribution is chosen by the data it is being used to judge.

It is an approximation with no exactness claim. Unlike the one-sample t, which is exact under normality, this is a moment match and is right only to the extent that the matched $\chi^2$ resembles the real distribution.

And it works anyway #

All three of those discomforts are answered by the same measurement, which is why the region is the right way to look at this rather than the formula.

Welch’s size across the twenty-five cells runs from 4.63% to 5.51%. The worst cell is out by half a percentage point, against the pooled test’s worst of nearly fourteen. Every one of the objections above is about how ν is arrived at, and the thing that matters is what the test does, and what it does is hold its level.

The reason the approximation is good is worth one sentence, because it is not obvious that a moment match should be. The quantity being approximated is a ratio of the two variance estimates’ weighted sum to its own expectation, and a ratio of quadratic forms is very well described by a scaled $\chi^2$ over the range where the weights are not extreme — which is the whole of a region where neither group is tiny.

The cost is visible in the same figure. At eight and thirty-two with equal variances, Welch has 12.4 degrees of freedom against the pooled test’s 38. That looks ruinous and is not: the critical value at 12.4 degrees of freedom is 2.171 against 2.024 at 38, a 7% wider interval, and the pooled test’s own size in that cell is 5.13% against Welch’s 5.31%. Seven per cent of width is what the insurance costs.

The degrees of freedom is a random variable, and it does not matter #

The objection that feels strongest — the reference distribution is chosen by the data — deserves the measurement rather than the reassurance.

At eight and thirty-two units with equal variances, ν averages 12.4 and has a standard deviation of 4.16 across samples. So the test is not comparing its statistic with one fixed distribution; it is comparing it with whichever of a family the sample nominated, and the nomination is made from the same two variance estimates that are in the statistic’s denominator.

That is the structure which, in the two cases below, breaks things. A critical value computed from the sample it is applied to reproduces its own nominal level by construction and proves nothing; a pre-test that chooses a method contaminates what follows. Both are cases of the data choosing its own reference.

Welch escapes for a specific reason: ν is chosen to make the approximation good rather than to make the result come out a particular way. The dependence is on the variance estimates’ relative magnitudes, which carry no information about the difference in means under the null, so conditioning on ν does not condition on anything about the numerator. That independence is what makes the data-dependent reference harmless here and is precisely what a pre-test on the means, or a critical value computed from the observed statistic, would not have.

The measurement is the evidence rather than the argument. If the dependence were harmful the size would show it, and across twenty-five cells the size is between 4.63% and 5.51%.

Why the pre-test is worse than either #

The natural response to a region like that is to test for equal variances first and choose accordingly. It is the worst of the three options and the reason is the same as everywhere else in this collection.

A pre-test is a decision made from the data, so the procedure that follows is a conditional procedure and its error rate is not the error rate of either branch. The pooled test’s 5% is its rate over all samples; its rate over the samples that passed a variance test is different, because the samples that pass are the ones whose estimated variances happened to be similar, and that is a selected set.

The direction is unhelpful. At small samples the variance test has almost no power, so it passes almost always and the procedure is the pooled test — in the region where the pooled test is worst. At large samples the variance test has power, so it detects small variance differences that would not have mattered, and the procedure is Welch — in the region where the pooled test was fine.

So the pre-test does the wrong thing at both ends, and in between it produces an error rate that is neither branch’s and that nobody has computed for the study in hand. The same structure makes a normality pre-test useless: the test’s power is lowest exactly where the assumption matters most.

Where the unequal variances come from #

The region is a sweep and it is worth saying which of its cells are the ones studies land in, because the answer is not “the middle”.

A treatment that helps some subjects and not others is more variable than a control. If everyone in the control arm responds identically and the treated arm splits into responders and non-responders, the treated variance is larger by the between-group term. That is the commonest mechanism and it is not a nuisance — a variance difference is itself a finding about heterogeneity of response.

Unequal allocation is deliberate. Two-to-one and three-to-one randomisations are used to expose more subjects to a new treatment, to recruit faster, or because a control arm is shared between trials. So the unequal-size column is the design and the unequal-variance row is the biology, and they arrive together.

And either corner is reachable. The new treatment is the one allocated more subjects and the one with the larger variance, which by the rule above puts the larger group with the larger variance — the conservative corner. But a trial allocating fewer subjects to an expensive or risky new treatment puts the smaller group with the larger variance, which is the liberal one, and that is the arrangement of a first-in-human or a dose-finding study.

So both corners are occupied, which is why the rule matters more than the direction. A reader checking a two-sample comparison needs the two group sizes and the two standard deviations, all four of which are in any table of results, and the four decide which corner the analysis was in. That is the same one-line check a reversal needs: the margins of the table say whether the analysis in the middle of it can be trusted.

The same four numbers decide whether a weighting argument holds in a neighbouring case, which is a reminder that the standard deviation column of a results table is doing more work than its placement suggests.

What the recommendation actually rests on #

Putting the three claims from the earlier essay against the measurement:

“It costs nothing.” Not quite: it costs 7% of interval width at the configuration above, and more where the split is more uneven. The honest version is that it costs a little power in the case where the pooled test is correct.

“It is barely less powerful when the variances are equal.” True. In the equal-variance column the two tests’ sizes are 5.13% and 5.31%, and the power difference follows the same small gap in the critical value.

“It is substantially more reliable when they are not.” True and understated. The pooled test’s worst cell is out by a factor of nearly four in one direction and nine in the other, and Welch’s worst is out by ten per cent of its own value.

The conclusion is the one the earlier essay reaches, arrived at by measurement rather than by authority: use Welch by default. What the measurement adds is the shape of the failure being avoided, which is not the shape most people expect — the pooled test is not uniformly liberal, it is liberal or conservative depending on which group is which, and a reader who remembers only “it can be anti-conservative” will not recognise the conservative half when they meet it.

The same problem with more than two groups #

Two groups is the case with a name. Several groups is the case that gets analysed, and the analysis of variance pools every group’s variance into one number.

The consequences transfer directly and get worse. The F test’s size departs from its label whenever the groups differ in both size and variance, and with several groups there are more ways to be unbalanced — one small variable group among five large stable ones is enough. The analogue of Welch’s repair exists and is the Welch one-way test, which computes its own denominator and its own Satterthwaite degrees of freedom by the same logic.

It is much less used than the two-sample version, for a reason that is worth naming because it is not statistical. The two-sample Welch test is the default in common software; the one-way version is an option that has to be asked for, and defaults are what get run. So a field that has largely adopted the repair for two groups has largely not adopted it for three, and the underlying problem is identical and slightly worse.

The last observation there is the one to carry away. Welch’s degrees of freedom is smallest where the variances are most unequal, which is where the pooled test’s count of thirty-eight is most misleading. The two move in opposite directions, and the pooled test’s constant thirty-eight is a number that does not know anything has happened.

What is claimed here, and what is not #

The statements are made over the region rather than at a point, which is the only way to state a claim about a region.

The pooled test departs from 5% by more than three points somewhere in the region. Over the whole grid, so it would not hold if the sample sizes had been equalised by accident — the middle row is flat, and every cell collapsed onto it would produce a perfectly reasonable picture with nothing wrong in it.

Welch’s does not, anywhere in it. A bound over all twenty-five cells rather than an average, because an average over a region that includes the flat middle row would be satisfied by a method that failed at the corners.

Welch’s degrees of freedom never exceeds the pooled test’s, which is the price it pays and is a mathematical fact rather than a measurement. It is worth stating because it is the sign that ν is computed rather than fetched: a version returning $n_1 + n_2 - 2$ would look correct everywhere the variances are equal.

The reading that does not survive is the pooled test with unequal variances and unequal sizes. A test’s own standard is that a 5% test rejects a true null 5% of the time, and at eight and thirty-two units with a variance ratio of four it reads 0.55%. Choosing that corner rather than the liberal one is deliberate: a comparison that only caught the liberal corner would look right on a version that had the pooling backwards.

Still open: what the region looks like when the shapes differ too #

Every cell above is two normal populations. The two failures measured here and in the essay before it — skew unbalancing a one-sample interval’s tails and unequal variances breaking the pooled two-sample test — have been measured one at a time.

They are not independent, and the interaction has a plausible sign. Welch’s ν is computed from two estimated variances, and under skew each of those is correlated with its own group’s mean, so the reference distribution is chosen by a quantity that is correlated with the statistic it is judging. That is exactly the coupling the one-sample case shows is not repaired by sample size.

Whether Welch’s excellent behaviour on the grid above survives two skewed populations, whether it survives two populations with different skews — which is the realistic case, since the more variable group is usually the more skewed one — and whether the failure lands on the size or on the balance of the two tails, are three questions the arithmetic here could answer and has not.

Still open: skew and unequal variances arriving together #

One guess can be made from what is already measured and it points the wrong way for comfort. A two-sample statistic is a difference of two means, and a difference of two independent quantities is less skewed than either — the skews partly cancel when the groups are similar. So two groups with the same skew should behave better than the one-sample case, and two groups with different skews should behave worse, because then nothing cancels. The realistic configuration is the second one, and it is the one nobody has drawn.

The measurement that would settle it is the region above with a skew parameter added to each axis, and it is four times the work of the region that exists. That it has not been done is a statement about this collection rather than about the literature, which has the same gap for the same reason: both departures are usually studied one at a time, and the interesting case is where they meet.