The reference the sandwich is read against

The essay before this one found the variance: three hundred rows in five clusters of sixty carry 6.9 times the variance a row-counting calculation reports, and an interval built on the row count covers 53%. The repair is the cluster-robust estimator, which sums residual contributions within a cluster before squaring them, and which is consistent for the right variance.

Consistent in what. The estimator’s large-sample argument is a limit in the number of clusters, not of rows, and a study with three hundred rows in five clusters has five of whatever the limit is in. So the estimator can be exactly the recommended one, applied exactly correctly, and the interval still misses.

Twenty points of coverage at five clusters, and half of it recoverable by changing a table lookup.

The rest of this essay is about which half is which, because the two causes want different repairs and only one of them is free. A study that applies the free one and stops has an interval that is still short; a study that applies neither has one that is short by twice as much and says 95% on the label.

The number of clusters is not the sample size #

The distinction is easy to state and easy to lose, so it is worth a figure of its own that removes every other explanation.

Every study on that axis has three hundred observations. Every one of them reports a sample size of three hundred, has the same degrees of freedom in its regression output, and passes every check that counts rows. The coverage runs from 74.28% to 94.63%, and the quantity that moves it is one nothing in the output is keyed on.

The reading is that the cluster count is the sample size for this purpose. A study with five clusters has, for the purpose of estimating how variable its own slope is, five observations — and an interval built from five observations is not an interval anybody would read against ±1.96 in any other context.

There is a second reading of the same figure that is worth separating, because it points the opposite way from the usual intuition about sample size. At seventy-five clusters of four rows each the interval covers 94.63% — four rows per cluster, which sounds desperate — and at five clusters of sixty it covers 74.28%. More rows inside a cluster buy almost nothing, because rows inside a cluster repeat each other; more clusters buy everything, because clusters are what the estimator averages over.

That is the same arithmetic the design effect makes visible applied to the variance of a variance rather than to the variance of a slope. A study that can choose between surveying more people in each of its sites and adding sites is choosing between an axis that saturates and one that does not.

Two causes, and only one of them is a reference #

The undercoverage has two sources and separating them decides what the repair is. The first is that the estimator is too small.

At five clusters the estimator averages 68.6% of the true standard deviation. The bias is structural: the cluster-robust sum uses residuals from a fit that has already used the data, and with few clusters those residuals are systematically too small — the same reason an uncorrected sum of squared residuals underestimates a variance, arriving one level up.

If that were the whole story, the arithmetic would say what the coverage should be. An interval whose standard error is a constant fraction s of the truth covers 2Φ(1.96·s) − 1, which at s = 0.686 is 82.09%.

It covers 75.05%. So the mean shortfall accounts for 12.9 of the 19.9 points lost, and the remaining 7 points come from the estimator’s variability: a standard error estimated from five numbers is itself a noisy quantity, and an interval built by multiplying a noisy scale by a fixed constant misses more often than one built by multiplying an accurate scale by the same constant.

That second part is exactly what a t reference is for. A t distribution’s heavier tails are the correction for dividing by an estimated rather than a known standard deviation, and the degrees of freedom are how many numbers the estimate was made from. At five clusters that is four, or three once the covariate’s own degree of freedom is taken out, and reading the same estimate against a t on three covers 87.95%.

Why the estimator is short, and why that is not a bug #

The downward bias has a cause worth stating, because it is the same one that makes an uncorrected residual variance too small and it explains why the size of the bias is what it is.

The cluster-robust sum is built from the fit’s own residuals, and a fitted line is fitted to the data it is being measured against. Some of the variation the residuals should carry has been absorbed into the fitted coefficients, and with G clusters the absorption is a fraction of order 1/G of the total. At eighty clusters that is a per cent or two; at five it is a fifth, which is what the 68.6% is.

The correction every implementation applies — multiplying by G/(G − 1) times (n − 1)/(n − 2) — is aimed at exactly this and it is already in every number above. It is not enough at five clusters, which is why the shortfall is measured rather than assumed corrected away. That is the same pattern the four heteroskedasticity corrections show: a finite-sample factor derived under a favourable arrangement helps and does not finish the job.

What the correction does and does not buy #

The remaining gap — 87.95% against 95% — is the estimator’s mean shortfall, and no reference distribution repairs a systematic bias in a scale. Two readings follow.

The cheap half is free. Changing ±1.96 to a t quantile costs nothing: the same fit, the same estimator, a different number. It recovers 12.9 points at five clusters, 5.9 at eight and 3.1 at twelve, and by forty clusters it is worth under a point. There is no setting in the sweep where it costs anything, because the t quantile exceeds the normal one at every finite degrees of freedom, so the interval is only ever made wider.

The expensive half is a different problem. Repairing the estimator’s bias needs a bias correction to the estimator — the family the four robust corrections belong to, in their cluster form — or a resampling scheme that never forms a standard error at all. Both are real work and neither is a table lookup.

So the decomposition is also a recommendation about order of operations: take the free half always, and decide about the expensive half by looking at how many clusters there are.

The correction is not a universal fudge #

It is worth showing what the t reference does not do, because a correction that rescues everything is a correction that means nothing.

Take the interval the previous essay condemned — the one that counts rows and ignores clustering entirely — at five clusters of sixty. It covers 40.33%. Reading the same interval against a t on four degrees of freedom takes it to 55.77%, which is still a catastrophe.

The reason is that what is wrong there is the variance, not the reference: the row-counting standard error is short by a factor governed by the design effect, which at that arrangement is several-fold, and no adjustment to a critical value the size of a t-versus-normal gap touches it. A t reference corrects for a scale having been estimated. It does not correct for a scale having been computed from the wrong formula, and the two failures are not on the same scale.

Where the threshold sits #

Collecting the measured column gives a rule of thumb with its own error bars attached. Read against a t on G − 1, the interval covers 85.30% at five clusters, 89.20% at eight, 90.97% at twelve, 92.67% at twenty, 93.55% at forty and 94.03% at eighty.

Two features of that column are worth reading. It approaches 95% from below and does not reach it inside the sweep — at eighty clusters the interval is still a point short, because the estimator’s own shortfall is still 1.7%. And the steepest part is below twenty, which is where most applied studies with clustered data sit: a study with fifteen school districts, eight hospitals or six countries is in the region where the answer depends on which correction was applied.

What a defensible procedure looks like #

Say what the degrees of freedom were. Software that offers a cluster-robust option generally also chooses a degrees-of-freedom convention, sometimes G − 1, sometimes G − 2, sometimes n − k, and the last of those is the normal reference by another name at any realistic row count. Two packages can report different intervals from identical estimates, and the difference is worth ten points of coverage at five clusters. It is the same defect a reference distribution with no table exhibits in a different field: the statistic is right and the thing it is read against is a choice nobody records.

Report the cluster count beside the sample size. It is the number the interval’s properties depend on and it is almost never printed. A reader given “n = 300” cannot distinguish a study that covers 94.63% from one that covers 74.28%.

Use a t on the cluster count by default. It is free, it is never worse, and it is the correction most software already offers under a flag. The choice between G − 1 and G − 2 is worth two points at five clusters and nothing beyond twenty, and G − 2 is what the derivation for a single regressor gives.

Prefer the wider of two candidate analyses. Every correction discussed here makes the interval wider, so an analyst who has two analyses in hand has reason to be suspicious of the narrower one rather than pleased by it. That is the standing direction of this whole field — a robust standard error is not a safety margin, and it can run either way — but at few clusters the direction is not ambiguous: everything that is wrong with the interval makes it too short.

Do not add rows to fix it. The instinct when an interval looks unreliable is to collect more data, and at a fixed number of clusters that buys almost nothing here: the fixed-row figure shows five clusters of sixty doing worse than seventy-five clusters of four. Where the clustering is the binding constraint, the useful direction is more clusters and the useless one is bigger clusters, and an analysis that reports only n cannot tell which it has.

And treat fewer than about twenty clusters as a different problem. At twenty clusters the corrected interval covers 92.67%, which is a couple of points short and is the kind of shortfall a study can carry. At eight it is 89.20% and at five 85.30%, and those are not intervals a 95% label describes. Below that threshold the honest options are a bias-corrected estimator, a resampling scheme, or a statement that the data does not support an interval — which is the same conclusion a variance estimated from two units reaches from a different direction and about a different quantity.

The shape this shares with the rest of this field #

Six essays in this field have now found the same thing in six places, and it is worth naming because the seventh instance is the one most likely to be mistaken for a special case.

A robust procedure replaces an assumption with an estimate, and the estimate has properties of its own. The robust standard error is not a safety margin because the estimate can come out smaller than the model-based one. It is not free because the estimate is more variable. The four corrections disagree at a leverage point because the estimate is a weighted sum and the weights differ. And here the estimate is made from five numbers, so the constant it is multiplied by has to be the one that accounts for having estimated it.

Every one of those is the same sentence: the quantity that was assumed has become a quantity that was estimated, and everything an estimate does now applies to it. That is what makes this essay’s repair predictable rather than a discovery — a t reference is what statistics has always done when a scale is estimated, and the only question was how many numbers it was estimated from.

What is claimed here and what is not #

Equal cluster sizes throughout. Every sweep here uses clusters of the same size, which is the favourable case: unequal sizes make the effective number of clusters smaller than the nominal one, so the coverage at a stated G is an upper bound rather than a typical value. The previous essay measured what unequal sizes do to the variance; what they do to the reference is not measured here, and the direction is the unhelpful one.

One intraclass correlation. The sweeps are at ρ = 0.2. A larger correlation makes each cluster carry less independent information and worsens every column; a correlation of zero makes clustering irrelevant and all three references agree. Nothing in the argument depends on the value, since the estimator’s finite-cluster behaviour is about the count and not the strength, but the numbers do.

The decomposition uses a normal approximation for one of its two parts. “A standard error short by a constant factor s covers 2Φ(1.96·s) − 1” treats the estimated scale as fixed at its mean, which is precisely what it is not — that is the point being made. The figure is therefore a reference line showing what a pure mean-shortfall would give, not a competing estimate of the coverage, and the gap between it and the measurement is the quantity of interest rather than a disagreement between two methods.

The row-counting comparison is at a single setting. The 40.33% and 55.77% figures come from five clusters of sixty at the same intraclass correlation as the rest, and they are quoted to show that a t reference does not rescue a wrong variance rather than to characterise row-counting intervals in general, which the previous essay does properly.

And “consistent” is used in its technical sense. The cluster-robust estimator converges to the right variance as the number of clusters grows without bound, and every measurement above is consistent with that: the shortfall runs 68.6%, 82.4%, 87.4%, 93.2%, 96.7%, 98.3% across the sweep. Nothing here is a claim that the estimator is wrong. It is a claim about what a limit does and does not say about a study.

Still open: what happens when the clusters are unequal and few #

The two essays in this part of the field that deal with clustering have each held one thing fixed. The previous one varied the sizes and their arrangement at a comfortable cluster count; this one varied the count at equal sizes. The case studies actually present is both at once, and it is the case with the least reason to behave.

There is a specific reason to expect it to be worse than either measurement suggests. Unequal sizes mean a few clusters dominate the sum the estimator is built from, so the effective number of terms in that sum is smaller than G — and if the right degrees of freedom is an effective count rather than G − 2, then the t correction applied at G − 2 is itself too optimistic exactly where it is most needed. What that effective count is, whether it can be computed from the sizes and the covariate arrangement the way a design effect can, and whether a t reference on it closes the gap, is a measurement this field has not made.