Where the two tails disagree

The t correction repairs the interval for a mean when the standard deviation is estimated, and the repair is measured there by counting coverage: one number per setting, against the 95% it claims.

One number per setting is exactly the statistic that hides what follows.

The number that looks fine #

Draw from an exponential — skewed, one-sided, and the standard hard case. At a hundred and twenty observations the 95% t interval covers 94.81%.

That is a fifth of a point short of its label, and a fifth of a point is inside the noise of most coverage tables. It reads as an interval that works.

Split it by side and it is two errors of opposite sign.

Both columns should read 2.5%. At a hundred and twenty the interval lies entirely below the population mean on 4.08% of samples — 63% more often than it should — and entirely above it on 1.11%, 56% less often. The two errors nearly cancel in the sum, and the sum is what gets printed.

At eight observations the imbalance is 31 to 1 and the total is bad enough to notice anyway. The dangerous region is the middle: a sample large enough that the total looks acceptable and small enough that the sides do not.

And the fifth column is the one that decides how long that region lasts. Between thirty and five hundred observations the total moves from 92.85% to 94.64% — most of the way to its label — while the ratio moves from 7.94 to 1.62, which is nowhere near 1. The two quantities converge at different rates, so there is a wide band of sample sizes over which a coverage table is reassuring and the interval is not symmetric in any useful sense.

Which departure does it #

Heavy tails are the departure from normality everyone worries about, and they are not the one that does this.

At thirty observations:

The Laplace and the t on five degrees of freedom have much heavier tails than a normal — the t’s fourth moment barely exists — and both are balanced to within a tenth. The exponential and the lognormal are symmetric in nothing and both are not.

So the t interval is robust to heavy tails and fragile to skew. That is close to the opposite of the caution usually attached to it, which is about outliers and extreme values. Extreme values on both sides average out; extreme values on one side do not.

Why skew and not tails #

The t distribution is derived from three facts about a normal sample. The mean is normal; the scaled variance is $\chi^2$; and the two are independent. That last one is the property only the normal has, and it is the one that fails here.

Under skew, x̄ and s are correlated, and the correlation does not vanish with the sample size. For the variance it converges to

$$\operatorname{corr}(\bar{x}, s^2) \;\longrightarrow\; \frac{\gamma}{\sqrt{\kappa + 2}}$$

with γ the skewness and κ the excess kurtosis. For an exponential that is $2/\sqrt{8}$ = 0.7071, and the counted values approach it: 0.7023 at eight observations, 0.7093 at eight hundred.

The mechanism follows directly. A sample of exponentials that happens to be low is a sample with no large values in it, and a sample with no large values in it is also a sample with a small spread. So a low sample gives an interval that is centred too low and too short, and both errors push the interval away from the mean in the same direction. A high sample gives one centred too high and too wide, and the extra width pulls it back over the mean. The two cases are not symmetric, and the asymmetry is exactly the correlation.

For a symmetric source the correlation is zero at every size — 0.0088 for the Laplace at eight hundred — so the interval’s two failures are independent of its position and they cancel.

What the limit theorem does and does not repair #

The central limit theorem is the reason the total coverage improves at all. x̄ becomes normal, which fixes the numerator of the t statistic, and the totals in the first table converge on 95% as they should.

It does nothing about the correlation, because the correlation is not a statement about the marginal distribution of x̄. It is a statement about the joint distribution of x̄ and s, and the limit theorem is a theorem about marginals. Both x̄ and $s^2$ are asymptotically normal, individually, and their correlation converges to 0.7071 rather than to zero — so what converges is a bivariate normal with a correlation in it, and a t interval built on the assumption of independence remains wrong about a bivariate normal with a correlation in it.

That is worth stating as a general caution, because the same reasoning is applied loosely everywhere. “Large samples make it normal” is a claim about one quantity at a time, and a procedure that depends on two quantities being independent is not repaired by either of them becoming normal.

The imbalance does fade — 31 to 1 at eight observations, 1.6 to 1 at five hundred — because the interval’s width shrinks like $1/\sqrt{n}$ and the correlation’s effect on the position shrinks with it. What does not fade is the ratio’s dependence on the skew: a more skewed source needs a larger sample to reach the same balance, and the lognormal is still at 3.6 to 1 where the exponential is at 1.6.

Two intervals that cover the same and are not the same #

The clearest way to see what the total hides is to put two settings beside each other that a coverage table would call equivalent.

An exponential source at 120 observations covers 94.81%. A lognormal source at 500 covers 93.83%. Neither is far from 95% and a table sorted by coverage would put them next to each other.

Their tails are 4.08/1.11 and 4.83/1.35. Both are unbalanced in the same direction, which is the point: every skewed source’s interval is unbalanced the same way, and the total is the only number that ever looks reassuring.

Now the comparison that matters. A normal source at any size covers about 95% with tails near 2.5/2.5. A Laplace source at thirty covers 95.19% with tails of 2.30/2.51. Those two are genuinely equivalent, and no table can tell them apart from the two above.

So a coverage number is a valid summary for symmetric sources and an invalid one for skewed sources, and it carries nothing that says which case it is in. That is the general shape of the complaint this collection keeps making about a single number standing in for a distribution — with the particular sting that the failure and the summary are both about the same interval, so there is no second number a careful reader could have looked for.

What it does to a one-sided test #

A two-sided interval’s imbalance becomes a one-sided test’s error rate directly, and that is where it stops being an aesthetic complaint.

A one-sided 5% test of “the mean is at least $\mu_0$” rejects when the interval’s upper limit falls below $\mu_0$ — which is the “falls short” column. On the exponential at a hundred and twenty observations that column reads 4.08% against a nominal 2.5%, so a one-sided test at the 2.5% level is really a 4.1% test: its error rate is 63% above its label.

The mirror test — “the mean is at most $\mu_0$” — rejects at 1.11%, less than half its label, and is correspondingly short of power.

Neither of those is visible in any two-sided coverage study, and one-sided tests are what regulatory comparisons, non-inferiority margins and safety limits are built from. A quantity that is one-sided by nature is exactly the kind of quantity a one-sided test gets applied to, which puts the failure and its worst consequence in the same place.

What a reader should do about it #

Three things, in order of how much they cost.

Report both tails. A coverage study that prints one number per setting has thrown the information away, and printing two costs a column. This is the cheapest fix available and it is not standard practice anywhere — the same omission a single interval’s coverage invites, one level down.

Check the skew before trusting the interval, not the tails. A sample’s own skewness is a poor estimate at small n, but the subject usually says: times to an event, counts, concentrations, incomes and durations are one-sided by nature, and a symmetric interval on any of them is in the regime above whatever the sample says.

Use a method that does not assume the coupling away. A bootstrap interval that adjusts for skewness — the bias-corrected and accelerated form — exists precisely for this, and the acceleration constant it computes is an estimate of the skewness. It costs resampling, and it is the standard answer. The percentile bootstrap on its own is not: it inherits the same asymmetry, because it resamples from a sample that carries it.

There is a fourth that is often the best and is rarely considered: change what is being estimated. The mean of a one-sided quantity is frequently not the quantity anyone wants, and the median or a log-scale mean is both more interpretable and symmetric enough for the interval to behave. That is a subject-matter decision rather than a statistical repair, and it removes the problem rather than correcting it.

Why the direction is always the same #

Every skewed row above misses low more often than high, and the uniformity is worth explaining because it makes the failure predictable rather than merely present.

All five sources are skewed to the right — a long tail of large values, which is what a duration, a concentration or an income looks like. The large values are rare, so a typical sample does not contain one, and a sample without one has both a mean below the population mean and a spread below the population spread.

The interval is then centred low and built short. Two errors, the same direction, compounding. The occasional sample that does contain a large value is centred high and built very wide, and the width is more than enough to reach back over the mean — so it does not become a miss on the other side, it becomes a very wide interval that covers.

That asymmetry in the consequences is what produces the asymmetry in the rates. The many slightly unlucky samples miss; the few very unlucky ones do not, because their own bad luck widened the interval enough to save them.

For a left-skewed source everything above reverses exactly, and the interval overshoots more often than it falls short. That case is rarer in practice and behaves identically in every other respect, which is what makes the rule easy to carry: a symmetric interval on a skewed quantity misses on the side the tail is not.

The same mechanism is why a log transformation works so well here. Taking logs of a lognormal makes it exactly normal, decouples the mean from the spread completely, and restores 2.5/2.5 — at the cost of the interval being about the median of the original scale rather than its mean, which is a real change in what is being claimed and not a repair to the old claim.

What is claimed here, and what is not #

Two statements, one of which is deliberately a comparison rather than a threshold.

Every skewed source is further from an even split than every symmetric one. A comparison, because a threshold would be a fact about the sample size rather than about skew — at a thousand observations the exponential’s ratio is 1.5 to 1, which clears no fixed threshold and is still further from balance than every symmetric source at the same size. The comparison holds at all six sizes drawn.

The correlation between the mean and the spread reaches its predicted constant, $2/\sqrt{8}$, to within 0.03 — paired with the statement that the value at eight observations is not already at it. Without the second, the first would be satisfied by a number written in rather than computed.

The reading that does not survive is a total coverage taken as evidence that the interval works. That reading needs the lower tail to be 2.5%, which is what a reader assumes on seeing 94.81% and stopping, and it is 4.08%. A 95% interval that covers 94.8% by missing twice as often on one side as the other is not a 95% interval with a small error; it is a different object whose summary happens to land nearby.

Both statements are made across the five sources at once rather than one at a time, which matters because the claim is about which kind of departure unbalances the tails. Five separate per-source statements would each hold and none of them would say that skew is the one that does it.

The same split, applied to every other interval here #

The reading this page makes is not about the t interval. It is about coverage as a summary, and every interval in this collection is reported the same way.

Every one of them has two tails and only one of them is counted. An interval for a proportion, whose coverage oscillates with the sample size, oscillates in each tail separately and the two do not oscillate in step. A bootstrap interval on a skewed statistic inherits the statistic’s skew. A forecast interval several steps ahead is built on a sum of errors whose distribution is more skewed the further out it goes.

None of those has been split here, and the split is cheap: it is one extra counter in a loop that is already running.

The reason to expect it to matter is that the sources of asymmetry are common. Skewness is the norm rather than the exception for measured quantities, estimated variances are correlated with estimated means whenever the underlying quantity is positive, and a bounded parameter near its boundary produces the same effect from a different direction. An interval for a proportion at p = 0.05 is about a quantity that cannot go below zero, and the sampling distribution of p̂ is skewed accordingly.

So the honest description of this page is that it measures one instance of something not measured anywhere else here, and that the instance was chosen because it is the one with a clean explanation rather than because it is the worst.

Still open: what the imbalance costs a decision #

Everything here is about coverage, and coverage is a property of an interval rather than of anything anyone does with one.

The consequence a decision feels is one-sided, and it has not been priced. An interval that falls short of the truth 4% of the time and overshoots 1% of the time systematically understates the quantity — so a safety margin set from its upper limit is too small more often than a symmetric account would suggest, and a claim of superiority based on its lower limit is too easy to make. Which of those matters depends on which end the decision reads.

What is missing is the arithmetic that turns the two tail rates into an expected cost under a stated loss, and the comparison of the skew-corrected methods on that scale rather than on coverage. It is possible the corrections that most improve total coverage are not the ones that most improve the side a decision uses, and nothing here would have noticed.

There is a reason to think they are not the same. A correction that widens an interval symmetrically improves total coverage and leaves the ratio alone; a correction that shifts it improves the ratio and can leave the total alone. Those are different repairs and the literature evaluates both against the same total, so a method that does the first well and the second badly scores as well as one that does both. Separating them needs only the second column, which is the habit this whole page argues for and which costs a counter.