The side a bound is read from

Where the two tails disagree split a t interval’s misses by side and found them unequal on skewed data, and it ended on a question it could not answer from coverage alone: what the imbalance costs someone who uses the interval. A decision rarely reads both ends. A safety margin reads the upper limit, a claim of superiority reads the lower one, and the literature that evaluates corrections for skew scores them on the total.

The suspicion stated there was that the corrections that most improve total coverage are not the ones that most improve the side a decision reads. This essay counts it.

The decision that reads one end #

Take thirty measurements of a right-skewed quantity — a contaminant concentration, a repair time, a cost — and set a limit from the upper end of a 95% interval for its mean. The limit claims that the true mean exceeds it no more than 2.5% of the time. That is the promise a regulator, a warranty or a budget is built on, and it is a one-sided promise — the same kind of promise a one-sided p-value makes about its own error rate.

The t interval’s upper limit breaks it on 6.38% of samples. At fifteen observations, 8.31%; at a hundred and twenty, still 4.19%. The mean sits above a limit that was supposed to contain it between one and a half and three and a third times as often as promised, and the opposite end — which nobody in the decision reads — is correspondingly too safe.

A coverage table reports this interval as covering 92.88% at thirty observations, which reads as a modest shortfall that a slightly wider interval would fix. Whether a wider interval fixes it depends on how it is widened.

Four ways to build the interval #

The comparison is between four constructions, chosen because each represents a kind of repair.

The t interval, $\bar x \pm t\,s/\sqrt{n}$, is the baseline.

Widened to exactly 95%. The same interval with its multiplier raised until the total miss rate on these very samples is 5.00%. No analyst can do this, since it requires knowing the population; it is here because it is the best a symmetric repair can possibly do. Any method that works by widening both ends equally — a larger quantile, a small-sample adjustment to the degrees of freedom, a variance inflation — is doing a worse version of this.

Shifted by the skewness term. The t interval moved upward by $\hat\gamma\, s / (6n)$, where $\hat\gamma$ is the sample skewness. This is the centring half of Johnson’s 1978 correction for skew; the other half, a quadratic term, makes the statistic non-monotone and is left out, which is how it is usually applied.

Hall’s transformation. A cubic transformation of the studentised mean, from 1992, that carries the same skewness information as Johnson’s but is monotone and so can be inverted exactly. It does not move the interval rigidly; it stretches one end and compresses the other.

What each one does to the side #

The widened interval achieves the total exactly and fixes less than half of the side. Its upper limit is still exceeded on 4.69% of samples, nearly twice the promise. What it did with the extra width is visible in the second column: the high misses fell from 0.74% to 0.30%, so a large share of the widening went to the end that was already too safe.

The skewness shift barely moves anything. Moving the whole interval up by a term of order $1/n$ is the right direction and the wrong size: at thirty observations with a sample skewness near two, it moves the centre by about a hundredth of a standard deviation.

Hall’s transformation is the only one that changes the ratio. Its upper limit is exceeded on 3.31% of samples — still above 2.5%, and about a fifth of the t interval’s excess — while its lower limit is exceeded on 1.96%, much nearer its own 2.5%. It has a worse total than the widened oracle, 5.27% against 5.00%, and on the dimension a decision uses it is the best of the four by a distance.

That is the suspicion confirmed with numbers: ranked by total coverage, the widening wins; ranked by the side a bound is read from, it comes third.

As the sample grows, and on a more skewed source #

Hall’s transformation is nearest 2.5% at every size, and at a hundred and twenty it is there to within the counting error. The widened interval converges slowly, and not because widening is failing to converge: its total is exactly 5% at every size. It converges only because the skew itself fades as the sample grows. At fifteen observations the widened interval puts 4.93% of its 5.00% total on the low side — ninety-nine misses in a hundred on one end.

On a lognormal source, more skewed than the exponential, the pattern is the same and larger. At thirty observations the t interval’s upper limit is exceeded 11.93% of the time, the widened interval’s 5.00% — every one of its misses on that side — and Hall’s 6.17%. The best symmetric construction there is puts all of its misses where they hurt.

The t interval’s own imbalance is what the other three are repairing, and on the lognormal it is extreme enough to see without a table.

The low curve starts near seventeen per cent and is still near five at five hundred observations; the high curve is almost flat on the floor. Every construction above begins from that picture. Widening lifts the whole interval’s reach on both sides, so the low curve comes down and the high curve, already near zero, has nowhere to go and the widening there is wasted. A shift moves the pair together. Only a construction that treats the two ends differently — by stretching the upper end more than the lower — can move the low curve down without spending on the high one, which is what a transformation does and a correction for an estimated spread does not.

The same shape governs the two-sample case. When two groups are compared, the imbalance is set by the skewness of the difference of their means, and a symmetric reference distribution for that difference fails the same way a symmetric interval for one mean does here.

What the margin costs #

A safety limit is not free: a higher limit is a more expensive product, a larger reserve, a stricter standard. So the fair comparison is not only how often each limit is exceeded but how high each puts the limit to get there.

The margins, in population standard deviations above the true mean, at fifteen observations:

The tuned fixed multiplier is an oracle, like the widening: it is the constant $k$ in $\bar x + k\,s/\sqrt{n}$ that this population needs, 3.43 at fifteen observations where the t interval uses 2.14. It marks what an honest one-sided limit costs when its shape is fixed — 60% more margin than the t interval spends.

The surprise is Hall’s row. At fifteen observations it spends more margin than the oracle — 1.044 standard deviations against 0.841 — and is exceeded more often, 4.49% against 2.50%. It pays more and gets less.

Why the correction that reads the sample overspends #

The reason is the one the Edgeworth correction ran into when its skewness was estimated rather than given. Hall’s transformation reads $\hat\gamma$ off the sample, and the sample skewness of a long-tailed source is biased low and highly variable: at fifteen observations it is often a fraction of the true value of two.

Now consider which samples produce an interval that misses low. They are samples with no large values in them — a low mean, a small spread, and, for exactly the same reason, a small sample skewness. The coupling between a sample’s mean and its spread that unbalances the t interval extends to the third moment: the samples on which the correction is most needed are the samples that report the least skew. Counted at fifteen observations, the samples whose Hall upper limit the mean exceeded have a median sample skewness of 0.71; the samples whose limit held have 1.15; the population’s is 2. On those samples Hall’s transformation barely adjusts, and they miss. On the samples that do contain a large value, the sample skewness is large, the transformation stretches the upper end a long way, and it spends margin on intervals that would have covered anyway.

So the correction is adaptive in the wrong direction. It is generous where generosity is wasted and stingy where it would have helped, and its average margin overstates the protection it provides. The fixed oracle does the opposite: it spends the same multiple of $s$ on every sample, and since $s$ is itself small on the dangerous samples, a large fixed multiplier lands its margin where the misses are.

The figure measures the bias on a different quantity — a tail probability rather than an interval’s end — and the mechanism is the one at work here. The sample skewness of a long-tailed source is short of the truth on most samples and very short on the samples that contain no large value, so any correction proportional to it is short on exactly those samples. The interval version adds the conditioning: the samples that are short of large values are also the ones whose intervals sit too low.

At sixty observations the overspending has nearly gone — Hall spends 0.325 standard deviations against the oracle’s 0.316 and is exceeded on 2.89% — and at a hundred and twenty the two are indistinguishable. The failure is a small-sample failure of the skewness estimate, not of the transformation.

Pricing a limit under a stated loss #

Margin and exceedance can be put on one scale once a loss is stated. Suppose each decision pays its limit’s margin, measured in standard deviations of the quantity, and pays a penalty every time the true mean turns out to exceed the limit. Call that penalty $c$, in the same units. The expected cost of a construction is its mean margin plus $c$ times its exceedance rate, and each pair of constructions has a break-even $c$ above which the more cautious one is cheaper.

At thirty exponential observations:

So at thirty observations Hall’s transformation is the cheapest of the three real constructions once an exceedance costs more than about eleven standard deviations of margin, and the plain t interval is cheapest only when an exceedance costs less than about three. For a safety limit, where an exceedance is the event the limit exists to prevent, a penalty of eleven margins is modest.

At fifteen observations the arithmetic changes because of the overspending measured above. The break-even between the widened interval and Hall’s rises to 84.8: Hall spends 0.377 more standard deviations of margin to cut the exceedance rate by less than half a point. Below that penalty the symmetric widening is cheaper, if it could be had — and it cannot, because it is tuned on the population. Against the t interval, which can be had, Hall’s transformation breaks even at a penalty of 13.6.

The tuned one-sided multiplier is the cheapest of all four at a penalty of twenty or fifty at both sizes, and at a penalty of five at thirty observations; at fifteen observations and a penalty of five the widened interval edges it, 0.914 against 0.966. That is the formal version of the conclusion below: once an exceedance is expensive, knowing the shape is worth more than estimating it, and the smaller the sample the more it is worth.

Two cautions about the table itself. The penalties are in units of the quantity’s own standard deviation, so the same penalty means different things for quantities of different spread, and a reader has to translate a real cost into those units before reading off a row. And the rates and margins are counted over twenty thousand samples each, so a break-even that rests on a difference of a few tenths of a point — the 84.8 at fifteen observations is one — is itself uncertain by a large factor, and should be read as “large” rather than as that number.

What this means for setting a limit #

Three conclusions, in order of how widely they apply.

Evaluate a one-sided procedure on its one side. A method compared on total coverage can win by a margin and be worse for the purpose it is used for. The widened oracle, which no real method beats on the total, is beaten on the upper side by a method whose total is worse.

A symmetric interval on a skewed quantity cannot be repaired symmetrically. Every symmetric construction’s best case is the widened row, and its best case at fifteen exponential observations still puts ninety-nine of every hundred misses on the side a safety limit reads.

At small samples, a skewness correction estimated from the sample is not a substitute for knowing the skew. If the quantity’s skewness is known from the subject — a waiting time is close to exponential, a concentration close to lognormal — a one-sided multiplier built for that shape, like the tuned row, reaches the promised rate with less margin than a correction that has to discover the shape from fifteen points. Where it is not known, Hall’s transformation is still the best of the four on the side that matters; the reader should expect it to overspend.

What the counts establish, and what they do not #

The widened interval misses exactly 5% in total at every size, by construction, and at every size it puts a larger share of its misses on the low side than the t interval does. Both halves are stated together because the first is what makes the second a statement about symmetric repair rather than about an interval that happens to be too narrow.

Hall’s transformation brings the low side nearer 2.5% than the t interval and than the skewness shift, at every size from fifteen to a hundred and twenty on an exponential source.

Reaching 2.5% on the upper side takes more margin than the t interval spends, and the widened interval spends less than that and misses more than 2.5% — at every size and on both sources.

What does not survive is an interval widened until its total coverage is right, read as a safe upper limit. At thirty exponential observations its upper limit is exceeded on 4.69% of samples against 2.5%.

Not claimed: that Hall’s is the best available skew correction. A studentised bootstrap and a bias-corrected and accelerated bootstrap both carry the same skewness information by resampling, and the resampling essays price them on other grounds; they have not been counted on this comparison. And the two oracles are yardsticks, not methods — each was tuned on the samples it is scored on, which is exactly what no analysis can do.

Every rate is a count over twenty thousand samples, with a counting error of about 0.11 points near 2.5% and 0.17 points near 6%, so differences of a tenth of a point between rows are not read as findings.

Still open: a limit tuned to the shape rather than to the sample #

The tuned fixed multiplier did best, and it cheated: it knew the population. The honest version of it knows only the population’s family — exponential, lognormal, gamma of some shape — and estimates the one parameter that decides the multiplier. Whether a limit built that way inherits the oracle’s efficiency or Hall’s overspending depends on how badly the shape parameter is estimated at fifteen observations, and that can be counted exactly as the four limits here were, and has not been.

The other direction is a limit that is wrong on purpose. The pricing above held every construction at its nominal 97.5% and compared the costs that followed. Under a stated penalty the cheapest limit is generally not at 97.5% at all: each construction has its own best level, found by moving its quantile until the marginal margin equals the marginal exceedance saved, and the comparison that matters is between the four constructions each at its own best level. That optimisation is a one-dimensional search per construction on counts this essay already has, and it has not been run.