A limit from the family it came from

The side a bound is read from compared four upper limits for the mean of a skewed quantity on the side a safety decision reads, and found the best of them to be an oracle: a fixed multiple of $s/\sqrt n$ tuned on the population, which reached exactly 2.5% at fifteen exponential observations with a margin of 0.841 standard deviations. Hall’s transformation, the correction that reads the skew from the sample, spent more margin and failed more often, because the sample skewness is smallest on exactly the samples that miss. The essay ended on the honest version of the oracle: an analyst who does not know the population but knows its family — that the quantity is gamma-shaped, say, as waiting times and concentrations often are — and estimates the one parameter that fixes the family’s shape. Whether such a limit inherits the oracle’s economy or Hall’s overspending turns on how badly the shape is estimated.

It turns, it will emerge, on how the shape is estimated.

A limit that needs only the mean #

For a gamma source the family supplies something better than a correction. If the observations are gamma with shape $a$ and scale $\theta$, their sum $S$ is gamma with shape $na$ and the same scale, so $S/\theta$ has a distribution that does not depend on $\theta$ at all — a pivot. The mean is $a\theta$, and inverting the pivot gives an exact upper limit:

$$U = \frac{a\,S}{G_{0.025}(na)},$$

where $G_{0.025}(na)$ is the lower 2.5% point of a gamma distribution of shape $na$. With the true shape it is exceeded exactly 2.5% of the time, whatever the scale — 2.59% on twenty thousand exponential samples of fifteen, which is 2.5% up to the counting.

It also costs less than the oracle of the comparison on the bound’s side: a mean margin of 0.787 standard deviations at fifteen observations against 0.841 for the tuned multiple of $s/\sqrt n$. The difference is that $U$ uses the mean and not the spread. The fixed multiple divides by a sample standard deviation that is small on exactly the samples whose mean is low — the coupling that unbalances the t interval — and has to compensate with a large multiplier; the family’s pivot never estimates the spread, so the coupling never enters.

The shape, estimated two ways #

The analyst does not know $a$. The family has one shape parameter, and there are two ordinary ways to estimate it from the sample.

From the skewness. A gamma of shape $a$ has skewness $2/\sqrt a$, so $\hat a = 4/\hat g^2$ from the sample skewness $\hat g$. It is the obvious estimator and the one a reader who has just computed a skewness would reach for.

By maximum likelihood. The likelihood equation for the shape is $\log a - \psi(a) = \log \bar x - \overline{\log x}$, where $\psi$ is the digamma function and the right-hand side is the gap between the log of the mean and the mean of the logs. It uses the logarithms of the observations rather than their cubes.

The two estimators read the same fifteen numbers very differently. The likelihood estimate has a median of 1.10, close to the true shape of 1. The skewness estimate has a median of 3.20: it reads the sample as three times less skewed than it is. That is the bias a tail the sample never saw measured for the sample skewness — a median of 1.30 against a true 2 at thirty draws — seen through the reciprocal square that turns a skewness into a shape, which magnifies it. A shape of 3.2 is a distribution much closer to the normal than an exponential is, and a limit built on it is set far too low.

What each limit delivers #

On the same twenty thousand samples of fifteen:

The family limit with the likelihood shape is exceeded less often than Hall’s and spends 28% less margin to do it. It is not the oracle: it fails 4.04% of the time rather than 2.5%, because an estimated shape is sometimes too large. But it sits below and to the left of Hall’s point in the hero figure — safer and cheaper at once — which no correction in the comparison on the bound’s side managed.

The same family limit with the skewness shape is the worst limit on the page, exceeded 17.51% of the time, twice as often as the plain t limit. It has the family’s structure and the wrong parameter, and the parameter matters more than the structure: a gamma limit that thinks the data are nearly normal is a t limit without the t’s modest protection.

At thirty observations the ordering holds and the gaps close. The likelihood family limit is exceeded 3.20% of the time at a margin of 0.470; Hall’s, 3.49% at 0.561; the known-shape limit, 2.59% at 0.483 — so at thirty observations the estimated shape costs less than a point of failure rate and nothing in margin.

A more skewed source #

The advantage is largest where skewness is largest, which is where it matters most. On fifteen observations from a gamma source of shape one half — skewness 2.83, the shape of a squared normal — the t limit is exceeded 11.83% of the time, nearly five times its promise. Hall’s transformation is exceeded 5.12% at a margin of 1.295 standard deviations. The likelihood family limit is exceeded 3.94% at a margin of 0.944, and the known-shape limit 2.57% at 0.984: the estimated shape costs a point and a half of failure rate and saves margin, since an estimated shape that comes out a little large is a limit a little lower. The skewness family limit is exceeded 18.78%.

The likelihood estimate of a shape of one half has a median of 0.54 at fifteen observations. The skewness estimate has a median of 1.79 — it reads a squared normal as something closer to an exponential than to itself, and sets a limit accordingly.

A pivot rather than a correction #

The family limit is not a correction of the t limit, and the difference is worth stating because it explains why it can beat an oracle built from the t limit’s parts.

The t distribution exists because the spread is estimated: $(\bar x - \mu)/(s/\sqrt n)$ is a pivot — its distribution free of every unknown — when the data are normal, and only then. On skewed data it is not a pivot; its distribution depends on the shape, and every construction compared on the bound’s side is an attempt to patch that dependence: widen it, shift it, transform it, or tune its multiplier to one population. Each patch keeps the ratio, and the ratio’s denominator is where the coupling between mean and spread lives.

The family limit uses a different pivot. For gamma data $S/\theta$ is exactly gamma with a known shape once $a$ is fixed, so the limit is exact by the same argument that makes an exact interval for a proportion exact: invert a statistic whose distribution does not depend on the unknown. There is no denominator, no spread, and no coupling. What it needs instead is the shape, and the shape is the one thing the family does not fix — which moves all of the difficulty into a single number, where it can be estimated well or badly and the difference measured, as here.

That is also why the skewness of a difference would change the picture for two groups. A difference of two gamma means is not a gamma quantity, and the exact pivot is lost; a family limit for a difference has to be built by other means, and whether the likelihood shape keeps its advantage there has not been measured.

Why the logarithms know the shape #

The comparison on the bound’s side found Hall’s correction adaptive in the wrong direction: the samples whose limit misses are the samples with no large value in them, which are the samples reporting the least skew, so the correction is weakest where it is needed. The same test can be applied to the two shape estimates here, by splitting the samples by whether the likelihood family limit was exceeded.

On the samples where it was exceeded the likelihood shape has a median of 1.38, against 1.10 on the samples where it held: the estimate drifts the wrong way on the dangerous samples, as Hall’s did, but by a quarter rather than by a factor. The skewness shape on the same samples has a median of 4.04, against 3.17 — already three times too large on the samples that held and larger still on the ones that missed.

The reason is where each estimator looks. The sample skewness is a mean of cubed deviations, dominated by the largest one or two values, and a sample that happens to lack a large value has almost no skewness to report. The likelihood equation uses the gap between the log of the mean and the mean of the logs, which is a property of the whole sample — how spread out the observations are on a multiplicative scale — and is only mildly moved by the presence or absence of one large value. The shape of a gamma distribution is as visible in its bulk, on a log scale, as in its tail, and the likelihood reads it there.

Against the four limits already measured #

Set beside that comparison, the family limit changes what the best available construction looks like. At fifteen observations none of the four measured there came near the promised 2.5% on the upper side except an oracle: the symmetric widening tuned on the population still failed 4.93% of the time, and only the one-sided multiple, also tuned on the population, reached 2.5%. Hall’s was the best a real analyst could build and failed 4.49% of the time, nearly twice as often as promised, at the largest margin of any. The likelihood family limit is a real analyst’s construction, needing only the family, and it fails about one and a half times as often as promised — less often than the symmetric oracle — while spending less margin than the one-sided oracle and much less than Hall’s.

The comparison across the two essays is between different sets of samples, so the digits are not to be subtracted from one another; the ordering is what carries over, and it is the same in every run. A reader choosing among constructions for a real limit on fifteen skewed observations now has one that needs no population and beats the best of the previous four on both of the numbers a safety decision reads.

It does so by giving up the thing all four of those constructions share. Every one of them is centred on the sample mean and scaled by the sample spread; they differ only in how the multiplier is chosen. The family limit is scaled by the mean itself, which on a gamma source carries the scale exactly, and the spread never appears. That comparison’s conclusion — that knowing the shape is worth more than estimating it — is refined rather than reversed: knowing the family is worth nearly as much as knowing the shape, provided the one remaining parameter is estimated from the part of the sample that carries it.

When the family is wrong #

The argument so far had the family right. The same analyst facing a lognormal source — also right-skewed, also positive, and not a gamma — gets a different answer.

On fifteen lognormal observations the likelihood family limit is exceeded 11.29% of the time, at a margin of 0.532; Hall’s transformation, 8.07% at 1.264; the t limit, 14.44%. At thirty observations the family limit is exceeded 10.08% and Hall’s 6.09%. The family limit has lost its advantage and is now between the t limit and Hall’s on the failure rate — cheaper than Hall’s, much less safe.

A lognormal source has a heavier right tail than any gamma with the same mean and spread, so a gamma fitted to it — even perfectly — describes a tail that is too light, and its limit is set too low. The likelihood estimate of the shape is now estimating the wrong thing well. This is the cost of the family, stated exactly: knowing the family buys the oracle’s economy when it is the family and a limit exceeded more than four times too often when it is a plausible neighbour.

What the choice between them comes to #

The three real constructions now sit in a clear order, and the order depends on one thing the analyst has to decide before computing anything.

If the family is known — from the mechanism, from a long record, from physics — use its exact pivot with the shape estimated by likelihood. It is the cheapest limit here that fails near its nominal rate, and at thirty observations it is within a point of the oracle.

If the family is only plausible, the family limit is a gamble on it. A gamma limit on lognormal data fails more than four times too often, and fifteen observations hold little of the far tail in which the two families differ most — the region a sample of thirty was found to barely enter. Even the normal-against-skewed question that a single number for a quantile plot asks is hard at forty; gamma against lognormal is harder. Hall’s transformation, which assumes no family, fails less badly on the wrong family and overspends on the right one.

Never estimate the shape from the sample skewness. Of the estimators here it is the only one that makes the family limit worse than doing nothing, and it is the one most easily reached for.

What the family’s pivot delivers with its shape estimated two ways #

The gamma family’s exact limit with the true shape is exceeded 2.59% of the time on fifteen exponential observations at a margin of 0.787, less than the tuned multiple of $s/\sqrt n$ spent in the comparison on the bound’s side, because it uses only the mean.

With the shape estimated by maximum likelihood it is exceeded 4.04% at 0.746; from the sample skewness, 17.51% at 0.421. Hall’s transformation is exceeded 4.61% at 1.038 on the same samples.

On a lognormal source the likelihood family limit is exceeded 11.29% at fifteen observations and 10.08% at thirty, against Hall’s 8.07% and 6.09%.

Every rate is a count over twenty thousand seeded samples, the same samples for every limit at each setting. The gamma quantiles are exact inversions of the regularised incomplete gamma function, and the likelihood shape is found by bisection on the digamma equation.

Not claimed: that the gamma is the right family for any real quantity, or that likelihood is the best estimator of its shape at fifteen observations — a bias-corrected version would do better still. Not claimed either that the lognormal is the worst plausible neighbour; a source with a heavier tail than the lognormal would make the family limit fail more often.

Still open: choosing the family from the data #

Everything here takes the family as given before the data are seen. An analyst who picks between a gamma and a lognormal by which fits the sample better has made the family a function of the data, and the limit that follows inherits a selection: the family chosen will tend to be the one that makes this sample look typical, which is not the same as the one that sets a safe limit.

How much that choice costs is measurable on sources drawn from both families — how often the fitted family is the wrong one at fifteen observations, and what the limit’s failure rate is after the choice — and it is the question that decides whether a family-based limit is usable by someone who does not already know the family. It is the pre-test problem again, with a likelihood ratio in place of a test, and it has not been measured here.