A correction that goes below zero

A bound written for a coin found that the Berry–Esseen guarantee is the wrong instrument for a smooth skewed source, and found the right one by accident: the worst distance between an exponential sum and the normal was predicted to four digits by the first term of an expansion. The tail converges last names the same expansion as the systematic repair and warns, in a sentence, that it can go negative in the far tail. Neither measured what the repair does in the place a probability is read.

This essay does, on the same source — a sum of exponential draws, whose exact tail is an incomplete gamma function and can be evaluated at any threshold to full precision.

What the expansion adds #

The normal approximation reads a standardised sum’s distribution off its first two moments and nothing else. The Edgeworth expansion writes the true distribution function as the normal plus corrections carrying the higher moments, in powers of $1/\sqrt{n}$:

$$F_n(z) \;=\; \Phi(z) \;-\; \varphi(z)\,\frac{\gamma\,(z^2 - 1)}{6\sqrt{n}} \;-\; \varphi(z)\left[\frac{\kappa\,(z^3 - 3z)}{24\,n} + \frac{\gamma^2\,(z^5 - 10z^3 + 15z)}{72\,n}\right] \;-\; \cdots$$

with $\gamma$ the source’s skewness and $\kappa$ its excess kurtosis. For exponential draws $\gamma = 2$ and $\kappa = 6$. The first bracket is the one-term correction and carries the skew; the second carries the kurtosis and the square of the skew, and is smaller by another factor of $\sqrt{n}$.

Each correction is a polynomial in $z$ multiplied by the normal density. That shape is the whole story of this essay. Where $z$ is moderate the polynomial is modest and the density is not small, so the correction is a sensible adjustment. Where $z$ is large the density still decays like the normal’s, but the polynomial grows, and the correction ends up larger than the thing it corrects.

Two standard deviations: the repair works #

At the threshold a two-sided 5% test reads, the correction does what it is advertised to do.

These are ratios at two standard deviations above the mean, where the exponential sum’s long tail is. At ten draws the normal gives 0.617 of the true tail — a p-value that should be 3.69% reported as 2.27%. One term moves that to 1.08; two terms to 1.01.

The first correction removes at least two thirds of the normal’s error at two standard deviations, and more as the sample grows — 68% at five draws, 95% at a hundred — and the second removes most of what is left. At a thousand draws the one-term correction is exact to the digits drawn while the normal is still 7% short. The skewness term is doing exactly the work the theory assigns it: the normal approximation’s leading error is the skew, and adding the skew back removes it.

One reading, at thirty observations #

The ratios are easier to feel as a single decision. Thirty measurements of a right-skewed quantity — waiting times, say, which is what an exponential is — and a one-sided test whose statistic lands 2.58 standard errors above the null mean. The normal table says the p-value is 0.50%.

The exact tail of the thirty-draw sum at that threshold is 1.03%. The p-value is twice what it was reported as, and a result that looked comfortably past a 1% threshold is not past it. One Edgeworth term with the true skewness gives 1.00%; with two terms, 1.06%. The first correction has taken a factor of two out of a reading at a threshold people actually use, at a sample size people actually have.

With the skewness estimated from those thirty observations, the corrected reading lands at a median of about 0.84% — most of the way from the normal’s answer to the true one, not all of it, for the reason the section on estimated skewness below measures. Every version of the correction does better than the table, and none of the numbers on this page change the direction of that comparison at a threshold this close in.

That is what makes the rest of the essay worth reading carefully. The correction is not a curiosity; at two to three standard deviations it is the difference between a reported p-value and a true one on the most ordinary kind of skewed data. The question is how far out that stays true.

Further out: the repair runs out #

The hero figure shows what happens past two standard deviations, and the table puts numbers on it at a hundred draws.

Every correction improves on the normal at every threshold here, and every one of them eventually falls away from 1. At four standard deviations a single term gives 61% of the tail. At six, two terms give 21% of it: better than the normal’s 0.7%, and still wrong by a factor of five.

The cleaner way to see it is to ask how far out each one can be trusted to within 10%:

A single correction roughly doubles the reach of the normal approximation, which is a large practical gain: at a hundred draws it moves the trustworthy range from a threshold nobody reads to one past a two-sided 0.5% test. But every column still grows slowly. A hundredfold increase in the sample takes the one-term range from 3.09 to 5.64 standard deviations, and the ten-draw row shows two terms reaching less far than one — the second correction overshoots at 2.7 standard deviations before it undershoots further out.

Why the range grows so slowly #

The rate at which that range grows is set by the shape of the corrections, and it is worth making explicit.

The normal tail beyond $z$ is about $\varphi(z)/z$. The $k$-th correction is a polynomial of degree $3k - 1$ times $\varphi(z)$, divided by $n^{k/2}$. Divided by the tail it is correcting, it behaves like

$$\frac{z^{3k}}{n^{k/2}} \;=\; \left(\frac{z^3}{\sqrt{n}}\right)^{k}$$

so the whole series is organised in powers of $z^3/\sqrt{n}$, not of $1/\sqrt{n}$. The expansion is useful only while $z^3$ is small compared with $\sqrt{n}$, which is to say while $|z|$ is small compared with $n^{1/6}$.

The sixth root is a brutal rate. To move the useful range of any fixed number of terms out by a factor of two takes sixty-four times the data. Six standard deviations corresponds to $z^3 = 216$, which is small compared with $\sqrt{n}$ only once $n$ is well past fifty thousand. The table agrees: at a hundred thousand draws two terms are good to 10.38 standard deviations, and at a hundred they are not good to five.

This is also the precise sense in which the expansion is asymptotic rather than convergent. At a fixed sample size, adding terms improves the approximation near the centre and does not rescue the far tail, because every term is a larger power of the same ratio, and past the point where that ratio exceeds one the terms grow instead of shrinking. The series is a statement about what happens as $n$ grows at a fixed threshold. A tail probability is read at a threshold that has to move out as the probability gets smaller.

The short tail #

Everything above is the long tail, where the exponential sum has more probability than the normal and the corrections add some back. The other side is where the expansion stops being a probability at all.

A sum of positive draws cannot be negative, so its standardised version has a floor at $-\sqrt{n}$ — at $-2.24$ for five draws — and the true lower tail is very short: two standard deviations below the mean it holds 0.022% of the distribution. The normal approximation, which has no floor, puts 2.28% there, which is 103 times too much.

The one-term correction subtracts the skew from the normal, which is the right direction. It subtracts too much. At two standard deviations below the mean it gives −6.33 times the true tail — a negative probability — and two terms give −30.8 times. The correction has overshot zero, and the second term, which helped so much on the long side, makes it five times worse on this one.

Nothing is wrong with the arithmetic. The correction is a polynomial times a density and has no reason to respect the fact that a probability cannot be negative; it was built to be accurate to a stated order near the centre and says nothing at all about the edges. What goes wrong is only the reading: a number between −1 and 0 printed where a p-value was expected.

The negative region never goes away #

It would be natural to assume the negative region is a small-sample curiosity that disappears once there are enough draws. It recedes. It does not disappear.

The one-term lower tail is $\Phi(z)$ minus the skew correction, and $\Phi(z)$ falls like $\varphi(z)/|z|$ while the correction is $\varphi(z)$ times a quadratic. They cross where $|z|(z^2 - 1) \approx 3\sqrt{n}$, so roughly at $|z| = (3\sqrt{n})^{1/3}$ — the same sixth root of $n$ again.

At a hundred draws the region begins where the true tail is a little over two in ten thousand. That is not exotic. It is the size of a p-value in a study with a dozen comparisons corrected for the number of analyses that were run. At a thousand draws the region has moved to probabilities below one in a million, where it troubles only the most extreme applications. At every size it is there.

Two terms behave better on this side at moderate sizes — from thirty draws on they stay positive across the whole short tail — which makes the point about asymptotic series in the other direction. Adding a term can repair a defect, create one, or both, depending on where it is read, and nothing in the formula says which.

The skewness it would really be given #

Every number so far hands the correction the true skewness of the source, which is 2. Nobody using the correction has that. They have a sample, and the correction’s coefficient is whatever that sample’s skewness happens to be — so the measurement that matters in practice is the correction with the skewness estimated from the same observations it is applied to.

The sample skewness of a long-tailed source is biased low, for the reason an interval’s tails disagree on the same source: the skew is made by rare large values, and a small sample usually does not contain one. At ten exponential observations the median sample skewness is 0.93 — less than half the truth — and one sample in ten reads below 0.20. At thirty the median is 1.38; at three hundred, 1.85, still short.

The correction inherits that bias directly, because its size is proportional to the skewness put into it:

At ten observations the estimated correction recovers a little over half of the normal’s shortfall rather than all of it and more. It is still an improvement — the normal gives 62% of the tail, the corrected reading gives 83% at the median — and on only 4.9% of samples is it further from the truth than the normal was. The repair is real and it is systematically partial, pulled back towards the approximation it was correcting by exactly the amount the sample understates the skew.

Two consequences follow for how the expansion is read. The first is that the tables above, computed with the true skewness, are the correction’s best case at small samples, and the gap between the fourth and fifth columns is the price of not knowing the source. The second is more useful: the bias is in a known direction. A skew-corrected tail computed from a small sample of a right-skewed quantity is too small, not merely uncertain, and a reader who has nothing better can at least read it as a lower bound on how much the normal understated.

The same bias explains a pattern the resampling essays keep meeting. A method that estimates the skewness from the sample and corrects for it — the accelerated bootstrap’s constant is one such estimate — gets most of the repair at moderate samples and a fraction of it at small ones, and the bootstrap’s reach into a tail is limited by the same missing large values that bias the skewness here.

Where a skew correction does its useful work #

The measurements make the correction’s proper use easy to state. It is a repair for moderate thresholds on skewed sources, and at moderate thresholds it is very good.

Its most important application is not a tail probability at all but a critical value. The Cornish–Fisher expansion is the same series inverted: it adjusts the quantile rather than the probability, and a skew-corrected critical value for a one-sided test at 2.5% sits inside the range where the one-term correction is at its best. The imbalance between an interval’s two tails on an exponential source — 4.08% missed on one side against 1.11% on the other at a hundred and twenty observations — is a first-order skew effect of precisely this kind, and it is what a skew-corrected interval is built to remove.

The same series is also the reason some resampling methods are more accurate than others. A bootstrap that resamples a studentised statistic gets the skewness term right automatically, where the plain normal interval and the percentile interval do not, and that is the content of the claim that the studentised bootstrap is “second-order accurate” — a claim whose price is measured elsewhere. Second-order accuracy means exactly the first correction and nothing more, which the tables here show is very good at two standard deviations and not good at six.

What the correction should not be used for is a far-tail probability read off its own formula. Past $|z| \approx n^{1/6}$ the corrections dominate the quantity they correct, the ratio to the truth can take any value including negative ones, and there is no internal signal that the region has been entered.

What the counts establish, and what they do not #

One correction removes most of the normal’s error at two standard deviations. At a thousand draws it takes the ratio from 0.931 to within a hundredth of 1, and at every size in the tables the one-term error at two standard deviations is under a third of the normal’s.

The one-term lower tail turns negative at every sample size measured — five, thirty, a thousand and a hundred thousand draws — at a threshold that moves further out as the sample grows and sits within a tenth of a standard deviation of the cube-root prediction at each.

One term reaches further into the upper tail than the normal does, at every size. Stated as a comparison at each size because the ranges themselves are properties of the exponential source, and a different skew would move all of them.

What is not claimed is anything general about the series beyond these corrections. The divergence of the full Edgeworth series for a fixed sample size is a known result about the expansion and is not measured here; what is measured is that two terms do not reach six standard deviations at a hundred draws. Nor is it claimed that the negative-probability region is a practical hazard at large samples: at a thousand draws it begins at a probability of $7.9 \times 10^{-7}$, and whether any reading happens that far out depends on the application.

Every number is exact rather than simulated. The exponential sum’s tail is an incomplete gamma function, the corrections are closed forms, and the table of ranges is a search over those closed forms — so, like the figures in the collection’s methods essay, each can be recomputed independently and none carries Monte Carlo error.

Still open: an approximation built at the threshold #

Every correction here is built at the centre of the distribution and extrapolated outward, which is why its error is organised in powers of $z^3/\sqrt{n}$ and grows with the threshold. The remedy is to build the approximation somewhere else.

A saddlepoint approximation tilts the distribution so that the threshold being read becomes its centre, applies the normal approximation there — where the normal is good — and tilts back. The error is then an expansion about the threshold rather than about the mean, and it does not have to grow as the threshold moves out. In the range figure above it reaches the edge of the search at every size. Whether that holds on a source where it is not favoured, and what it costs to use on a statistic whose moment generating function is not written down, are the questions an approximation built at the threshold takes up.

The inverse question is also untouched. A Cornish–Fisher critical value is the one-term correction applied to a quantile, and it inherits the sixth-root range; the one-sided test it produces has an actual size that could be counted against its nominal one at each threshold, exactly as the ratios here were, and the point past which it starts to mislead has not been located.