The same correction, inverted

A correction that goes below zero measured the Edgeworth expansion’s first term on an exponential sum, used as it usually is: added to the normal tail probability at a threshold. It repaired the probability near the centre, reached 3.09 standard deviations within 10% at a hundred draws, and in the short tail produced probabilities below zero. It also named an inverse use it did not measure. The same term can be applied to a quantile instead — correcting the critical value a test is compared with, rather than the probability read at a fixed one — and the resulting critical value has an actual size that can be counted against its nominal level.

That inverse is the Cornish–Fisher expansion, and it is how the skewness correction is most often used in practice, since a test needs a critical value and a report needs a level. This essay measures it on the same exponential source, where every tail is an exact incomplete gamma function, and finds that inverting the correction changes its reach far more than a change of bookkeeping ought to.

The critical value, corrected #

A one-sided test at level $\alpha$ on a standardised statistic compares it with the normal quantile $z = \Phi^{-1}(1 - \alpha)$. If the statistic is skewed, with skewness $\gamma$ and excess kurtosis $\kappa$, the Cornish–Fisher expansion corrects that quantile:

$$c_1 = z + \frac{\gamma}{6}(z^2 - 1), \qquad c_2 = c_1 + \frac{\kappa}{24}(z^3 - 3z) - \frac{\gamma^2}{36}(2z^3 - 5z).$$

For the sum of $n$ exponential draws $\gamma = 2/\sqrt n$ and $\kappa = 6/n$. The actual size of the test that rejects above $c$ is the exact probability that the standardised sum exceeds $c$, which is an incomplete gamma function and carries no approximation.

At thirty draws and a 5% upper test the normal critical value is 1.645 and the exact one is 1.742. One term gives 1.749 and two give 1.742. The actual sizes:

The normal critical value misses a 0.5% upper test by a factor of two and a genome-wide threshold of $5 \times 10^{-8}$ by a factor of two hundred. One Cornish–Fisher term brings the 0.5% test to within half a per cent of nominal and the genome-wide one to within a third. The hero figure shows the whole range: on the upper side the one-term line stays near one until well past $10^{-5}$ and drifts to 1.42 at $10^{-8}$; the normal’s climbs to 464.

The same term, reaching further #

Measured in standard deviations rather than levels, the difference from the direct correction is large and consistent.

The two middle columns use the same term — $\gamma(z^2 - 1)/6$, the skewness correction and nothing else — and at a hundred draws the version that moves the critical value reaches two standard deviations further than the version that adds to the probability. In levels that is the difference between about $10^{-3}$ and about $1.7 \times 10^{-7}$: four orders of magnitude of tail over which the inverted correction is good to 10% and the direct one is not.

The lower tail tells the same story from a shorter start: at a hundred draws the direct term reaches 2.58 standard deviations and the inverted one 4.31.

Why inverting helps #

The two uses are not algebraically the same approximation, and the difference is where the correction is put.

The Edgeworth correction adds a term to the tail probability, $\varphi(z)\,\gamma(z^2 - 1)/6$. The normal tail itself is about $\varphi(z)/z$, so relative to the tail the correction is about $\gamma z(z^2 - 1)/6$ — a polynomial that grows like $z^3$ and eventually exceeds one. Once it does, the corrected probability is more correction than probability, and in the short tail it goes below zero, which is what the essay on the direct correction found.

The Cornish–Fisher correction moves the threshold by $\Delta = \gamma(z^2 - 1)/6$, and a tail moved by $\Delta$ changes by a factor of about $e^{-z\Delta}$. To first order that is the same $\gamma z(z^2 - 1)/6$ — which is why the two agree near the centre — but in the exponent rather than added on. A large correction becomes a large factor, which is what a far tail needs: the exact tail of a skewed sum differs from the normal’s by a factor that grows like $e^{\gamma z^3/6}$, and a correction that lives in the exponent can follow that where an additive one overshoots. The inverted expansion is, in effect, the direct one exponentiated.

That is the same lesson the saddlepoint taught from a different direction. An approximation to a tail is good when its error is a relative one that does not grow, and each improvement in this line of essays — from the normal to the one-term correction, from the direct correction to the inverted one, from both to the saddlepoint — is a step towards putting the correction where the tail’s own behaviour lives, which is in the exponent.

Where it turns back #

The inverted correction has a flaw of its own, and it lives in the short tail, where the direct correction’s did.

On the lower side the normal quantile $z$ is negative and the skewness term $\gamma(z^2 - 1)/6$ is positive, so the correction pulls the critical value up towards the centre — correctly, since a sum of exponentials has a short lower tail. But the term grows like $z^2$ while $z$ itself only grows like $z$, and the corrected value

$$c_1(z) = z + \frac{\gamma}{6}(z^2 - 1)$$

has derivative $1 + \gamma z/3$, which is zero at $z = -3/\gamma = -1.5\sqrt n$. Below the level $\Phi(-1.5\sqrt n)$ the one-term critical value rises as the level falls: a stricter test is given a laxer threshold.

At ten draws the turning point is a level of $1.05 \times 10^{-6}$, where the critical value reaches its lowest point, −2.477. The exact critical value there is −2.756 and keeps falling. So a lower-tail test at $10^{-7}$ built on one term rejects 1,149 times as often as it claims, and at $10^{-8}$, 21,383 times. Two terms postpone the collapse without preventing it: at $10^{-8}$ they reject 62 times too often.

The turning level recedes quickly with the sample — at thirty draws it is about $10^{-16}$ and at a hundred about $4 \times 10^{-51}$ — so it matters only for short sums read very far into their short tail. But inside that region the failure is worse than the direct correction’s. A negative probability announces itself; a critical value of −2.46 looks like a critical value, and the test built on it reports a level four orders of magnitude smaller than the one it actually has.

Short of the turning point the lower side is still where the correction is weakest. At thirty draws a lower-tail test at $5 \times 10^{-8}$ has 4.52 times its nominal size with one term, against 1.33 on the upper side at the same level, because the true short tail falls off faster than any polynomial correction to a normal quantile can follow as it approaches the hard edge at $-\sqrt n$.

Ten draws, at ordinary levels #

The extreme levels are where the difference between the two uses of the correction is largest, but the reason to use the inverted one at all shows at levels every analyst reads.

On a sum of ten exponential draws, which is a small study of waiting times or a short run of counts, a lower-tail test at 5% built on the normal critical value has an actual size of 2.51%, and at 1% an actual size of 0.043% — twenty-three times too strict. It is the mirror of the upper side, where the same normal value gives 6.36% at a nominal 5% and 2.17% at 1%. A two-sided test built on the normal quantile therefore puts almost all its errors on the long side, which is where the two tails disagree measured for the t interval, and a reader looking only at the two-sided total sees a test that is roughly right.

One term repairs both sides at once: 4.72% and 0.975% on the lower side at nominal 5% and 1%, and 4.85% and 0.976% on the upper. Two terms put all four within a few hundredths of a point. At ordinary levels on short sums, the correction is not a refinement. It is the difference between a test whose level is what it says and one whose level depends on which way the effect points.

This is also where the direct correction was at its best, since ordinary levels are close to the centre, and the essay on it found one term nearly as good there. The inverted correction’s advantage at ordinary levels is not accuracy but form: it produces a critical value directly, which is what a test needs, whereas the direct one produces a probability at a threshold that then has to be searched for the level the analyst wanted — and the search passes through the region where the direct correction’s probability can be below zero.

What the second term adds #

The second Cornish–Fisher term carries the kurtosis and the square of the skewness, and it does two different things on the two sides. On the long side it extends the reach dramatically, from 5.10 to 8.63 standard deviations at a hundred draws, because the exact tail’s excess over the normal is not purely a skewness effect and the $\gamma^2$ term captures the next piece of the exponent. On the short side it postpones the turning point without removing it. At ten draws the two-term critical value stops falling at a level of $4.05 \times 10^{-10}$, at −2.795, where the one-term value stopped at $1.05 \times 10^{-6}$ — and long before that it is falling more slowly than the exact value, so its test grows too lax, 62 times too lax at $10^{-8}$, well before it turns.

That asymmetry is characteristic of polynomial corrections to a skewed distribution. The long tail of a gamma sum is close to exponential in shape and a polynomial correction to a normal quantile follows it well; the short tail ends at a hard edge, $-\sqrt n$ standard deviations, and no polynomial in $z$ can bend to meet an edge. The tail that converges last found the normal approximation weakest in the tail; the corrections here find the short tail the harder one to repair, even though it is the one in which the normal approximation’s errors are conservative.

What a test built this way can claim #

The practical reading is a good deal more favourable than the one for the direct correction.

A one-sided test on a skewed statistic, at a conventional level, should use the corrected critical value. At thirty draws one term brings a 0.5% upper test from twice its nominal size to within half a per cent of it, and the lower-tail version from a fifth of its nominal size to within a fraction of a per cent. The correction needs only the skewness, which for a sum of known source is known and for a sample is estimated — with the downward bias that a tail the sample never saw measured, which will pull the corrected critical value back towards the normal one and part of the way back to the normal one’s error.

At extreme levels on the long side, one term is good to a factor of about 1.4 and two to within 10%, down to $10^{-8}$ at thirty draws. That is the region of genome-wide and particle-physics thresholds, and it is well outside the reach of the direct correction, which at thirty draws is good to 10% only to 2.82 standard deviations.

On the short side, the one-term critical value must not be used below $\Phi(-1.5\sqrt n)$, and should be distrusted well before it. A lower-tail test at an extreme level on a short sum is a case for the exact distribution or the saddlepoint, both of which were measured on this source and neither of which turns back.

And the turning point is a hard limit, not a safe range. The one-term correction cannot be used past $1.5\sqrt n$ standard deviations on the short side, but it stops being good long before that. At ten draws the bound is 4.74 standard deviations and the test’s size leaves 10% of nominal at 2.76; at thirty the bound is 8.22 and the size leaves at 3.34; at a hundred, 15 against 4.31; at a thousand, 47 against 7.48. The measured reach grows far more slowly than the bound, so the bound tells a reader where the formula becomes nonsense and not where it becomes trustworthy — for that the measured reaches above are the guide, and beyond them the exact tail, computed as a sum or a root-find, is not expensive and is the only answer that cannot turn back.

The same idea in the bootstrap #

The Cornish–Fisher expansion also explains a fact about the bootstrap that is usually stated without a reason. A bootstrap percentile interval for a skewed statistic is approximately the normal interval with its limits corrected by the statistic’s estimated skewness — it is a Cornish–Fisher correction computed by resampling instead of by formula, which is why its errors on a skewed source are about the size of the one-term errors in the tables above, and why the studentised bootstrap, which corrects the quantile of a pivot rather than of the estimator, does better — the repair an interval that carries its scale measures.

The comparison runs the other way as well. Where the bootstrap has no resamples — beyond one over the number of resamples, and beyond the largest value the sample holds, which is where the bootstrap lies — the formula still gives a critical value, and at levels like $10^{-7}$ that is the only reason a skewness-corrected threshold exists at all. The formula’s reach, measured here, is what a reader relying on it at those levels is relying on.

How far the inverted term reaches, and where it turns back #

One Cornish–Fisher term keeps a one-sided test’s actual size within 10% of nominal out to 5.10 standard deviations at a hundred exponential draws, where the same term applied to the probability reaches 3.09; at thirty draws the reaches are 4.17 and 2.82, and two terms on the quantile reach 6.16 and 8.63 at thirty and a hundred.

At thirty draws the normal critical value gives a 0.5% upper test an actual size of 1.031% and one term gives 0.4978%; at $5 \times 10^{-8}$ the normal is 212 times too large and one term 1.33 times.

The one-term lower critical value turns back at the level $\Phi(-1.5\sqrt n)$ — $1.05 \times 10^{-6}$ at ten draws — and a test at $10^{-7}$ built on it has 1,149 times its nominal size.

Every size is an exact incomplete gamma function evaluated at the corrected critical value, and every level below $10^{-10}$ is computed from the normal tail directly rather than by inverting a normal quantile, which loses accuracy there. The reaches are scanned in steps of 0.01 standard deviations, out to twelve.

Not claimed: that these reaches carry to sources other than the exponential. The source here has a skewness large enough to make the corrections matter and a generating function smooth enough for them to work; a lattice source, or one whose skewness is estimated from a small sample, will reach less far, and the gap between the direct and inverted corrections on such sources is not measured.

Still open: a critical value for a studentised statistic #

Every critical value here corrects the distribution of a sum with its spread known. A t statistic divides by an estimated spread, its skewness has the opposite sign to the source’s and a larger magnitude, and its Cornish–Fisher correction has an extra term from the correlation between the mean and the spread. That is the correction a skew-adjusted t test uses, and it is the one the side a bound is read from needed and did not have.

Whether the inverted correction for a studentised statistic reaches as far past the direct one as it does for a sum — and whether its short tail turns back at a comparable level — is measurable on the same source, against t tails computed exactly by integrating over the sample variance, and has not been measured. It would say whether a one-sided safety limit on thirty skewed observations can be set by a formula at levels where the resampling that currently sets it runs out.