The plot is about the wrong quantity

A quantile plot of the data is drawn to decide whether a t interval can be used. The interval does not require the data to be normal. It requires the sampling distribution of the mean to be normal, which is a different quantity with a different answer.

The table that does not sort twice #

Three thousand samples of forty from each. The first column is what a reader sees; the second is what they are trying to predict.

If the plot answered the question it is used for, sorting the table by either column would give the same order. Sorted by the plot, the order is two lumps, lognormal, t on 3 df, uniform, normal. Sorted by coverage, worst first, it is lognormal, two lumps, normal, uniform, t on 3 df. Four of the five rows change position, and the two that swap places at the top are the whole difficulty: they look equally bad and only one of them is.

The two-lump source fails the plot on every single sample and its interval covers 94.80%. Forty observations drawn from two clusters at ±1 produce a quantile plot with an unmistakable kink, which any reader would reject — and the mean of forty such draws is very nearly normal, because averaging forty values from a symmetric bounded distribution is exactly the case the limit theorem is fastest on.

The t on three degrees of freedom fails the plot on 57% of samples and has the best coverage of the five, at 95.73%. Its tails are heavy enough that its fourth moment does not exist, its quantile plots are visibly S-shaped, and the interval is faintly conservative — which is the symmetric-source behaviour a split coverage count measures.

The lognormal fails both. Skew is the departure that breaks the interval, and it also produces the worst plot, so the one source where the plot’s verdict is right is the source where a reader needed the least help.

What the interval actually needs #

Three conditions, and only the third is about the data’s shape at all.

The mean’s distribution close enough to normal. That is what the limit theorem supplies, at a rate that depends on the source’s skewness and almost not at all on its tails. A symmetric source’s mean is very nearly normal at small n; a skewed source’s mean is not normal at much larger n.

The mean and the spread close enough to independent. They are independent only for a normal source, and the correlation converges to a constant rather than to zero under skew. This is where the real damage is.

The spread’s distribution close enough to $\chi^2$. This is the one the tails affect, and it is the least important: a heavy-tailed source’s sample variance is much more variable than a normal’s, which widens the interval on average and leaves its coverage roughly right.

So the ranking of departures by how much they matter is skew first, dependence second, tails a distant third — and a quantile plot shows all three with about equal prominence. It is a display of the data’s shape, and the data’s shape is three things of which the reader needs one.

The plot gets better and the problem gets smaller #

The two columns move in opposite directions with the sample size, which is the second reason the plot is a poor instrument for this.

At ten observations the two-lump source fails its band on 10% of samples. At twenty, 88%. At forty and above, 100%. The plot’s power to detect non-normality grows with the sample.

At the same time the interval’s coverage is 94.40% at ten, 94.83% at twenty, 94.80% at forty and 94.23% at a hundred and sixty. The interval works throughout, and its problems — where it has them — shrink with the sample.

So the plot is least informative exactly where the question is live, and most emphatic exactly where the question has gone away. A reader following the plot rejects normality on larger samples, which is where the limit theorem has already repaired the thing the rejection was about.

That is the same structure as a pre-test for a variance assumption and a pre-test for a tolerance factor, and it recurs for one reason: a test’s power depends on the sample size and the assumption’s consequences depend on it the other way.

What “the plot looks wrong” is actually detecting #

The five rows differ in what makes their plots fail, and separating the causes explains why the column carries so little information about the interval.

The two-lump source fails on shape. Its quantile plot has a flat step in the middle where the gap between the clusters is, and every sample shows it because the gap is a feature of the distribution rather than of the draw. The plot is detecting multimodality, which is a real and important property of the data — and one the interval for a mean is entirely indifferent to, because a mean averages over it.

The t on three degrees of freedom fails on tails. Its plot is S-shaped, the departure lives in the extreme few points, and it appears on about half of samples because a given sample may or may not contain an extreme draw. The plot is detecting kurtosis, and the interval is robust to kurtosis.

The lognormal fails on skew. Its plot bows one way throughout, and skew is the one departure that matters. This is the row where the plot’s verdict and the interval’s behaviour agree, and it agrees for a reason rather than by luck.

The uniform fails mildly on short tails. 17.5% of its samples leave the band, its plot is S-shaped the other way, and its interval covers 95.33% — very slightly conservative, which is what short tails do.

So the column is a mixture of four detections of which one matters. The plot is reporting the union of several properties and the interval depends on one of them, and no amount of care in reading the plot separates them unless the reader already knows which shape to look for — which is knowing the answer.

What to plot instead #

The question is about the mean’s distribution, so plot the mean’s distribution.

From one sample of n, resample with replacement, take each resample’s mean, and make a quantile plot of a thousand of those. That is a plot of the sampling distribution the interval depends on, it needs no assumption about the data, and it is one step past what a bootstrap already computes.

Three things are worth knowing about it before it is recommended without qualification.

It answers the right question. A two-lump source’s bootstrap-mean plot is a straight line at n = 40, which is the correct verdict. A lognormal’s is visibly bowed, which is also correct.

It inherits the sample’s limitations. A bootstrap resamples the observed values, so it cannot see tail behaviour beyond the largest observation — and at small n a skewed source’s sample often contains no large value, so its bootstrap-mean plot looks better than it should. The failure is in the optimistic direction, which is the wrong one.

It is not a test. The plot still has to be read, and reading it has all of the band problems a pointwise envelope has.

The simpler alternative is to skip the diagnostic entirely and use a method that does not need it. A skew-corrected bootstrap interval costs resampling and holds its coverage on all five sources, and the decision to use one can be made from the subject matter — a positive, one-sided quantity is skewed — without any plot at all.

The one row that matters, priced #

The lognormal is the only source of the five whose interval genuinely fails, so it is worth seeing how far that failure goes and what a reader would have to notice to catch it.

At forty observations its interval covers 88.93% against a claimed 95%. That is a six-point shortfall, which is large — a nominal 95% interval behaving like an 89% one — and it is worse than the total suggests, because the misses are almost all on one side.

To catch it from the plot, a reader has to notice the bow specifically and not merely that the plot is unacceptable. And 99.6% of the lognormal’s samples leave the band, against 100% of the two-lump source’s, so the strength of the verdict is no help at all: the two look equally rejected and one interval works.

That is the page’s finding in its most compressed form. The plot is emphatic about both and correct about one, and it says nothing that distinguishes them except the shape, which is not what a band reports and not what a normality test returns.

Why the plot is used anyway #

Three reasons, and the first is the honest one.

It is the only diagnostic that is free. The data are already there and the plot takes a line. A bootstrap-mean plot needs a thousand resamples, which is nothing on a computer and is a step nobody takes by default.

The textbook assumption is stated about the data. “The observations are normally distributed” is how the t interval’s conditions are written, and it is a sufficient condition rather than a necessary one. A reader checking the stated condition is checking the right thing for the wrong reason: if the data are normal, everything follows exactly. The condition is simply far stronger than needed.

It catches other problems. A quantile plot with a staircase in it says the data are discrete; one with a hard edge says they are censored or bounded; one with a lump at the end says there is a recording convention. Those are real findings, they are worth having, and none of them is about normality. Two of the three are worse problems than non-normality would have been — a censored sample has no unbiased mean at all, and a discrete quantity’s intervals behave badly for reasons of their own.

That last one is the reason not to stop plotting. The plot is a good general instrument and a bad specific one, and the mistake is only in reading its verdict as an answer to the interval’s question.

What a reader should take from a plot they have already drawn #

The plot is drawn, it looks bad, and the question is what to do. Four steps, and none of them is about the verdict.

Name the departure. Bow, S, kink, staircase, hard edge. That word is worth more than the plot’s overall impression, and it is what decides everything that follows.

If it is a bow, take it seriously. A bow is skew, skew is the departure the interval is fragile to, and the remedies are real: a transformation, a skew-corrected bootstrap, or a different estimand.

If it is an S, a kink or a staircase, note it and carry on with the interval. Heavy tails, multimodality and discreteness are all findings about the data worth reporting, and none of them breaks a mean’s interval at a sample size where they are visible.

If it is a hard edge, look at the data-collection rather than the distribution. A boundary in a quantile plot is usually censoring, a detection limit or a recording convention, and it is a much larger problem than normality — the mean of a censored sample is not an estimate of the mean of anything.

The reordering is the content. The plot’s shape is highly informative and its verdict is not, and current practice reads the verdict.

What is claimed here, and what is not #

Two statements, and the second is the finding in the only form that survives.

The normal source’s interval covers at its nominal rate, to within two points. That is the calibration: coverage counted wrongly would produce a plausible table with every number slightly out, and this is what would show it.

Some source whose plot looks worse than another’s has an interval that covers better. Stated as the existence of an inversion rather than as a comparison of the two extremes, because the extreme case is not the finding — the lognormal is worst on both columns, and comparing only the worst of each would be satisfied by a table that sorted perfectly apart from one row. What the page’s claim needs is that the two orderings differ somewhere, and the t on three degrees of freedom against the uniform is where.

The reading that does not survive is the plot’s verdict taken for the interval’s. It would require a source failing the plot to have an interval that fails, and the two-lump row settles that directly: 100% of its plots are rejected and its interval covers 94.80%, which is inside the noise of the normal source’s own 95.20%.

Where the plot’s question is the right one #

Nothing above says a quantile plot of the data is useless, and there are procedures whose assumption really is about the data. Naming them keeps the rule from being over-applied.

A prediction interval for one future observation is about a single draw from the population, so its assumption is about the population’s shape directly. No averaging happens, no limit theorem applies, and a prediction interval on a skewed source misses entirely on one side. For that interval, the plot answers exactly the right question.

A tolerance interval is about a share of the population, and its factor assumes normality twice over. The same applies.

Any statement about a quantile — a reference range, a percentile, a threshold — is about the distribution’s shape, and the plot is the right diagnostic for all of them.

So the division is clean and it is the same one that separates the three bands an earlier essay draws apart. A statement about a parameter estimated by averaging inherits the limit theorem and does not need the data to be normal. A statement about the data does, and cannot be rescued by sample size. The plot is drawn most often for the first case and is the right instrument for the second.

Still open: how much departure the interval can take #

The table has five sources and the sources were chosen to make a point. What a reader needs is a threshold: how much skew is too much at a given sample size.

The pieces are here. The relevant quantity is the source’s skewness γ, the sample size enters as $\gamma/\sqrt{n}$ in the leading correction to the mean’s distribution, and the interval’s coverage shortfall is approximately proportional to that. So a rule of the form $\gamma^2/n$ below some constant is the shape of the answer, and the constant is measurable by sweeping γ and n and finding where the coverage crosses a stated tolerance.

What makes it more than an exercise is that γ is estimable from the sample — badly at small n, but estimable — so the rule would be checkable from the data in hand. That would be a diagnostic answering the interval’s own question, computed from one number rather than read off a picture, and it would be the thing a quantile plot is currently standing in for.

Whether the sample skewness is estimated well enough at the sizes where it matters is the obstacle, and it is the same obstacle every pre-test runs into: the quantity that decides whether a method works is hardest to measure exactly when the method is most fragile.

There is one reason to think this case is better than the others. The rule would be a continuous adjustment rather than a switch — the interval widened or shifted by an amount depending on the estimated skewness — and a continuous adjustment degrades gracefully when its input is noisy, where a switch between two methods does not. A badly estimated skewness would produce a slightly wrong correction, and a badly estimated pre-test produces the wrong method entirely. That distinction is what makes a skew-corrected bootstrap work in practice while a normality pre-test does not, and the rule above would be the closed-form version of the same idea.