A ratio whose interval has to be the whole line

A ratio of two estimated means is one of the most common numbers in applied work, and it is usually reported under another name. A relative potency is the dose of one preparation over the dose of another that does the same thing. A cost-effectiveness ratio is extra cost over extra benefit. A fold change is one mean over another. An instrumental-variable estimate with one instrument is a reduced-form effect over a first-stage effect. The number needed to treat is one over a difference in risks. And the location of a fitted optimum is one regression coefficient over another, which is where the problem first appeared: the delta method’s interval for that location covered 68.8% where the curvature was weak, and Fieller’s set covered 95% by being unbounded on most datasets.

That essay established that one exact construction beats one approximate one. It left open whether the delta method was simply a poor approximation — one that a wider interval, a better standard error or a bootstrap might repair — or whether something about a ratio forbids a finite answer altogether. The second is true, and it is a theorem. This essay computes the theorem rather than quoting it.

Two estimates and the plane they are drawn in #

Strip the problem to its skeleton. Two independent estimates, each normal with a known standard error, divided by those standard errors so that each has variance one: zx with mean a, and zy with mean d. The target is the ratio of the means, ρ = a/d. Every ratio of two normal estimates with known standard errors becomes this one after scaling, so nothing is lost by it, and everything becomes computable.

The figure below is that plane at a = 3 and d = 1: a numerator three standard errors from zero, a denominator one standard error from zero. Each dot is one pair of estimates.

The exact interval comes from refusing to divide. At the true ratio, zx − ρ·zy has mean zero and variance 1 + ρ², so

$$\frac{z_x - \rho\, z_y}{\sqrt{1+\rho^2}} \sim N(0,1)$$

exactly, whatever a and d are. Fieller’s set is every ρ at which that pivot is under z in size. Squaring the condition gives a quadratic inequality in ρ,

$$(z_y^2 - z^2)\,\rho^2 - 2 z_x z_y\,\rho + (z_x^2 - z^2) \;\le\; 0,$$

and a quadratic inequality has three kinds of solution. If zy² > z², the leading coefficient is positive and the set is a bounded interval — which happens exactly when the denominator is significantly different from zero. If zy² ≤ z² but zx² + zy² > z², the set is everything outside an interval. If zx² + zy² ≤ z², the set is the whole line. In the plane those are the region outside a horizontal band, the band with a disc cut out of it, and the disc.

Because the pivot is exactly normal the set covers exactly 95%, and because the regions are a band and a disc their probabilities are closed forms: one minus a non-central χ² distribution function on one degree of freedom in d², and a non-central χ² distribution function on two degrees of freedom in a² + d². At a = 3 and d = 1 they are 17.01% bounded, 75.03% an exterior and 7.96% the whole line; counted on 40,000 draws, 16.89%, 74.79% and 8.32%. On those draws Fieller covers 95.21% of the time, and the delta interval ρ̂ ± z·√(1 + ρ̂²)/|zy| covers 82.48%, against a quadrature of the same interval’s coverage of 82.36%.

Widening the delta interval rescues nothing #

Hold the numerator at three standard errors and walk the denominator towards zero. Fieller’s counted coverage never leaves the band from 94.90% to 95.21% across twelve denominators, which is sampling error around an exact 95%.

The delta interval covers 95.61% at d = 8 and 95.06% at d = 4, where the denominator is so well determined that the ratio behaves. Then it falls: 90.78% at d = 2, 82.36% at d = 1, 67.85% at d = 0.5, 51.63% at d = 0.25, 33.96% at d = 0.1, 24.31% at d = 0.05, 15.48% at d = 0.02, 10.97% at d = 0.01, 3.48% at d = 0.001 and 1.10% at d = 0.0001.

The natural response to an interval that covers too little is to make it wider, and the figure tries that twice. Three times as wide, the interval covers 98.70% at d = 1 — briefly better than its promise — then 55.42% at d = 0.1, 18.88% at d = 0.01 and 1.90% at d = 0.0001. Ten times as wide, it covers 84.55% at d = 0.1, 33.79% at d = 0.01, 10.96% at d = 0.001 and 3.48% at d = 0.0001. Widening moves the curve to the right. It does not change where the curve is going.

A percentile bootstrap, which resamples the two estimates and reads the ratio’s 2.5% and 97.5% points, is a different kind of interval with the same property that matters: its answer is always finite. On 1,000 datasets with 399 resamples each it covers 95.1% at d = 4, 97.4% at d = 1, 86.3% at d = 0.25, 14.3% at d = 0.05 and 0.0% at d = 0.01. It gives up faster than the delta interval, for a reason the bootstrap’s other failures here share: it resamples a statistic whose distribution has no second moment and whose extreme quantiles are exactly the part a finite number of resamples reads worst.

How fast a wider interval loses #

The shape of the fall can be worked out, and it says why no fixed widening can win. Far from the flat part of the curve, the true ratio ρ is enormous. The delta interval can contain it only if its half-width is of order ρ, and its half-width is about √(zx² + zy²)/zy², so the denominator’s estimate has to land within about √(kz·|zx|/ρ) of zero. For a normal estimate that has probability proportional to the width of the window, which gives

$$\text{coverage} \;\approx\; 2\varphi(0)\,\sqrt{kz/\rho}\;\,\mathrm{E}\,|z_x|^{1/2}.$$

At a = 3 the half-moment E|zx|^½ is 1.7048, and the formula agrees with the quadrature to better than 2% at all three widths at d = 0.0001. Its content is in the square root. A tenfold widening buys a factor of √10 and no more: 1.10% becomes 3.48% at the smallest denominator, a ratio of 3.16. To hold its coverage as ρ grows the interval would have to grow in proportion to ρ, and ρ has no largest value. Any rule that fixes its width in advance, or ties it to the data in a way that is finite on every dataset, is overtaken somewhere.

The theorem, in two steps #

That argument is a calculation about one family of intervals. Gleser and Hwang proved in 1987 that it holds for every interval procedure that always returns a finite set, and the proof is short enough that the figure below can evaluate it.

Step one: nearby laws cannot be told apart. The pair of estimates at (a, d) and the pair at (a, 0) are two normal laws whose means are d apart. The largest difference between the probabilities they give any event — the total variation distance — is 2Φ(d/2) − 1: 38.29% at d = 1, 3.99% at d = 0.1, 0.40% at d = 0.01 and 0.004% at d = 0.0001. So for any interval rule at all, the probability that its interval contains ρ = a/d when the truth is (a, d) is at most the probability of the same event when the truth is (a, 0), plus that distance.

Step two: at a zero denominator a finite interval cannot reach a huge ratio. When the truth is (a, 0) the ratio does not exist, and the rule still returns some finite interval on every dataset. The probability that this interval happens to contain the number a/d goes to zero as a/d goes to infinity, simply because a finite interval has to stop somewhere.

Put together, coverage at (a, d) is squeezed under something that goes to zero as d does. For the delta interval at ten times its width the figure draws each piece: at d = 0.01 the coverage is 33.79% and the bound is 34.19%; at d = 0.0001 they are 3.476% and 3.480%. The first step costs almost nothing: the coverage and the bound differ by little more than the distance itself, so the whole of the failure lives in the second step — a finite interval, computed from data that cannot tell d from zero, failing to reach a ratio that grows without limit.

Nothing in either step mentions linearisation, standard errors, resampling or normality of anything but the two estimates. The only property used is that the answer is always finite. So the only way to cover a ratio at every value of its parameters is to return an unbounded set on some datasets, and Fieller’s construction is not one repair among several but a representative of the only kind that can work. Dufour extended the same argument in 1997 to instrumental variables, where the estimate is a ratio whose denominator is a first stage — the same object whose just-identified estimate has no mean at all.

What a reader sees depends on the path to zero #

The theorem is about the worst case over every value of the parameters, and it is worth being careful about what that does and does not imply for any one study. The figures above hold the numerator fixed and shrink the denominator, so the ratio runs away to infinity. Shrink both together, holding the ratio fixed, and the delta interval does something else.

Held at a ratio of three, its coverage falls from 82.36% at d = 1 and settles at 60.85%, however small d gets. Held at a ratio of zero it rises instead, to 99.991% — which is exactly the number the squared mean’s delta interval covers at its flat point, for the same reason: with both means at zero the question of whether the interval contains zero is the question of whether za·zb/√(za² + zb²), a normal with variance ¼, exceeds 1.96.

That is how a single weak instrument can make the conventional interval over-cover without contradicting anything here. Along a path where the effect is held fixed while the first stage shrinks, the reduced form shrinks with it and the ratio stays put, so no single sweep sees the collapse. The collapse is there along the path where the ratio itself is large, and an interval’s promise is a promise about every parameter value, not about a favourable path through them.

Near zero, the honest answer is mostly an exterior #

With the denominator a quarter of a standard error from zero the cloud of estimates has moved into the band. Fieller’s set is a bounded interval on 5.72% of datasets, an exterior on 83.92% and the whole line on 10.36%.

The stacked shapes show where those numbers come from and where they stop. At d = 2 the split is 51.60% bounded, 45.08% exterior and 3.32% the whole line. As d goes to zero the bounded share falls to exactly 5.00%, which is the denominator’s own test rejecting at its size, and the whole line settles at 10.54%, the probability that a χ² on two degrees of freedom with non-centrality nine falls under z². The rest, 84.46%, is exteriors.

An exterior is easy to dismiss as a non-answer and it is not one. It says that the ratio is not between two particular numbers, and that on either side of that gap it could be anything, of either sign, as large as wished. The number needed to treat has exactly this shape whenever a trial’s interval for the risk difference straddles zero: some benefit, some harm, and a gap of values that the data rule out passing through infinity in the middle. Reported as an interval from the estimate’s two ends, the same information becomes a claim of precision the trial does not have. A finite interval that looks informative is often the shortest one and the one that misses, and here the looking-informative is guaranteed by construction.

The bounded answers are the ones to distrust #

Overall, Fieller’s set covers 95% at every denominator. That average is made of three kinds of answer, and they do not share it. The whole line covers every time it is given. The exteriors cover 72.03% of the time at d = 4, where there are only 801 of them in 40,000, then 96.95% at d = 1, 98.64% at d = 0.25 and 99.92% at d = 0.01.

The bounded intervals carry the difference. Taken alone they cover 95.45% at d = 4, 84.40% at d = 1, 32.59% at d = 0.25 and 1.18% at d = 0.01. The last is a quadrature over the rotated pair of pivots; counted, 2,012 bounded answers at that denominator covered 0.89% of the time. A reader handed a tidy, finite Fieller interval from a study with a weak denominator is holding one of the rare datasets on which the denominator’s estimate happened to land far from zero, and the interval it produced almost never contains the truth.

This is the winner’s curse in a different costume. Selecting the studies that reached significance selects the ones whose estimates were inflated by noise; selecting the Fieller answers that came out bounded selects the ones whose denominators were. In both cases every individual computation is correct, the unconditional promise is kept, and the subset that looks most informative is the subset that is least reliable. It is also why an interval’s 95% belongs to the procedure and not to the interval in hand: this procedure’s 95% is an average of 100% and 1.18%.

Where weak denominators actually occur #

The failure needs one thing: a denominator whose estimate has appreciable probability near zero, which is the same as saying the denominator is not clearly established. That is the ordinary state of several quantities that are reported routinely. A relative potency from an assay whose reference preparation has a shallow dose-response. A cost-effectiveness ratio where the incremental benefit is small. A mediated share where the total effect is near zero. An instrumental-variable estimate with a weak first stage. The number needed to treat for a treatment whose effect is uncertain. The location of an optimum on a flat response surface.

In every one of them the delta method’s interval is finite on every dataset, and so in every one of them the theorem guarantees a region of the parameters where it fails as badly as it likes. And in every one of them the ratio’s sampling distribution has tails like a Cauchy’s, which is the case the central limit theorem simply does not reach: a ratio of two normal quantities centred near zero does not settle towards anything as data accumulate, because accumulating data makes the denominator’s estimate more precise around a value that is itself near zero.

The escape, where one exists, is outside the data. If the denominator can be ruled out of a neighbourhood of zero on grounds nobody disputes — a dose-response known to be steep, a treatment known to have some effect — then the parameter space no longer reaches the boundary, the theorem’s second step no longer applies, and a finite interval can be exact over what remains. The theorem forbids a finite interval that covers everywhere; it does not forbid one that covers everywhere a stated assumption allows. What it forbids is leaving the assumption unstated.

What is proved, what is computed and what is counted #

The impossibility is proved, by Gleser and Hwang, and does not rest on anything computed here. What is computed is the theorem’s inequality for one family of intervals, by quadrature on a step of a thousandth of a standard error, and the far-out rate, which comes from a first-order argument and is checked against the quadrature rather than bounded. The three shape probabilities are closed forms in non-central χ² distribution functions and are counted again. Fieller’s exactness is proved by its pivot and counted at twelve denominators; the bounded answers’ conditional coverage is a one-dimensional quadrature and is counted where there are enough bounded answers to count.

Two refusals sit beside those claims, each applying a standard to something that must fail it. The standard Fieller’s set passes — coverage within a point of 95% at every denominator — is applied to the delta interval at ten times its width, and rejects it at 3.48%. And Fieller’s own set, truncated to the range from −1,000 to 1,000 so that its answer is always finite, is held to the same standard and rejected: at d = 0.001 the ratio is 3,000, outside every truncated answer, and its coverage there is zero. That second refusal is the theorem in its smallest form. Truncation changes nothing on any dataset whose answer was already bounded, and it is enough on its own to destroy the guarantee.

Where this goes next #

The exterior is an answer nobody has a standard way to report. It is a finding — the ratio is not in this gap — but it is not the kind of finding a table has a cell for, and the obvious reduction is to ask a bounded question about an unbounded parameter: the sign of the ratio. The sign of a/d is the product of two signs, each with an ordinary test, and a statement like “the ratio is positive” can be made with a stated error rate and without any interval at all. Whether that statement can be made on the datasets where Fieller returns an exterior, how often, and what it costs, is a distinct question from anything here, because it gives up the size of the ratio deliberately rather than failing to find it.

Beyond that lies the case this essay removed by assumption: standard errors that are estimated rather than known, and a numerator and denominator estimated from the same data and therefore correlated. Fieller’s construction survives both, with a t reference and a covariance term in the pivot, and so does the theorem. What changes is the geometry of the three regions, which stop being a band and a disc — and with many instruments rather than one, where the ratio becomes a vector, the Anderson–Rubin set replaces Fieller’s and can be unbounded in some directions and not others.