A floor that reads the first stage

A fixed-width interval is a promise made before the data: the estimate will be within $d$ of the truth with a stated probability. Without knowing the spread, nobody can say how many observations that takes, and the rules that try sit between two kinds of failure. Stein’s two-stage rule reads the spread once and is exact, at twice the observations the spread would have asked for. The purely sequential rule keeps reading, spends about what it should, and misses its level, for reasons the essay on the stop it chose took apart.

The fifth a third stage never reaches tested the rule in the middle. Take a first stage, then take a fraction of what its spread asks for, re-read the spread on everything, and take the rest. At a half-width of 0.4 spreads and a first stage of five, that covers 88.4%, worse than the sequential rule it was meant to repair. The cause is a specific one: in a fifth of experiments the first spread reads low enough that half of what it asks for is already in hand. The second stage takes nothing, the third stage reads the same five observations again, and those experiments stop at about eight observations and cover about 70%.

The repair that essay found was a floor: make the second stage run to some fixed number of observations whatever the first reading says. At about the number the oracle would ask for, it works, and it works better than any constant added at the end. And it was left with an embarrassment. A floor set at the right size has been set by someone who already knows roughly what the spread is, which is the one thing the rule was invented to do without.

Its closing suggestion is the natural one. Make the floor a function of the first stage. Run the second stage to a fraction $\rho$ of the target computed with Student’s $t$ on $n_0 - 1$ degrees of freedom in place of the normal point, so that a low first reading is inflated by Stein’s own factor exactly when it is the only reading the rule has. That keeps the rule free of any guess, and it aims the inflation at the stuck experiments alone. Whether it beats the fixed floor at equal spend, and whether its coverage holds across the range of requirements without any constant at all, is a computation the same integral can make.

It is, and it does not, on either count. What is worth carrying away is why.

One change to one number #

Write $s_1^2$ for the first stage’s spread. The three-stage rule’s second stage runs to

$$T = \max\bigl(n_0,\ \lceil \rho\, z^2 s_1^2 / d^2 \rceil\bigr),$$

and the proposal replaces $z$ with $t_{n_0 - 1}$. At a first stage of five, $t = 2.776$ against $z = 1.960$, and so the target doubles: $(t/z)^2 = 2.006$. Nothing else changes. The third stage still reads the normal point on the spread of all $T$, the final interval is still the fixed half-width $d$, and since every quantity in the rule is a spread, the final mean is still independent of them all. So the coverage is still the average of $2\Phi(d\sqrt{N}) - 1$ over the distribution of $N$ and can be computed exactly on a grid of first readings. Every number in this essay is exact in that sense, which matters for a comparison at equal spend, since there is no noise for a difference of a tenth of a point to hide in.

The change does what it sets out to do, to a point. At the half-width of 0.4 and a fraction of a half, coverage rises from 88.4% to 92.1%, and the experiments left stuck fall from 20.3% to 6.6%. On the requirement that the rule was failing, that is most of a repair, and no number has been guessed.

The spend, however, rises from 0.86 of the oracle to 1.17.

At equal spend #

The figure at the head of this essay is the comparison the proposal has to survive: coverage against spend at this one requirement, with three ways to pay. The fixed floor runs from nothing to twenty-four observations. A constant adds to the third stage’s target in every experiment, which is what the earlier essay showed a floor beats. The new rule’s dial $\rho$ runs from 0.2 to 1.

The read rule is below the constant almost everywhere and well below the floor. At about 1.17 times the oracle:

That is a difference of more than four points against the floor, and the direction holds at every requirement. Reading the whole grid, from a half-width of 0.7 down to 0.15, the fixed floor at equal spend covers between 1.6 and 4.3 points more:

To reach 95% on the read rule’s dial at this requirement takes $\rho = 0.77$ and 1.65 times the oracle, against about 1.08 for a floor. The two-stage rule, which is exact and has no dial, costs 2.03. The proposal sits about two thirds of the way from the floor to Stein, which is not where anybody hoped to find it.

A low reading times two is still a low reading #

The reason is visible once the stuck experiments are read separately.

At the half-width of 0.4 the read rule leaves 6.6% stuck where the old rule left 20.3%. Those that remain, though, cover 62.9%, where the old rule’s stuck experiments covered 70.4%. The same is true at every requirement: stuck 13.9% covering 73.6% at 0.5, stuck 2.4% covering 49.8% at 0.3, stuck 0.1% covering 26.3% at 0.15.

This is not a paradox, and it says what the repair cannot do. An experiment is stuck when $\rho t^2 s_1^2 / d^2 \le n_0$. Doubling the multiplier moves the line, so that only lower readings get caught. But the experiments below the new line are exactly those with the lowest readings of all, and they read lower than the group that was stuck before. Their third stage reads the same five observations, and it reads them with the normal point, and asks for very little. A factor multiplies whatever it is given, and what it is given in the stuck experiments is the near-zero reading that is the problem.

A floor is different in kind. It does not look at the reading at all, and so the very experiments whose reading is worthless are the ones that gain the most from it: they get fresh observations and a new reading of the spread on all of them, regardless. What the fixed floor buys that an inflated reading cannot is a second reading in every experiment, and that is what breaks the dependence on a bad first one. The rule was trying to solve a problem of too little information about the spread by rescaling the information it had.

The rescaling is also spending in the wrong place. The factor applies to every experiment, not just the low readers. A high first reading has its second stage doubled too, past the point where the third stage would have asked for those observations anyway, and those observations are not recoverable. That is where the 1.17 goes.

A rule with no constant, and a constant it needs #

The second half of the proposal was that the rule would hold across requirements with no constant. Here the dial itself can be set to reach exactly 95%, by bisection on the exact coverage, at each requirement.

It needs 0.64 at a half-width of 0.7, 0.79 at 0.5, 0.77 at 0.4, 0.68 at 0.3, 0.60 at 0.25, 0.51 at 0.2 and 0.43 at 0.15. So the fraction for the level is nearly twice as large at one end as the other. The spend at that fraction falls from 1.68 to 1.15 across the same range.

Which means that $\rho = 0.5$ overcovers a demanding requirement, at 95.5% with 1.25 times the oracle at 0.15, and undercovers an easy one, at 91.8% with 1.16 at 0.5. To set $\rho$ for the level, a user has to know which requirement they are facing in units of the spread, and that is the same knowledge the floor was accused of needing. Nothing has been gained on this count either.

The rise at the left of the figure is worth a sentence, because it looks wrong. At very easy requirements the oracle asks for only a few observations, the second stage is so often stuck or nearly so that the third stage does most of the reading, and the first stage of five is itself most of what is needed. The rule has less to do there, and so a smaller fraction will do.

What a floor costs when it meets the wrong spread #

All of this leaves the original accusation against the fixed floor standing. It works, but somebody has to set it. So the fair comparison is not between the read rule and a floor set correctly; it is between the read rule and a floor set for the wrong spread. That can be computed directly. Every rule here depends on $d$ and $\sigma$ only through their ratio, so a floor set for a half-width of 0.4 and a spread of one, then met by a spread of two, is the same floor at a half-width of 0.2.

The floor is set at 26, which is about right for 0.4. Take the two kinds of mistake one at a time.

The spread turned out larger than was assumed, by a factor of up to about 2.7, which puts the requirement at 0.15 and an oracle of 171 observations. The floor of 26 is then small relative to what the experiment needs, and the rule falls back towards its old behaviour. But the stuck experiments it was protecting against are also rarer there, at under one per cent. Coverage is 94.1% to 94.4%, about a point below the level, at almost exactly the oracle’s spend. The read rule at the same requirements covers 94.1% to 95.5% at 1.22 to 1.25 times. On a thousand-observation experiment that difference in spend is hundreds of observations for one point of coverage.

The spread turned out smaller than was assumed, by up to a factor of about 1.75, which puts the requirement at 0.7 and an oracle of 8 observations. The floor then wastes 18 observations in an experiment that needed eight. As a ratio it looks awful, at 3.32 times the oracle and covering 100.0%. As a count it is 18 observations. And it fails safe: the promise is overkept. The read rule, in the same situation, spends 1.19 times the oracle and covers 93.8%.

So the floor’s guess is less delicate than the argument against it assumed. Getting the spread wrong in the high direction costs about a point of coverage, near the oracle’s spend. Getting it wrong in the low direction costs a bounded number of observations and over-covers. The read rule is only better when a reader counts ratio of observations at the easy end of the table and ignores the absolute number, and it has its own failure of more than three points at the middle of the table, 91.8% at a half-width of 0.5.

The practical upshot is unusual in this line of essays: the rule with no knowledge in it is not the better rule, and a rough idea of the spread is worth having and costs little to be wrong about. A floor of about one oracle’s worth of observations, set from a pilot, a similar past trial, or a reasonable upper bound of the requirement, holds coverage within a point across a range of nearly a factor of five in the requirement. Most experimenters have at least that much knowledge. It is the same conclusion, in a different setting, as how many subjects reaches about a planning value: its exactness matters less than having one.

The case Stein’s factor was made for #

It is worth asking why a device that does this much for the two-stage rule does so little here, since the proposal’s whole appeal was that Stein’s factor is what makes the two-stage rule exact.

In the two-stage rule the factor applies to the final sample size, and it is compensating for a final interval whose half-width will be read on a spread with $n_0 - 1$ degrees of freedom. The whole size of the experiment is being chosen by that one reading, and the $t$ point is the exact price of that one reading’s uncertainty. It does not fix a low reading any more than here. A two-stage experiment that reads low stops early, and it covers less, and the guarantee holds because the high readers overcover by precisely enough. The rule’s coverage is a mixture.

The three-stage rule spends that same factor differently. Its third stage reads the normal point, so the over-coverage from high first readings is mostly lost: the third stage re-estimates the spread and brings the sample back near the target. The low readers remain low, and nobody balances them. A factor that is exact where one reading decides everything is doing nothing in particular where three readings decide things, and the observation that it “reads with Stein’s own factor” was more like a pun than a theorem.

A stricter form of the repair would apply $t$ at the third stage as well, on the $T - 1$ degrees of freedom available there. That just inflates the end of the experiment, and the essay on the constant has already measured what inflating the end of the experiment does.

How the numbers were made #

Coverage and spend come from the integral of the earlier essay: the first stage’s sum of squares on a grid of 800 of its own quantiles, the second stage’s size determined by it, the third stage’s by a chi-square on the second stage’s added degrees of freedom. The only change for the read rule is the point used at the second stage. The comparison floor is found by bisection on whole observations to match the read rule’s spend, then interpolated between the two nearest whole floors, which is why floors in the tables are not integers.

Two routes meet at the old rule: 88.4% and 0.86 reproduce the earlier essay’s exact values, and Stein’s 2.03 at a first stage of five reproduces that field’s. Four claims are required to hold. Reading with $t$ must leave fewer experiments stuck and cover more than reading with $z$, at every requirement. The experiments still stuck must cover less than before, at the central requirement. And the fractions the level needs must differ by more than 0.3 across requirements. The reading the essay tested, that a factor on the reading covers as much as a floor at the same spend, is refused at all seven requirements.

Everything here is for normal observations with a first stage of five, and the exactness depends on normality: it is the independence of mean and spread that makes the coverage an average over the sample size. Nothing here says what happens under a skewed law, where the spread and the mean are not independent, and the earlier essays’ shortfall from reading the spread on the observations would return.

Still open: a second reading of everyone, and the pilot worth paying for #

A floor set from the first stage’s own width. The finding is that a second reading is what helps and that a factor on the first reading is not. So the rule to try is a second stage of fixed size $m$ with no reference to the first reading at all, which has no units in it and can be said in advance; a third stage then reads the spread on $n_0 + m$. That is the same thing as a first stage of $n_0 + m$ and a two-stage rule with $z$, which looks too simple to be different from what is measured here, and it is short work to check. If it is not different, it means that the whole three-stage construction is a fixed floor by another name.

What a pilot is worth. The guess above was an arbitrary one. A pilot of $m$ observations supplies a reading of the spread with known error, from which a floor can be set before the experiment begins; the observations are not part of the experiment’s own mean, so its coverage has no complicated dependence. The size of pilot at which further precision in the floor stops being worth what it costs, on the two-sided mistake costs above, is a one-dimensional calculation and it has not been made.

What this rules out, and what it does not #

This essay takes the three-stage fixed-width rule for a normal mean, first stage of five, with the second stage reading its target with Student’s $t$. Its claims are that this reading raises coverage at a half-width of 0.4 from 88.4% to 92.1%, and leaves 6.6% of experiments stuck instead of 20.3%, at 1.17 times the oracle’s spend; that those still stuck cover 62.9%, less than the 70.4% before; that a fixed floor at the same spend covers 96.4%, and more than the read rule at all seven requirements measured; that the fraction reaching 95% runs from 0.43 to 0.79 across them; and that a floor of 26 observations met by the wrong spread holds coverage at or above 94.1% across a nearly fivefold range of requirements, at a bounded cost in observations.

What stays out, as a decision: a first stage larger than five, where everything here is milder because the first reading is better; non-normal observations; and any statement that fixed-width intervals are the right object, which the width a trial stops for takes up on its own terms. The boundary against the fifth a third stage never reaches is that it found the failure and the floor, and this one tests the proposal to replace the floor, and ends up defending it.