The estimate after the choice

A two-stage trial that drops arms can be given a boundary that holds its error rate exactly. The previous essay solves for it: 2.313 rather than 1.96, at eight arms.

That fixes the test. It does not fix the estimate, and the estimate is what gets published, quoted in a guideline and used to decide whether the effect is worth having.

The bias, and where it comes from #

Eight arms, sixty observations each, the best carried forward, thirty more. Every arm’s true effect is zero.

The first stage’s estimate for the arm that was carried forward averages +0.184. It is the maximum of eight noisy estimates of zero, so of course it is positive: the selection did not find an effect, it found a large draw, and the number it selected on is the number being reported.

This is the winner’s curse — the same argument as an effect inflated by having cleared a significance threshold — moved inside a single trial. The threshold there was statistical significance; here it is being the largest of eight.

The second stage’s estimate for the same arm averages −0.0011, and that is not luck. The arm was chosen using stage one, and stage two was collected afterwards; nothing about stage two entered the choice. So it is unbiased by construction, which makes it one of the few claims on this site that can be asserted at the simulation’s own error rather than at a tolerance.

And the published estimate — all ninety observations on the surviving arm against all ninety on the control — averages +0.123. That is exactly two thirds of the first stage’s bias, because the first stage is two thirds of the data:

bias(combined) = (n₁ / (n₁ + n₂)) × bias(stage one)

Measured 0.1226 against a predicted 0.1230. The published estimate does not dilute the selection away, it carries a fixed and computable share of it.

How the bias grows with the number of arms #

At one arm there is nothing to select and every estimate is unbiased — the refusal that makes the rest of the column mean something. From two arms on the bias is real, and it grows like the expected maximum of k draws, so it rises quickly and then slowly: most of the damage is done by the fourth arm.

The bias in closed form #

“It grows like the expected maximum of k” is exact rather than approximate here, and checking it turns five measured numbers into one formula.

The stage-one estimate of an arm’s effect is a mean of sixty observations, so with unit noise its standard error is $1/\sqrt{60} = 0.1291$. The arm carried forward is the largest of k such estimates of zero, so its expectation is that standard error times the expected maximum of k standard normals — 0.5642, 1.0294, 1.4236 and 1.6293 at two, four, eight and twelve.

Multiplying gives 0.0728, 0.1329, 0.1838 and 0.2103, against 0.074, 0.134, 0.184 and 0.211 measured. Four predictions from no fitted constants, each within two thousandths.

And the published column follows from it by the two-thirds rule, which holds just as tightly: 0.0493, 0.0893, 0.1227 and 0.1407 predicted against 0.049, 0.088, 0.123 and 0.141.

So the whole table is one closed form, and the closed form says the bias is a property of three things only — the noise, the size of the first stage, and how many arms were on offer. Nothing about the effects being estimated enters it, which is why the row of zeros in the second column is not a coincidence of this simulation.

What the formula says about more arms and more data #

Having it as a formula makes the two obvious remedies checkable without running anything.

More arms is worse, slowly. The expected maximum grows like $\sqrt{2\log k}$, so going from eight arms to sixteen takes the stage-one bias from 0.184 to about 0.228 — an increase of 0.044 for a doubling — and going all the way to a hundred arms only reaches 0.324. A screening design with a great many arms is not catastrophically worse than one with eight; it is worse by a logarithm.

More second-stage data is worse than useless as a fix. The published bias is $n_1/(n_1+n_2)$ times the stage-one bias, so halving it takes $n_2 = 2n_1 = 120$ — four times the thirty currently collected. Cutting it to a tenth takes $n_2 = 540$, eighteen times the second stage, at which point the trial has spent nine tenths of its budget diluting a bias rather than estimating anything.

That is the argument for the second column stated in its strongest form. The unbiased estimate is available at a cost of thirty observations; buying the same reduction by dilution costs five hundred and forty and never quite arrives.

The unbiased estimate is the worse estimate #

The obvious repair is to report the second stage alone. It is unbiased, it needs no correction, and it is sitting in the data already.

It is also the least accurate of the three:

The published estimate is biased by 0.123 and is closer to the truth on average than the unbiased one, because it is built from three times as many observations. Removing the bias costs more accuracy than the bias cost.

That is the trade in its cleanest available form, and it is not a paradox. Root mean squared error is bias squared plus variance, and the variance of an estimate from thirty observations is large enough to swamp a bias of a tenth of a standard deviation. A reader who wants one number should be given the combined one.

A reader who wants to know whether the effect is worth having should not, and that is the distinction this essay exists for.

Why the second stage is exactly unbiased #

The claim that stage two is unbiased is stronger than it sounds and is worth spelling out, because it is the one piece of machinery in this field that needs no correction and it is easy to state too loosely.

The selection is a function of the first stage’s data alone. Conditional on which arm was chosen, the second stage’s observations are draws from that arm’s distribution and from nowhere else — the choice cannot have changed them, because it happened first and they had not been collected.

So E[stage two | arm j selected] is arm j’s true effect, for every j, and the average over the trials that selected j is unbiased for whatever j’s effect is. That is a conditional statement and it survives averaging: the marginal expectation is a mixture of unbiased pieces.

Two things follow that are worth having.

It does not depend on the selection rule. Pick the largest, pick at random, pick by an oracle that knows the truth: the second stage is unbiased for the effect of whatever was picked. Nothing about how badly the choice was made can bias data collected afterwards.

And it does not survive re-using stage one. Any estimate that touches the first stage inherits a share of its bias, and the share is exactly the weight it gives it. That is the identity measured above, and it is why there is no clever weighting that gets both precision and unbiasedness: the two are traded against each other by a single number.

Which question is being asked #

The two uses of a trial’s estimate pull in opposite directions and are almost never separated.

Deciding whether the effect is real is what the boundary is for, and the boundary handles the selection completely. Nothing in this essay changes that: a design tested at 2.313 rejects a true null 5% of the time whatever the estimate is doing.

Deciding whether the effect is large enough to act on is an estimation question, and here the bias is the whole difficulty. A treatment whose true benefit is 0.1 and whose published estimate is 0.22 will look worth its cost when it is not, and no amount of statistical significance corrects that — the trial was right that something is there and wrong about how much.

And predicting what the next trial will find is where the bias is most expensive, because the next trial is usually powered on this one’s estimate. Powering a confirmatory trial for 0.22 when the truth is 0.1 gives it a fraction of the power it was designed for, which is the mechanism by which one optimistic estimate propagates into a literature of underpowered replications.

For the second and third of those, the second-stage estimate is the right one to quote in addition — not because it is more accurate, but because it is the only one that does not require the reader to trust a correction.

When one arm really is better #

Everything above is measured under a global null, which is the right setting for a bias and the wrong setting for a design decision. With one arm carrying a real effect of 0.3 among eight, the same design gives:

stage one +0.080, stage two −0.001, published +0.053.

The bias is less than half what it was under the null. The reason is that the selection is now usually picking the right arm — it is not choosing the largest of eight equivalent draws, it is confirming an arm that is genuinely ahead — and a selection that was going to be made anyway costs nothing.

That is the general shape and it is worth stating as a rule of thumb: the bias is largest exactly when the trial found nothing. A design that selects among arms that really do differ is close to unbiased; a design that selects among arms that are all the same reports the largest fluctuation it saw. Which case a particular trial is in is what the trial was run to find out, so the bias cannot be corrected by knowing which case it is.

The same shape, three times on this site #

This is the third time this site has measured a quantity that is inflated by having been selected, and the three differ only in what did the selecting.

The winner’s curse: an effect reported because it cleared significance is inflated by 2.07× at n = 16, and the literature is the thing doing the selecting.

The forking paths: an analysis chosen from twenty available ones rejects a true null 57% of the time, and the analyst is doing the selecting.

And this essay: an arm chosen from eight is inflated by 0.184, and the design is doing the selecting — deliberately, in advance, with the selection written into the protocol.

The last of those is the one with a defence, and the defence is exactly that it was written down. Because the rule is known before the data, its consequences can be computed: the boundary can be solved for, the bias can be predicted from k and the stage sizes, and the second stage can be reported separately. None of that is available to a literature that selects on significance or to an analyst who selects on a result, and it is the whole difference between an adaptation and an accident.

What to report #

Four things, and the first two cost nothing.

Report which arm was selected, and out of how many. A reader cannot apply any correction without k and cannot tell that one is needed without knowing there was a selection at all.

Report the stage-two estimate beside the published one. It is unbiased by construction, it is already computed, and the gap between the two is a direct measurement of what the selection did in this trial rather than on average.

Quote a bias-adjusted estimate if one is being quoted at all, and note that the adjustment is a function of k, of n₁/n₂ and of the true effect — which is unknown, which is why the adjustment is itself a design decision rather than an arithmetic one.

Say which stage the estimate came from. A published effect of +0.12 from this design and a published effect of +0.12 from a fixed two-arm trial are different numbers, and nothing on the face of either says so. The provenance is one clause and it is the only thing that lets a reader apply anything at all.

And do not power the next trial on this one’s estimate. The bias is largest exactly where the next trial is most at risk, and the second-stage estimate — or an explicit discount — is what belongs in that calculation.

What a conditional interval would need #

An estimate with a known bias is halfway to an interval that covers, and it is worth saying why this essay does not finish the journey.

The bias measured above is a marginal quantity: the average over all trials of this design, under a stated truth. What a reader of one trial wants is conditional — given that this arm was selected out of eight, and given the value it reported, where is the effect? That is a different object, its bias depends on the true effects of all eight arms, and those are exactly what is unknown.

Three routes exist and each has a cost this site would want measured before recommending it.

Condition on the selection and invert. The distribution of the reported estimate given that its arm was selected is computable for a stated set of true effects, so an interval can be built by inverting a test over them — the same construction Fieller’s set uses for a ratio, and with the same tendency to be wide or unbounded when the data has not settled much.

Use the second stage alone. Exact, needs nothing, and gives an interval a third the information wide.

Or shrink towards zero, which is partial pooling applied to the eight arms: the selected arm’s estimate is pulled back towards the others by a weight that is estimated from their spread, and the pull is in exactly the direction the selection pushed. That is the most promising of the three and the one that needs the most measurement, since the weight is itself estimated from eight numbers — which is where that field’s own trouble lives.

None of the three is attempted here. What is established is the quantity they would all be correcting, and the fact that it can be predicted from the design before the trial is run.

The refusals #

Three, and they are the reason the numbers above are worth anything.

One arm must be unbiased. With k = 1 there is no selection, and the same machinery is required to report a bias of zero to within the simulation’s own error. It gives −0.0003.

Stage two must be unbiased at every k. This is the strongest claim in the essay and it is asserted at four standard errors rather than at a tolerance, because it follows from the construction rather than from an approximation. If it ever failed, the selection would be leaking into data it could not see, which would mean the simulation was wrong rather than the design.

And the published bias must equal the share the first stage contributes. Measured 0.1226 against a predicted 0.1230, which is a two-routes check on a quantity that could otherwise be measured and believed without ever being explained.

A fourth was added while writing this, and it is the site’s standing gotcha caught in advance rather than after the fact. The identity above was measured at one pair of stage sizes — sixty then thirty — and a relationship asserted as a formula from one point is exactly the shape that turns out to be a fact about a default. So it is checked at a second pair with the weight flipped: selecting on a third of the trial rather than two thirds gives a published bias of 0.086 against a predicted 0.087, and the same design’s bias falls from 0.123 to 0.086 for the reason the formula says it should.

Which is also a design finding rather than only a check. An earlier interim publishes less of the selection’s bias, exactly as it needs less correction at the boundary — the two costs of selecting move together, and both are bought down by choosing sooner.

The third of those is what turns the essay from a warning into an account. A bias that can be predicted from the design — before the trial runs, from k and the two stage sizes — is a quantity a designer can trade against the units the selection saves, and that trade is the only honest way to decide how many arms to screen and when to drop them.