The guesser who learns

Guessing one arm in three gave a bound for how guessable a three-arm minimisation is: an investigator who knows the rule, every factor and every assignment so far names the next arm 86.2% of the time at a deterministic rule, and 71.8% when the rule takes its preferred arm with probability 0.8. A guesser who sees the totals asked how much of that survives when the investigator sees less. It found that a drug-supply screen’s arm totals alone name 48.9% at the same probability, where chance is 33.3%, and manufacture a little under half of the full guesser’s bias.

Those guessers each had one rule chosen in advance. A real investigator is not so stubborn. An open trial shows every assignment the moment it is made, and even a blinded one reveals some of them, when an adverse event forces an unblinding. Someone paying attention can see which of the partial signals has been working. So that essay ended by asking whether a guesser who keeps a running tally of how often each reading has been right at their own centre, and follows whichever is ahead, approaches the full bound over a trial. The answer would say whether those partial numbers are a floor for what an attentive investigator gets, or a ceiling.

Neither, as it turns out, and the reasons are not the ones either word suggests.

Following the leader #

The setup is the field’s standard trial: three arms, three factors with centre among them, a hundred and fifty patients. At each arrival an investigator at a centre has three readings available.

• The centre’s scored factors: apply the rule’s own imbalance score to the patients enrolled at this centre.
• The centre’s totals: name the arm this centre has given the fewest patients.
• The trial’s totals: name the arm with the fewest patients overall, as a drug-supply screen would show.

The learning guesser keeps a tally per centre of how often each reading would have been right, updates it whenever an assignment is revealed, and follows whichever is ahead. All three fixed readings and the learner are scored on the same patients, alongside the full guesser and the bound.

The figure at the head of this essay runs through a trial. Following whichever has been right names 48.9%, level from about the second tenth of the trial to the end. The three readings it chooses among name 48.5%, 49.9% and 48.9%. So over a trial, the guesser who pays attention does about as well as the average of what they are choosing from, and it is a full point below the best of them, the centre’s totals, simply followed from the start.

There is nothing subtle about why. A tally can only help if the readings differ reliably, in a way that a run of successes reveals. These three differ by about one point in fifty over a trial. To tell a 49.9% reading from a 48.9% one takes thousands of revealed assignments at a single centre, and the largest centre sees about forty-five. The leader changes more or less at random, and what is being measured is noise in an almost level race.

Nor does revealing fewer assignments hurt. At a tenth of assignments revealed, as in a blinded trial with occasional unblinding, the same guesser names 48.8%. When a signal is not informative, starving it costs nothing.

A centre that knows too much #

Tallying does surface one real thing. It is in the difference between the readings as a trial grows.

The centre’s scored reading is what a thorough investigator would prefer. It has more information than either of the totals: every factor of every patient seen at the centre, applied through the rule’s own score. And it is the only reading that gets worse as a trial grows. It names 49.1% at ninety patients, 48.5% at 150, 47.4% at 300, 46.2% at 600 and 45.3% at 1,200, and the fall is visible within a single trial: on six hundred patients it goes from 49.4% in the first tenth to 44.6% in the last.

The centre’s plain totals, using strictly less information, stay flat at 49.5%, 49.9%, 50.1%, 50.2% and 50.3%.

The reason is what the rule actually balances. Minimisation balances the trial’s margins. A factor’s level at this centre is a small part of that level’s total, and as the trial grows, the other centres’ contributions take over. So a score built on the local counts is reading something that decreasingly matters to the rule’s decision. It keeps pointing confidently at the arm this centre’s own factor mix would suggest, and that is a worse guide than a mere count of arms, because the count of arms at a centre is roughly self-correcting whatever the rest of the trial does: the rule evens its own centre margin along with the others, and the centre is one of the factors.

So here is a small thing a tally does learn. On trials of six hundred, the share of patients on which the factor reading leads falls from 58.5% in the first tenth to 13.7% by the last, while the centre’s totals rise from 13.4% to 55.2%. The learner has correctly noticed a slowly decaying signal.

It does not make the learner any better than the fixed reading it is converging on, which was right from the first patient. That is the lesson of the essay in small: an investigator who knows the rule does not need to learn which reading to use. They need to think about which reading the rule is actually responding to. The second is arithmetic, and it can be done before the trial opens.

What the readings say together #

There is a stronger idea of learning than following a leader. The three readings can disagree, and the pattern of disagreement carries more than any of them does alone. When the centre’s totals and the trial’s totals both name the same arm, that is stronger evidence than when they split. A guesser can keep, for each of the 27 joint answers the three readings can give, a count of which arm has actually been taken, and name the most frequent.

This has two versions, and the gap between them is the finding.

Learned inside this trial. The table starts empty and is filled from revealed assignments. It falls back on the centre’s totals where it has nothing yet.

Learned beforehand. The table has been filled over five hundred earlier trials of the same rule. This is the upper limit on what these three readings can yield, the most a guesser could ever learn from them, however patient and however experienced.

Learned beforehand, the table names 54.5% at 150 patients, against 49.9% for the best single reading. So the three readings together are worth nearly five points more than any one of them, and that fact is an answer of sorts to the original question: the partial numbers are not a ceiling. There is more in the signals than any fixed reading extracts.

Learned inside the trial, the same table names 45.8%, nearly four points below the single reading it was trying to improve on. It starts at 36.4% in the first tenth of the trial, close to chance, because a table with 27 cells has very few entries in it. By the last tenth it has climbed to 48.9%. On larger trials it does get there, passing the centre’s totals at somewhere between 600 and 1,200 patients: 44.2%, 45.8%, 47.5%, 49.1% and 50.7% across the sizes. Yet at 1,200 patients it is still under the 54.2% a table from earlier trials gets from the first patient.

So learning costs far more than it earns within any trial of an ordinary size. A guesser with a clean slate gives up four points to find out what a thoughtful one would have written down beforehand. And a thoughtful one, it should be said, is not hypothetical: the rule is public, and anybody can simulate it. The table “learned over earlier trials” could as easily be computed in an afternoon before the first patient. That is a sharper statement of the risk than anything to do with attentiveness. The dangerous guesser is someone who has run the rule on a laptop, not someone who has been watching the trial.

The gap to the bound stays wide. At 0.8 the table names 54.5% against 71.8%; at a deterministic rule, 63.7% against 86.3%. The two sides of that gap are not the same kind of thing. A table that learns forever hits 54.5% because it only has those three numbers: the missing seventeen points are what the other centres’ factor mix would have said, and no amount of practice at one centre can supply them. Attentive or not, that investigator does not have what the rule used. So what the partial readings bound is not attention but access, and a trial protecting against guessing should be thinking about access.

Worth, in bias #

As the earlier essay insisted, a guess rate is not the quantity that matters: the bias an investigator can manufacture by enrolling better patients when their guess names a preferred arm is. On the same construction, with a patient half a standard deviation better when the guess names the first arm.

The full guesser manufactures 0.468, on a standard error of about 0.005. The table from earlier trials manufactures 0.234, half as much. The centre’s totals manufacture 0.214. The trial’s totals, 0.195. Following the leader, 0.193. The scored centre, 0.181. The table learned inside the trial, 0.159.

Read as a share of the full bound: an investigator with a laptop gets half, a plain centre count gets 46%, and an attentive one following their record gets 41%. Bias tracks the guess rate closely here, and more closely than it did for some of the earlier essay’s readings. The ordering is the same in both columns, and the difference between “paying attention” and “just counting one’s own centre” is two hundredths of a standard deviation, in the direction of the plain count.

A trial’s position has not moved much from what the last essay reported: the numbers are close to its partial-reading numbers, and they do not approach the bound. Measured or not, the reader of a protocol can treat these partial guessers as a level of risk that exists at about half the bound as soon as anybody at a centre can count, and which attention does not raise.

What a protocol can do with this #

Report which readings exist. The rule that can be guessed and its successors have established that lowering the rule’s probability lowers everything in proportion. What this essay adds is who to worry about. A trial that hides arm totals from the supply screen but leaves centre counts accessible has hidden the wrong one: the centre’s totals are the best single reading on every row.

Remember that the analysis has already been told. The analysis after three arms showed what a re-run reference does for the level of a test under this rule, and none of that protects against a biased enrolment: the reference re-runs allocations and takes the patients as given, so manufactured bias passes through it untouched.

Do not count on opacity of the rule. A learned table does its work without any trial data at all, if the rule’s published. A central randomisation system that keeps its probability and its score private removes that table, and it is a step a trial can take. Whether it would be regarded as acceptable, in a field that also wants the rule reported, is a question that belongs to the people who’d have to sign it.

Do not overweight the attentive investigator. The fear is of someone who has been watching and has got good at it. The measurement says that person is not better than someone who counted from the first patient, and that learning at one centre is too slow to matter at trial sizes that most protocols set. This is not a comfort about guessing. It is a redirection about whom the protection should be aimed at.

How the numbers were made #

Fifteen hundred trials at each size and each setting of the rule, with the same patients and the same rule run for all seven guessers. Every guess is made before the assignment and every reading is computed from what its guesser is allowed to see. A coin’s tie-break is not revealed to any of them, as in the earlier essays. A guesser’s tally updates with probability equal to the reveal rate, which is one unless stated. The table learned beforehand is built on 500 earlier trials of the same size and rule, with separate seeds, and it is frozen before the scored trials begin. Guess rates over a whole trial have a standard error of about a third of a point at this many trials, and it is about 1.1 points for a single tenth of a trial of 150.

Four things are required of this. Following the leader must not beat the best fixed reading by more than half a point. The table learned inside a trial of 150 must lose to it. The table learned beforehand must beat it by at least two points. And the full bound must stay at least ten points above that. The reading the essay set out to test, that a patient guesser closes on the bound by the end of a trial of six hundred, is refused: the table has reached 51.4% by then, against a bound of 72.1%.

Still open: a richer reading, and a guesser at more than one centre #

What the missing points are made of. The table is limited to three derived readings, each of them a single arm named. A guesser could keep the three readings’ margins instead: how far ahead each named arm is, rather than which one. That can only help, and how much it adds to 54.5% is a small calculation. The other direction is more interesting: how much of the seventeen-point gap would one additional piece of information close, such as the trial’s factor totals without the patient-level data? A research coordinator with access to a monthly report has exactly that.

Collusion between centres. Everything here is one investigator at one centre, but two centres sharing their counts would see more of what the rule sees, and the other arms make the score harder to read from one vantage point. How the guess rate climbs with the number of centres pooled, towards the full bound at all of them, is a curve that would price what a coordinating site can do, and it would say how much of the protection comes merely from distance between sites.

Which partial numbers are bounds, and of what #

This essay takes a guesser under three-arm minimisation who learns from revealed assignments. Its claims are that at a rule probability of 0.8 and 150 patients, following the reading with the best record names 48.9% against the best fixed reading’s 49.9%; that the scored centre’s reading alone gets worse as the trial grows, from 49.1% at ninety patients to 45.3% at 1,200, while the centre’s plain totals do not; that a joint table of all three readings names 54.5% if learned beforehand and 45.8% if learned inside the trial; that none approaches the full bound of 71.8%; and that in bias the same order holds, with the full guesser at 0.468 and the table from earlier trials at 0.234.

What stays out, as a decision: unequal targets, the case the unequal-targets essay raises; a guesser reading partial factor summaries, which is the first deferral above; and any claim about actual investigator behaviour, since what is measured is what the information permits, not what anyone does with it. The boundary against a guesser who sees the totals is that its readings are fixed, and this one asks whether watching them work improves on any of them, and finds that it does not.