A guesser who sees the totals

The essay before this one measured how often the next assignment of a three-arm minimisation can be worked out, and it measured it against a guesser who has everything: the rule, the score, the factors of every patient enrolled so far, and every assignment. That guesser names the next arm 86.3% of the time under a deterministic rule, against the one in three that naming an arm at random would get, and that is the right number for a bound. It is what an unblinded investigator in an open, single-centre trial could in principle compute.

It is also the number a trial’s designers find easiest to dismiss. Nobody enrolling patients has the factor levels of every patient at every other centre, and a bound that rests on knowing them reads as a worst case that does not happen. The essay named this as what it left out — guessers with partial information, which is the realistic case and needs a model of what an investigator sees — and this one supplies three such models and measures each. The answer is that the bound is pessimistic by about a factor of two, and that what is left after the factor of two is large enough to matter, and is available to people who were never meant to see anything at all.

Four investigators, one trial #

Every measurement here is on the same trials: a hundred and fifty patients, three arms, minimisation on the range score over three prognostic factors — sex, an age band and the centre — with the preferred arm taken with probability one unless a slider says otherwise. The patients and the rule are identical across rows. Only what the person guessing is assumed to see changes.

Knowing everything is the earlier bound: compute the rule’s own score for each arm and name the one it prefers.

Knowing their own centre, scored is an investigator at one centre who sees every factor of every patient enrolled there, and nothing from the other centres, and applies the rule’s score to that partial record.

Knowing their own centre’s totals is the same investigator, who has stopped trying to reproduce the rule and simply names the arm their own centre has sent fewest patients to.

Knowing the trial’s arm totals is somebody who sees only how many patients have gone to each arm across the whole trial. That is not a thought experiment. A trial supplying blinded drug to its centres runs a supply system that tracks how much of each arm’s stock has been used, and anybody who reorders, reconciles or audits it sees the totals.

The full guesser names 86.3%. The guesser with the trial’s arm totals and nothing else names 54.5%. A random guess names 33.3%, measured on complete randomisation over the same patients. So the totals keep about two fifths of the full guesser’s excess over chance: 21.1 points of 53.0. In the multiple a guesser can trade on, the totals give 1.63 times chance where everything gives 2.59 times.

The own-centre investigator who applies the score gets 52.8%; the same investigator counting only their own centre’s arm totals gets 55.5%. The cruder rule wins, and the reason is worth a section.

Why counting beats scoring on a partial record #

Minimisation’s score for a patient is a sum over that patient’s factor margins: how uneven the arms are among patients of the same sex, among patients of the same age band, and among patients at the same centre. An investigator at one centre knows exactly one of those three margins completely — the centre margin, which is nothing but their own centre’s arm totals. The other two they know only in the part contributed by their own centre, which is a fraction of each.

Scoring the partial record treats the partial sex and age margins as if they were the real ones, and they are not; they are the real ones minus everything the other centres contributed, which the rule is busy balancing in directions the investigator cannot see. Counting the centre’s totals uses only the margin that is known exactly and ignores the two that would mislead. The general point is one the guessing literature tends to skip: a partially informed guesser should not imitate the rule. Imitating it is optimal only with the rule’s information. With less, the best guess is built from whichever pieces of the rule’s input are known exactly.

The trial-wide totals are useful for the same reason from the other side. Minimisation balances every margin of every factor, and summing any one factor’s margins over its levels gives the arm totals, so a rule that keeps every margin even keeps the totals even as a by-product. The arm with the fewest patients overall is, more often than not, the arm with the fewest patients in the arriving patient’s own margins. That is a weak inference from coarse information, and it is right 54.5% of the time.

What a weak inference manufactures #

A guess rate is not yet a harm. The harm is what a guessing investigator does with the guess: enrol a patient with a better prognosis when the next arm is expected to be the one they favour, and hold that patient back otherwise. Under a true null this manufactures a difference between arms out of nothing, and that is the measurement that matters.

The full guesser manufactures an apparent advantage for the first arm of 0.648 standard deviations under a true null, with a standard error of about 0.006. The arm-totals guesser manufactures 0.289, 45% of it. The own-centre totals guesser manufactures 0.301; the own-centre scorer 0.247. The random guesser lands at −0.007, inside its noise.

For scale, the permuted block — the design every account of selection bias starts with, because an investigator who knows the block size can count the remaining assignments in it — gives its own guesser 61.1% and a bias of 0.373. Minimisation guessed from a drug-supply screen sits between that and nothing: 0.289 against 0.373, from someone with no access to the randomisation system at all.

The bias is not the guess rate’s excess rescaled. The totals keep about two fifths of the excess guess rate and a bit under half of the bias, so the totals guesser’s correct guesses are worth slightly more each than the full guesser’s — which arrivals carry them is not measured here, and is the obvious place to look next. What the gap does say is that a guess rate understates a partial guesser: the quantity to report is the bias, not the rate.

More arms, smaller share, never small #

The earlier essay found that adding arms lowers the raw guess rate and raises its multiple of chance. The arm-totals guesser follows the same pattern at a lower level.

At two arms the full guesser names 87.9% against a chance of a half, and the arm totals give 68.1%: the totals keep 48% of the excess. At three arms they keep 40%, and at four arms, where the full guesser names 80.9% against a chance of a quarter and the totals give 45.8%, they keep 37%. As a multiple of chance the totals give 1.36, 1.63 and 1.83 times. The share falls with the number of arms because the factors carry more of the rule’s decision when there are more arms to choose among — the rule’s choice is less often the arm that is simply behind — but it does not fall below a third at any number of arms a trial runs.

The trial size makes almost no difference. At ninety patients the full guesser names 85.8% and the totals 54.7%; at three hundred, 86.7% and 54.4%. The totals’ advantage is a property of the rule, not of a phase of the trial in which the counts are small.

The one dial turns both down together #

Every minimisation trial has one dial against predictability: the probability with which it takes its preferred arm. The earlier essay measured what turning it down does to the full guesser; the question here is whether it also protects against the partial ones, or whether some of them are left untouched.

It protects against both, and in proportion. Over eight hundred trials at each setting, the full guesser names 47.2%, 55.5%, 63.6%, 71.6%, 79.2% and 86.2% as the probability runs from 0.5 to 1; the totals guesser names 39.6%, 42.8%, 45.9%, 48.7%, 51.7% and 54.5%. The share of the excess the totals keep runs from 45% at 0.5 to 40% at 1. The dial does not change who can exploit the rule, only how much.

That has a direct reading for a protocol. The usual defence of a deterministic or nearly deterministic minimisation is that the people who could work out the next assignment do not have the information to do it. The measurements here say that the defence is half right: the information needed for the full bound is rarely available, and the information needed for half of it nearly always is. A trial that sets its probability at 0.8 because the bound at 1 looks alarming has bought the same proportional reduction for the drug-supply clerk as for the investigator.

At the probability trials actually run #

A deterministic rule is the bound, and few trials run one. Most minimisation protocols take the preferred arm with a probability around 0.8, on the advice that a little randomness costs little balance and buys a great deal of unpredictability. The four investigators were measured there too, on the same thousand trials.

At a probability of 0.8 the full guesser names 71.6% of next assignments and manufactures a bias of 0.460. The own-centre scorer names 48.7%, the own-centre counter 49.9%, and the arm-totals guesser 48.7%, with biases of 0.189, 0.215 and 0.200. The random guesser stays at 33.3% and −0.007. So at the setting a protocol would actually choose, someone with nothing but the drug-supply totals gets half again what chance gives and manufactures an apparent effect of a fifth of a standard deviation in a trial where there is none.

A fifth of a standard deviation is not a small number in this setting. It is the size of effect many trials are powered to detect, and it comes out of a randomised trial with a valid randomisation test, because the rerandomisation reference set is built from the patients who were enrolled, and the patients who were enrolled are the ones the guesser chose. The bias is in the sample, not the analysis, and randomisation, which is not balance, cannot see it either.

What a trial can do about it #

Three repairs follow from where the information leaks, and they are ordered by cost.

Do not show anybody the arm totals. A blinded drug-supply system can track stock by kit number without exposing which arm a kit belongs to to the people who reorder; many do, and some report arm-level usage for convenience. This removes the arm-totals guesser entirely and costs nothing.

Do not stratify by centre if centres can see their own totals. The own-centre totals guesser works because the centre margin is one of the margins the rule balances and the one an investigator knows completely. Dropping centre from the minimisation factors and handling centre in the analysis removes that leak, at the cost of a centre imbalance the analysis then has to absorb.

Lower the probability. It protects against every guesser in proportion, and it costs balance, as the two-arm field’s version of this priced in detail. It is the only repair that does not depend on knowing which information leaks — which is also why it is the one to use when nobody is sure.

And a trial can measure its own exposure rather than argue about it. The allocation log exists, and running each of the three partial guessers over it at an interim look — counting how often each would have named the next arm — takes seconds and needs no outcome data, so it unblinds nothing. A guess rate near the numbers here says the leak is real for that trial’s factors and centres; one near chance says the trial’s own structure has closed it.

What does not help is the argument that a guesser who is right only half the time cannot do much damage. At three arms half the time is one and a half times chance, and the bias it manufactures is nearly half of what a perfectly informed guesser can manufacture.

Still open: the guesser who learns #

Every guesser here uses a fixed rule — imitate the score, count the centre, count the trial — chosen in advance. A guesser in a real trial sees whether each guess came true when the next patient’s assignment is revealed, in an open trial at once and in a blinded one at unblinding of an adverse event, and can use that to learn which of the partial signals is informative. The natural model is a guesser who keeps a running tally of how often each of the three partial rules has been right at their own centre and switches to whichever is ahead. Whether such a guesser approaches the full bound from below over the course of a trial, and how quickly, is the measurement that would say whether these partial numbers are a floor or a ceiling for what an attentive investigator actually achieves.

What is claimed here, and what is not #

This essay takes the predictability of a three-arm minimisation to guessers with partial information. The claims are that over a thousand trials of a hundred and fifty patients under deterministic minimisation on three factors, a guesser who sees only the trial’s arm totals names the next arm 54.5% of the time against a measured chance of 33.3% and a full guesser’s 86.3%; that one who sees their own centre’s totals does slightly better and one who applies the rule’s score to their own centre’s records slightly worse; that the arm-totals guesser manufactures 0.289 standard deviations of bias against the full guesser’s 0.648 and a permuted block’s 0.373; that the share of the full guesser’s excess the totals keep falls from 48% at two arms to 37% at four; and that lowering the rule’s probability lowers every guesser in proportion.

What stays out, named as a decision: the learning guesser deferred above; guessers who combine centre records with trial totals, which would sit between the rows here and is a refinement rather than a new finding; trials with unequal allocation, where the targets the rule balances towards change what the totals reveal; and the analysis, since the rerandomisation test keeps its level whatever a guesser does and the bias here is in what is enrolled rather than in how it is tested.

The boundary against the earlier essay in this field is the information set. That essay owns the bound, the closed form for two arms and the reason the multiple of chance rises with the number of arms; this one owns how much of the bound survives when the guesser has less. The boundary against the two-arm field is the number of arms, and against the field’s analysis essay it is the difference between a bias in the patients and a bias in the test.

The checks, and the refusal that makes them mean something #

Each figure checks its own claim at every slider value: the arm-totals guesser is more than eight points above chance and below the full guesser; its bias is more than five standard errors from zero and below the full guesser’s; at two, three and four arms it keeps between a quarter and three fifths of the full guesser’s excess; and the full guesser is never beaten by it at any probability. The random guesser is measured rather than assumed, which is the check that the apparatus can produce a zero. The refusal is a guesser who sees the arm totals of a trial randomised by a coin — which must be no better than chance, and is, at 33.31% over the same thousand trials. A check that reported it as predictable would be crediting the guesser with what belongs to the rule.