A number read with error

The rule that reads the number earned its place in this field with an exponent. Every rule that balances categories of a continuous covariate leaves an imbalance that falls as $1/\sqrt{n}$, because inside a category the assignment is still a coin; a rule that reads the covariate itself and assigns each arrival to maximise the information about the treatment effect leaves one that falls nearly as 1/n. The gap between them is not a factor that can be quoted but something that widens with every unit enrolled.

That result has an assumption in it that nobody states, because in the simulation it is automatically true: the number the rule reads is the number that drives the outcome. In a trial it is not. A blood pressure taken once at enrolment, a baseline score on a questionnaire, a weight read off a clinic scale — each is the quantity of interest plus an error, and the rule balances whatever the clinic wrote down. This essay asks what that does to the rule’s advantage, and the answer has a closed form short enough to fit on one line.

The floor a recording sets #

Write X for the covariate that drives the outcome, standard normal, and W = X + U for the number recorded, with U normal and independent of X. The reliability λ is var X / var W: the share of what was recorded that is the real thing. A reliability of 0.8 is a respectable instrument; 0.5 is a recording whose error is as large as what it records.

The rule reads W and nothing else. Regress X on W and the conditional mean is λW, with a residual X − λW of variance 1 − λ that is independent of W. Everything the rule can see about X is in the first part, and it can balance that as well as it ever balanced anything. The second part is invisible to it, and on an invisible quantity every rule is a coin. So the imbalance the rule leaves in X can never be smaller than a coin’s imbalance in the residual, and as a share of a coin’s imbalance in X that is $\sqrt{1 - \lambda}$, whatever the rule does with W.

The simulation lands on it. At two hundred units and six hundred trials at each reliability, the rule that reads W leaves 0.452 of a coin’s imbalance in X at a reliability of 0.8, against a floor of 0.447; 0.718 at 0.5 against 0.707; and 0.345 at 0.9 against 0.316. All the while it balances W itself as well as it ever did — between 0.125 and 0.133 of a coin’s at every reliability, which is the number the field’s earlier essay reported for an exact covariate at this trial size. The rule is doing its job perfectly on the wrong variable.

The point at λ = 1 is the original result: 0.125 of a coin’s. At 0.9 the same rule leaves nearly three times as much. A recording that is nine tenths signal has cost the rule most of what made it worth running, and the reason is the shape of $\sqrt{1 - \lambda}$: it is steepest near one. The first tenth of error raises the floor from 0 to 0.316; the next four tenths together raise it only to 0.707.

A cut is a recording #

The same algebra applies to a rule that does not read W at all but minimises on a median split of it, the way most trials handle a continuous covariate in practice. A median split of a normal variable sees 2/π of its variance — the between-category share the first essay of this field computed — so a split of W sees 2λ/π of X, and minimisation on it is held above $\sqrt{1 - 2\lambda/\pi}$. At a reliability of 0.8 that floor is 0.701 and the simulation gives 0.745; at 0.5 it is 0.826 against 0.833.

Put the two formulas side by side and something falls out. A rule that minimises on a median split of an exactly recorded covariate is held above $\sqrt{1 - 2/\pi}$ = 0.6028. A rule that reads a recording of reliability λ is held above $\sqrt{1 - \lambda}$. The two are equal at λ = 2/π = 0.637. A median split is, for the purpose of balancing, a measurement with reliability 0.637 — and the rule that reads the number beats it only when the number is recorded better than that.

That turns a design choice into a comparison of instruments. Categorising a covariate the clinic recorded well throws away over a third of what it knew; reading a covariate the clinic recorded badly throws away the same third through a different door. Both are information losses, and the field has been treating one of them as a choice and the other as somebody else’s problem.

The rate goes back to the coin’s #

The original rule’s case rested on a rate rather than a constant, so the decisive question is what the recording does to the rate.

Across trials of 50 to 800 units, four hundred trials at each size, the rule’s imbalance in W falls with a least-squares slope of −1.023 on log axes — the 1/n rate, intact. Its imbalance in X falls with slope −0.526, and a coin’s with −0.517. As a share of a coin’s, the rule’s imbalance in X runs 0.516, 0.476, 0.439, 0.477 and 0.471 across the five sizes: a constant near $\sqrt{1 - \lambda}$ = 0.447, with no trend that more units remove. At a reliability of 0.5 the share runs 0.787, 0.680, 0.754, 0.691 and 0.715, around 0.707.

So the rate a rule achieves is a rate on the variable it reads. On the true covariate a rule that reads a recording has the coin’s rate and a better constant — which is exactly the profile the earlier essay used to dismiss the rules that balance categories. With any error in the recording, the rule that reads the number is a categorising rule in that sense, with $\sqrt{1 - \lambda}$ where the cut had $\sqrt{1 - 2/\pi}$. The exponent that separated them belonged to an exact covariate.

The reason is visible in the decomposition. The rule’s imbalance in X is its imbalance in λW plus its imbalance in the residual. The first part shrinks at the fast rate, because the rule is balancing W at the fast rate and λW is a fixed multiple of it; the second shrinks at the coin’s rate, because nothing balances it. At fifty units the first part is still a visible share of the total. By a few hundred it is negligible, and what remains is a coin on a variable of variance 1 − λ. That is why the slope on X is slightly steeper than the coin’s over the small sizes and indistinguishable from it over the large ones.

That is not a small correction to a large result. The earlier essay’s headline was that the rule’s advantage “keeps going” as the trial grows. On a covariate recorded with reliability 0.8 it stops at once, at a bit under half a coin’s, and stays there at fifty units and at eight hundred.

What the analysis keeps #

Imbalance in a covariate is a means, not an end. What a trial wants is a precise estimate of the treatment effect, and the covariate matters because it is in the outcome. So the last measurement is of the estimate itself: three thousand trials of a hundred units each, with no treatment effect and an outcome y = X + ε, so that the covariate and the noise contribute equally to the outcome’s variance.

A coin with no adjustment spreads 0.1410, against a closed form of $\sqrt{2/100}$ = 0.1414. The rule that balanced W, with no adjustment, spreads 0.1124. That is a real gain over the unadjusted coin. But a coin whose analysis simply adjusts for the recorded W spreads 0.1099 — the same number, to within the simulation’s noise — and both sit on the closed form $\sqrt{(\sigma^2 + \beta^2(1 - \lambda))/n}$ = 0.1095. The part of X the recording missed is in the error of every analysis that has only W, whether the W was used to assign or to adjust, and neither route reaches past it. Adjusting for the true X, which is here only as a ceiling, spreads 0.1018.

At a reliability of 0.5 the pattern is the same and the gap from the ceiling wider: the rule unadjusted spreads 0.1255, the coin adjusted for W 0.1224, the closed form 0.1225. At a reliability of 1 the rule unadjusted spreads 0.1028 and the coin adjusted for X 0.1018: the field’s earlier finding that design and adjustment buy nearly the same thing, in the case where the recording costs nothing.

The error rates tell the same story from the other side. With no effect present, the rule’s unadjusted test rejects 0.70% of the time at a reliability of 1 — badly conservative, because its standard error is computed from a residual that still contains all of X while the estimate no longer does, which is the mechanism the essay on the analysis afterwards measured. At a reliability of 0.8 it rejects 1.30% and at 0.5 2.63%. The recording makes the unadjusted analysis less wrong, for the bad reason that the design has removed less of what the standard error assumes is still there. Adjusting for W after the rule restores the level at every reliability: 5.33%, 5.27% and 5.77% over three thousand trials.

A precise number cut, or a rough number read #

The equality at 2/π answers a question trials actually face, because the two instruments are often both on offer. A covariate can be measured carefully at a screening visit and then entered into the randomisation system as a category — “over 140”, “under 140” — or it can be taken quickly at the enrolment visit and entered as the number. The first is a precise number cut; the second is a rough number read. Which leaves the true covariate better balanced?

The closed forms say: whichever floor is lower, and the crossover is a reliability of 0.637. An enrolment reading with a reliability of 0.8 leaves 0.447 of a coin’s imbalance where the screening category leaves 0.6028 — the rough number wins by about a sixth of a coin’s imbalance. An enrolment reading with a reliability of 0.5 leaves 0.707, and the precise category wins. Neither answer depends on the size of the trial: at a hundred units the rule reading a recording of reliability 0.8 leaves 0.443 of a coin’s imbalance, and at four hundred 0.435, the same floor within noise at both.

This is a different kind of advice from the field’s usual. The earlier essays compared rules on a fixed covariate and found that reading the number beats cutting it by a margin that grows without limit. Once the covariate is recorded with error, the comparison is between instruments, it has a definite answer at every reliability, and the answer can go either way. A protocol that specifies a continuous-covariate rule and says nothing about how the covariate is recorded has specified half of the design.

It is also a comparison a trial can make before it starts. Test–retest reliabilities for blood pressure, body mass, most validated questionnaires and many laboratory values are published, and both floors are one square root away from them.

What this says about the rule #

None of this is an argument against reading the number. At every reliability the rule leaves less imbalance than a coin, and less than minimisation on a split as long as λ is above 2/π. What it removes is the argument that reading the number changes the kind of answer — the exponent — rather than the size of it.

Three things follow for a trial that runs it.

The reliability is the first number to ask for. Most covariates a trial balances have a published test–retest reliability, and the retest essay shows how few repeated measurements are needed to estimate one. $\sqrt{1 - \lambda}$ converts it straight into the share of a coin’s imbalance the rule will leave, before any patient is enrolled.

Balancing W and adjusting for W are substitutes, not complements. The rule that balanced the recorded number and a coin whose analysis adjusted for it reach the same precision. Doing both buys nothing further on precision and is worth doing anyway for the error rate. The ceiling — $\sqrt{\sigma^2/n}$ — is set by the recording and is reachable only by recording better.

A rule that reads two recordings of the same covariate can do better than either. Two independent readings average to a recording of higher reliability, and since $\sqrt{1 - \lambda}$ is steepest near one, the second reading is worth more to the rule than it looks. This is the one design lever the closed form points to that the balancing rule itself does not have.

The boundary with adjusting for a shadow is worth drawing exactly. There, a covariate recorded with error is used to remove confounding in an observational comparison, and what is lost is bias: the share of the confounding removed is below the reliability. Here the assignment is randomised, so nothing the recording misses can bias the estimate; what it loses is precision, and the loss is $\sqrt{1 - \lambda}$ of a coin’s imbalance rather than any bias at all. The same instrument, the same λ, and two different costs because the two designs ask different things of it.

Still open: two readings, and when the second is worth a slot #

The closed form says a second independent reading raises the reliability from λ to 2λ/(1 + λ), which moves the floor from $\sqrt{1 - \lambda}$ to $\sqrt{(1 - \lambda)/(1 + \lambda)}$ — at 0.8, from 0.447 to 0.333. What it does not say is how that compares with spending the same effort elsewhere: a second reading of the one covariate the rule reads, or a first reading of a second covariate, which balancing more than one number prices at a cost in how well each is held. The measurement is a rule given two recordings of X and one of a second covariate Z, in all three combinations, scored on the precision of the treatment effect when both drive the outcome. It would say whether the field’s covariate list should be chosen by importance or by reliability, which no trial protocol currently distinguishes.

What is claimed here, and what is not #

This essay takes a covariate-adaptive rule whose covariate is recorded with error. The claims are that the imbalance it leaves in the true covariate is held above $\sqrt{1 - \lambda}$ of a coin’s and lands on it; that minimisation on a median split is held above $\sqrt{1 - 2\lambda/\pi}$, so a median split is a recording of reliability 2/π for this purpose; that on the true covariate the rule’s rate returns to the coin’s −½ while its rate on the recording stays near −1; and that the unadjusted analysis after the rule and a coin’s analysis adjusted for the recording reach the same precision, $\sqrt{(\sigma^2 + \beta^2(1 - \lambda))/n}$.

What stays out, named as a decision: errors that are not classical — errors correlated with X, or with the assignment, where the residual is no longer invisible to the rule and the closed form fails; more than one covariate, deferred above; non-normal X, where E[X | W] is not linear and a rule reading W can see some of the residual through its curvature, which the skewed-covariate essay shows is the kind of effect that moves a few points; and the rerandomisation test, which stays exact under any recording because it re-runs the rule on the same W and is the experiments that could have happened whatever was recorded.

The boundary against the field’s first rule is the variable the rate is measured on. Everything about the criterion, the running score and why the exponent differs is established there and used here; what is new is that the exponent is a property of the recording rather than of the covariate.

The checks, and the refusal that makes them mean something #

The closed form $\sqrt{1 - \lambda}$ is checked against the rule at six reliabilities and three trial sizes, and the floor for the median split against minimisation at the same settings. The rate figure checks at every slider value that the rule’s slope on the recording is near −1 while its slope on the true covariate is no steeper than −0.7. The analysis figure checks that the rule unadjusted and a coin adjusted for the recording both sit within 0.006 of the closed form. The refusal is a recording with no error presented as one with error — λ = 1 claimed to leave $\sqrt{1 - 0.8}$ of a coin’s imbalance — which the checks must reject, because a check that passed it would be measuring the floor and not the recording.