A second reading or a second covariate

A number read with error found the limit a recording sets on a balancing rule. The rule sees $W = X + e$ and the outcome depends on $X$. With a reliability $\lambda = \operatorname{var} X / \operatorname{var} W$, the best it can do leaves $\sqrt{1 - \lambda}$ of a coin’s imbalance in $X$, whatever the trial size and however well it balances $W$. At 0.8 that is 45% of a coin’s root-mean-square imbalance, and 20% of its variance, back on the coin’s $1/\sqrt{n}$ rate: the rule’s gain has a ceiling, and the ceiling is set when the baseline form is designed.

That essay closed with an arithmetic fact and an unanswered question. A second independent reading raises the reliability from $\lambda$ to $2\lambda/(1 + \lambda)$, and so moves the floor on the variance from $1 - \lambda$ to $(1 - \lambda)/(1 + \lambda)$. At 0.8, from 0.2 to 0.111. What it did not say is how that compares with spending the same effort elsewhere, for instance on a first reading of a covariate the rule is not reading at all. Nor did balancing more than one number say: that essay priced holding a second function as a cost in how well each is held, on the assumption that every function is recorded exactly.

This is a question every protocol answers, usually without saying so, when it fixes the baseline form. Should the list of covariates be chosen by importance, or by reliability? A trial gets one more baseline measurement. Should it be the clinic’s blood pressure a second time, or the thing the investigators think might also matter?

The same units, four rules #

The measurement gives a rule four options, all on the same units within each trial. Each unit has a true $X$ and a true $Z$, independent normals, and each reading of either is the true value plus independent noise at the same reliability $\lambda$.

• One reads a single $W$.
• Twice reads the mean of two readings of $X$.
• Other reads one reading of $X$ and one of $Z$.
• Both reads two readings of $X$ and one of $Z$.

Each rule assigns on its readings, leaning to the better arm with probability 0.8. What is measured is the difference between the arms in the true values, whose variance as a share of what a coin leaves is the same quantity the earlier essay’s floor is about.

The figure at the head of this essay is the result at two hundred units and a reliability of 0.8.

One reading leaves 21.4% in $X$, against a floor of 20%, and 100.1% in $Z$: the rule has never heard of $Z$, and $Z$ gets a coin’s imbalance.

Two readings of $X$ leave 12.3% in $X$, against a floor of 11.1%, and 105.0% in $Z$, which is a coin’s within its sampling error.

One reading each leaves 21.7% in $X$ and 21.5% in $Z$.

Two of $X$ and one of $Z$ leaves 13.4% and 22.0%.

The third row is what the whole essay turns on. Holding a second covariate has cost the first almost nothing: 21.4% with one covariate held, 21.7% with two, at two hundred units. Most of what the two-function essay priced is a cost of holding functions that compete inside a small trial, and the floor here is a ceiling a recording sets well before that cost starts to bind. So a first reading of $Z$ brings its imbalance from 100% to 21.5%, a saving of nearly 79 points. A second reading of $X$ brings $X$’s from 21.4% to 12.3%, a saving of about 9.

It takes no more than this to see which way the comparison will go. What remains is how important $Z$ has to be for the 79 points to count for more than the 9.

At every ratio of importances #

Let the outcome depend on $\beta_X X + \beta_Z Z$, and let $w = \beta_Z / \beta_X$. An unadjusted estimate’s variance beyond its noise is then, in units of $\beta_X^2$ times a coin’s imbalance, just the left-over share in $X$ plus $w^2$ times the left-over share in $Z$. So four pairs of numbers give every comparison at every set of weights, with no further simulation.

The curves cross at $w = 0.335$. If $Z$ drives the outcome less than a third as strongly as $X$, a second reading of $X$ does more. If it drives it more, the first reading of $Z$ does. By $w = 1$, when the two covariates matter equally, the rule that reads one of each carries an excess of 0.43 against 1.17 for the one that read $X$ twice, which is a factor of nearly three. Even at $w = 1/2$ it is 0.27 against 0.39.

So the immediate answer is that a second reading of a covariate already on the form is a surprisingly poor buy at this reliability. It wins only against a covariate that barely matters.

The crossing is arithmetic #

That is not a fact about two hundred units, and it does not need a simulation at all. A rule reading one recording at reliability $\lambda$ can remove $\lambda$ of a covariate’s squared importance, because what it leaves is $1 - \lambda$. So:

• The first reading of a covariate is worth $\lambda\beta^2$.
• The second reading raises the reliability to $2\lambda/(1 + \lambda)$, and it is worth the difference, which works out to $\lambda(1 - \lambda)/(1 + \lambda)\,\beta^2$.

Set a first reading of $Z$ against a second reading of $X$, at the same reliability:

$$\lambda\beta_Z^2 = \frac{\lambda(1-\lambda)}{1+\lambda}\beta_X^2 \quad\Longrightarrow\quad w = \sqrt{\frac{1-\lambda}{1+\lambda}}.$$

That is the second-reading floor the earlier essay wrote down, in root-mean-square units: 0.333 at a reliability of 0.8, so the two calculations are the same one seen from each side. It has no trial size in it and no rule in it. It is a statement about how much of a covariate a recording exposes.

The counted crossings agree: at 200 units 0.185, 0.227, 0.335, 0.417, 0.614 and 0.739 at reliabilities of 0.95, 0.9, 0.8, 0.7, 0.5 and 0.3, against closed forms of 0.160, 0.229, 0.333, 0.420, 0.577 and 0.734. The points at a hundred units sit a little above the line at the high-reliability end: 0.213 against 0.160 at 0.95. There the small cost of holding two functions does show up, and it pushes the crossing up a little in favour of the second reading.

Read along the line, the picture is the reverse of what a measurement-minded protocol tends to assume. A covariate read at a reliability of 0.95, a good laboratory measurement, should be re-read only if nothing else on the list matters more than 16% as much. A covariate read at 0.5, a rough clinical scale, should be re-read only if nothing else matters more than 58% as much. Even at 0.3, the worst reading here, a second reading only beats something else that matters less than three quarters as much. A second reading looks like rigour and it is almost always outbid. The exception is a scale so bad that its one reading has given up most of the signal, together with a list of other covariates that all seem nearly irrelevant, which is a rare pair of beliefs to hold.

A small knapsack #

The same two lines are the first entries in a general table. The mean of $k$ readings has reliability $k\lambda / (1 + (k-1)\lambda)$, and the $k$-th reading of a covariate is worth the increase.

At a reliability of 0.9 the four readings are worth 0.900, 0.047, 0.017 and 0.009. At 0.8, 0.800, 0.089, 0.034 and 0.018. At 0.5, 0.500, 0.167, 0.083 and 0.050. At 0.3, 0.300, 0.162, 0.101 and 0.069.

Each covariate on a candidate list brings a column like that, multiplied by its own $\beta^2$. The best list of $m$ measurements for a rule to hold is then found by taking the $m$ largest entries across all the columns, and since each column falls, the choice never takes a later reading without the earlier one. There is no search and no simulation. It needs only a guess at each covariate’s importance and at its reliability, both of which a protocol committee has views about and neither of which is ever written down in that form.

That also answers the question in the form it was put: should a covariate list be chosen by importance or by reliability? By importance squared times reliability, for the first readings, and the second readings almost never reach the list. Consider the difference that makes. A covariate with half the importance of another, read twice as reliably at 0.9 against 0.45, has a first-reading value of $0.25 \times 0.9 = 0.225$ against $1 \times 0.45 = 0.45$. Importance, being squared, dominates reliability unless reliabilities are extreme. A trial’s first-reading list is chosen largely by importance, and reliability is a correction to it.

A worked list shows what the table does and does not say. Take three candidates. $A$ matters most and reads badly, with importance 1 and reliability 0.6. $B$ matters at 0.7 and reads at 0.9. $C$ matters at 0.4 and reads at 0.8. The first readings are worth 0.600, 0.441 and 0.128. The second readings are worth 0.150, 0.023 and 0.014, and a third of $A$ is worth 0.068. So a list of four measurements is $A$, $B$, a second $A$ and then $C$, and a fifth is a third $A$. The second reading of the unreliable important covariate does make the list, ahead of a first reading of the minor one, because the crossing at a reliability of 0.6 is 0.5 and $C$ sits at 0.4. A trial of this shape does right to read its most important and worst-measured covariate more than once, and no others. That is the kind of statement a protocol can defend, and it is rarely what a protocol’s list is based on, even though what a balanced trial is worth has long since set up the arithmetic of the benefit.

What the rule can and cannot do about it #

Two things limit how far this can be pressed.

The value is a ceiling and a real rule falls short of it. The rows of the first figure are slightly above their floors: 21.4% against 20%, and 12.3% against 11.1%. The rule’s own residual imbalance adds about a point in the variance. At a hundred units this grows, to about 25% against 20% at 0.8 when one covariate is held, and that is what pushes the crossing up at small samples. At small trials the arithmetic overstates the case for holding many things, because what the constraint costs has by then entered in earnest.

An unbalanced covariate can still be adjusted for, if it has been recorded. This entire comparison is on the unadjusted estimate, which is what a design controls. Recording a covariate has a second payoff that this essay leaves out: an analysis adjusting on the recording, which the earlier essay found recovers most of what a design leaves. Reading $Z$ pays twice in that way, and a second reading of $X$ pays twice too, by raising the reliability of the adjuster. The comparison between them after adjustment has the same shape. In the limit of perfect balance both reduce to how much of the outcome the adjusted covariates can explain, and that is again $\lambda\beta^2$ for a first reading and $\lambda(1-\lambda)/(1+\lambda)\beta^2$ for a second, so it should cross in the same place. That is an argument rather than a measurement, since no adjusted analysis has been run here.

There is a third, softer limit. A second reading is often cheap, since the unit is there and the machine is on, whereas a new covariate may need a separate assay. Nothing above has any cost in it. It is in units of one measurement, and if one reading of $Z$ costs ten of $X$ the entries in the table should be divided accordingly. With a 0.8 reliability, it is 0.089 against $0.8w^2$ per reading, and if a $Z$ reading costs ten of $X$, these balance at $w \approx 1.05$, which is a covariate about as important as the first. So cost could genuinely change the answer, at that kind of ratio, and it is worth doing the sum.

How the numbers were made #

Three thousand trials at each of three trial sizes and six reliabilities, with the true covariates and all readings drawn independently in each trial, and all four rules plus a coin assigning on the same units in that trial. The imbalance is measured on the true values, not on the readings, and reported as a variance ratio against the coin’s. At this number of trials the ratio has a relative error of about 2.6%, which is why $Z$ under the first two rules reads anywhere from 94% to 105% for what is exactly a coin’s imbalance.

Two routes meet at the crossing: the closed form above, and the crossing read off the four counted pairs. They are required to agree within 0.1 at two hundred units and above, at every reliability. At the same time holding a second covariate must cost the first less than three points at two hundred units. And the claim that a second reading of $X$ still does better against a covariate half as important is required to be refused at a reliability of 0.8, where it is, by 0.39 against 0.27.

The reliabilities are the same for both covariates throughout the sweep. The table of values does not need that, and the formula generalises at once: at different reliabilities the crossing is $w^2 = \lambda_X(1 - \lambda_X)/((1 + \lambda_X)\lambda_Z)$. An unreliable second covariate needs to matter more, as the square root of how much less reliable it is. Measuring that is a straightforward rerun and has not been done.

Still open: two covariates that are one, and readings that share an error #

A correlated pair. Everything here makes $X$ and $Z$ independent. In a real baseline table they are not, and a correlated $Z$ does some of $X$’s work: a rule reading $Z$ gets a share of $X$ for free. That cuts both ways. A $Z$ correlated with $X$ is less valuable as a separate target and more useful as a second reading of $X$, and in the limit of a correlation of one, it is a second reading. A rule’s value for a correlated $Z$ is a closed form in the same arithmetic. It would place the essay’s two options at the ends of one line, a second reading as the case of a perfectly correlated covariate, and that seems worth writing out.

Errors that are shared. Every second reading here has an error independent of the first. If two readings on the same morning share part of their error, because a blood pressure is high today for reasons that will be gone in a week, then the reliability ceiling of any number of readings is below one, and a second reading’s value falls faster than the formula. That makes the case above stronger, but by how much depends on a variance component that a repeat-measures substudy would have to supply.

What the protocol’s list is worth #

This essay takes a balancing rule choosing between a second reading of a covariate and a first reading of another, with independent normals and independent recording errors. Its claims are that holding the second covariate costs the first almost nothing at two hundred units; that a first reading of a covariate $w$ times as important beats a second reading of the first once $w$ passes $\sqrt{(1-\lambda)/(1+\lambda)}$, which is 0.333 at a reliability of 0.8 and has no trial size in it; that the counted crossings agree with this at every reliability measured; and that the value of each further reading has a closed form, from which a list of measurements can be chosen as a knapsack.

What stays out, as a decision: adjusted analyses, argued about above and not measured; correlated covariates and shared errors, deferred above; and costs, which are not data and which any protocol has to supply. The boundary against balancing more than one number is that it prices holding functions recorded exactly, and this one finds that once recording error is present its ceiling binds first. The boundary against the rule that reads the number is that it asks what reading a continuous value buys over reading a category, and this one asks how many such values to read.