How many functions a trial can hold

Three functions of one number taught a covariate-adaptive rule to balance a basis rather than a mean. Reading the covariate, its square and a threshold at one, the rule halved the variance of the treatment estimate against all three of those shapes, at a cost of two points against the shape the plain rule was built for, and took the worst case over them from a coin’s to about half of it. On two hundred units the insurance was cheap.

It also said why the sequence cannot go on. Every function added is a constraint the assignment has to satisfy with the same units; the constraints compete, as balancing several covariates at once had already measured; and a basis rich enough to cover every shape is a basis with as many functions as units, which leaves the rule nothing to do but follow it. Somewhere between three and $n$ the constraints exhaust the design. The essay named that point and left it unmeasured, because finding it needs a sweep over the basis and a sweep over the trial size at once.

This is that sweep. Six functions, added in the order a designer would add them, at four trial sizes. The rule is the one the field has used throughout: at each arrival it computes the information each arm would leave about the treatment effect under the model the basis defines, and takes the better arm with probability 0.8.

The basis, grown one function at a time #

The six functions are the covariate itself; its square; a threshold at one, which completes the earlier essay’s basis; the cube; a threshold at −1; and a median split. Each is standardised on the normal, so no function’s scale decides how much the rule attends to it. The trial sizes are 50, 100, 200 and 400 units.

The quantity measured is the one everything else follows from. What a design does to an unadjusted estimate of the treatment effect is the imbalance it leaves between the arms in whatever function the outcome is built from, so the measurement is that imbalance, directly: its variance under the rule, as a share of its variance under a coin, for four shapes an outcome might have — linear in the covariate, a threshold at one, quadratic, and a median split. A share of one is a coin; zero is perfect balance. Where the shape and the noise contribute equally to the outcome, as they did in the earlier essay’s table, the variance of the estimate against a coin’s is about half of one plus that share, so a share of 0.02 is the earlier table’s 0.51.

At a hundred units the picture has two parts, and they move in opposite directions.

A shape the rule is not yet reading keeps most of a coin’s imbalance. The quadratic keeps 105.9% of it while the rule reads only the covariate — a coin’s, within the sampling error, as it is at every size, from 95.9% to 108.5%. A threshold at one keeps 60.8%, because a threshold is correlated with the covariate and some of its balance comes along for free. The median split keeps between 27% and 39% until the sixth function, which is itself.

A shape the rule is reading keeps a small share, and the share grows with every function added. The linear shape keeps 3.0% with one function, 5.5% with two, 7.9% with three and 16.4% with six. Reading more never makes the rule worse at a shape outside its span; it always makes it worse at the shapes inside.

What each function read costs #

The second part is the one the earlier essay could not see at two hundred units, and it has a simple form.

Multiply the linear shape’s share by the trial size and the four sizes land on one curve: about 3.0 to 3.5 for one function, 5.0 to 5.6 for two, 7.3 to 8.6 for three, and 15.0 to 18.6 for six. So the residual falls like one over $n$ — the rate a rule that reads the number was found to have on its own covariate — and the constant in front of it is set by how many functions the rule is holding.

The constant does not grow evenly. The fourth function, the cube, costs the most — the linear shape’s constant jumps from about 8 to about 14 — and the fifth and sixth together cost less than it did alone. The likeliest reason is that the cube is the function most nearly confused with the covariate in a finite sample: it is orthogonal to it on the normal, but its sample values are dominated by the same few units in the tails that dominate the covariate’s, so a rule holding both is asking the same handful of arrivals to settle two quantities at once. Nothing here isolates that mechanism; the order-dependence it implies is the open question at the end.

The quadratic shape, once it is in the span, shows the same law with a larger constant — from about 8 with two functions to about 23 with six — because the square depends on the tails more heavily than the covariate does, and the tails are where the units are fewest.

The worst case #

A designer adds functions to protect the shape the trial is most exposed to, so the reading that matters is the worst of the four.

With one function the worst case is the quadratic at every size, at about a coin’s: 103.5%, 105.9%, 95.9% and 108.5% at 50, 100, 200 and 400 units. Reading the square removes that exposure at once, and the worst case falls to around 40% at every size — now held by a shape the basis still lacks.

After that the four sizes part company.

At four hundred units every further function pays. The worst case falls from 41.8% with two functions to 32.1% with four and 5.8% with six, because the in-span cost there is a few per cent and the out-of-span exposure is all that is left to remove.

At two hundred it pays the same way, more slowly: 44.2%, 32.9%, 11.7%.

At fifty it barely moves. The worst case is 38.6% with two functions, 37.2% with three, 37.0% with four and 41.7% with six. The functions added after the second remove exposure on the shapes outside the span and give it straight back on the shapes inside: by six functions the linear shape keeps 29.9% of a coin’s imbalance and the quadratic 41.7%, where with one and two functions they kept 6.8% and 18.0%.

Six functions, and the worst shape is one the rule is reading #

The most telling line in the table is the last one. With six functions, at every trial size, the shape that holds the worst case is the quadratic — the second function the rule was given. At four hundred units it keeps 5.8%; at fifty, 41.7%. The median split, which held the worst case at four functions at every size above fifty, has been added and is balanced to 2.0% at four hundred units and 17.7% at fifty.

That is the point the earlier essay predicted and could not place. A basis is protection only while the cost of holding it stays below the exposure it removes, and the cost scales with the number of functions over the trial size. At fifty units it overtakes the exposure after two or three functions; at four hundred it has not overtaken it by six.

So a rough exchange rate can be read off. Each function adds roughly three units of residual to the linear shape’s constant, and a constant of $c$ at $n$ units is a share of $c/n$. A designer willing to let an in-span shape keep a tenth of a coin’s imbalance can hold about as many functions as the trial size divided by thirty: one or two at fifty units, three at a hundred, six at two hundred. That is not a rule anybody should apply mechanically — the constant depends on which functions, and the cube was dearer than the two thresholds together — but it is the order of magnitude, and it is far smaller than the $n$ functions at which the design is literally exhausted.

What the closed forms leave out #

The shares this field and the dictionary field compute in closed form are projections: what a rule removes of a shape if it balances its basis exactly. On that reading a function added to a basis can never hurt — a larger span contains the smaller one, so every shape keeps at least what it had — and the only question is which functions are worth their place in the span. Where the guarantee is exactly zero and its neighbours are written entirely in that currency.

A trial of fifty units does not balance anything exactly. It balances the covariate to within a residual of about 3 over $n$ of a coin’s imbalance with one function and about 16 over $n$ with six, and the residual is paid by every shape the basis contains. So the closed forms describe the limit of this table as the trial grows, and the table describes how far a real trial is from that limit. At four hundred units the two readings agree on the ranking of bases — more is better, all the way to six. At fifty they disagree: the projection says six functions protect every shape here completely, and the trial leaves the quadratic with two fifths of a coin’s imbalance.

The disagreement is not a failure of either calculation. The closed form answers what could a basis remove, and the sweep answers what does a rule with this many arrivals remove; the first is a property of the span, the second of the span and the sample size together. A designer reading a closed-form table at fifty units is reading the right answer to a question the trial cannot ask.

A threshold in the tail found the same kind of gap from the other side: a threshold far from the middle is a function a mean-balancing rule removes little of, and also a function few units fall beyond, so a rule told about it has little to work with. The two thresholds in this basis are at one standard deviation either side, where about a sixth of the units sit; a basis holding a threshold at two would hold a function only one unit in forty-four exceeds, and its constant would be correspondingly larger.

The rule is not out of randomness; it is out of units #

It would be natural to read the fifty-unit curve as the rule losing its randomisation, the endpoint the earlier essay described. It is not that. The rule still takes the better arm with probability 0.8 at every arrival, whatever the basis, so its predictability is the same at one function and at six, and what a guesser can do with it does not change.

What changes is the information each arrival carries. A rule holding $k$ functions cannot balance any of them until it has seen $k + 2$ units, since its information matrix is singular before that; it spends the early arrivals on the arm totals alone. After that, each arrival has to serve $k$ imbalances with one binary choice. The linear shape’s residual is the price of the arrivals spent on the other $k - 1$, and the price is paid in units, which is why it falls like one over $n$ and why its constant counts functions.

That separates two endpoints that are easy to run together. A rule runs out of randomness when its probability of the preferred arm goes to one, which no basis size here forces. It runs out of units when the constant over $n$ becomes as large as the exposure it was meant to remove, which at fifty units happens at the second or third function. The second comes first, by a long way.

What the analysis can still do #

None of this is the analysis’s problem in the sense of making it wrong. An adjusted analysis that includes the shape the outcome actually follows removes that shape’s imbalance whatever the design left, and the analysis and the shape measured the size of that lever. What the design’s residual costs is precision in the unadjusted estimate and in any analysis that adjusts for the wrong function — which is every analysis, until the shape is known.

So the exchange rate above is the price of insuring the unadjusted analysis, paid in the imbalance of the shapes already insured. A trial that will adjust for a rich basis anyway has less reason to ask the design to hold one; a trial that reports an unadjusted primary analysis, as most do, has every reason, and the trial size decides how many functions of that insurance it can afford.

Two thousand trials, one coin and six rules on the same units #

Two thousand trials at each size. On every trial the same units are assigned by a coin and by the rule at each basis size from one to six, and the imbalance in each of the four shapes is the difference between the arms’ means of that shape, so every share is a ratio of two variances computed over the same draws of the covariate. The covariate is standard normal; the shapes are standardised to unit variance on the normal; the rule takes the arm with the larger information with probability 0.8 and breaks ties with a coin, as it has throughout the field.

Each share is a ratio of two variances and has a sampling error of about four or five per cent of its own size at two thousand trials, which is why the fifty-unit worst cases from two to five functions should be read as equal rather than ordered; the claims above rest only on differences several times that.

Each function read costs every shape already held #

The claims are that every function a rule reads raises the imbalance it leaves in the shapes already inside its span; that this residual falls like one over the trial size with a constant that grows from about three to about sixteen as the basis goes from one function to six; that at fifty units the worst case over the four shapes stays between 37% and 42% from two functions to six, while at four hundred it falls from 42% to 6%; and that at six functions the worst shape at every size is one the rule is reading.

Not claimed: anything about other orders of adding functions, where the cube’s unusual price suggests the answer depends on which function comes when; anything about rules with a different probability of the preferred arm, which would move every constant; and anything about several covariates, where the functions compete without the correlation that makes functions of one number cheap.

Still open: the order the functions are added in #

The cube cost more than the two thresholds after it together, and it was fourth only because a designer thinking in polynomials would put it there. A basis of six functions can be built in seven hundred and twenty orders, and the in-span constant depends on how strongly each new function is confused with the ones already held in a finite sample — which the normal’s orthogonality says nothing about. The measurement that would turn the exchange rate above into a design rule is the best order, or the best subset of a given size, at each trial size: the basis that minimises the worst case once its own cost is counted. At fifty units the prediction is that it holds two or three functions and none of them is the cube; whether the best four at a hundred units include a threshold the earlier essay never tried is the part that would be worth knowing.