The list it was chosen on

How many functions a trial can hold grew a basis one function at a time in the order a designer would reach for them: the covariate, its square, a threshold at one, its cube, a threshold at minus one, a median split. It found the cost of holding more functions at a small trial, the point at which a further constraint stops paying, and an awkward fact about the cube, which cost more than the two thresholds after it together. It asked whether the cube was fourth only because a designer who thinks in polynomials would put it there.

The obvious framing of that question is a search over orders. Six functions can arrive in 720 sequences, and the cost of each new one depends on how confused it is with the ones already held, so the best sequence looked like the design rule the essay wanted. This essay was started in order to run that search.

It does not need to be run. There is no such thing as the order of addition.

Sixty-three designs, not seven hundred and twenty #

The rule reads the basis through one number: at each arrival it computes how much information each arm would leave about the treatment effect under a linear model on the basis, and leans towards the arm that leaves more. That information is $S_{aa} - S_{ax}' S_{xx}^{-1} S_{ax}$, where $S_{xx}$ is the basis’s own cross-product matrix. Reorder the columns and every one of these matrices is permuted the same way; the scalar that comes out is unchanged. A basis is a subspace reached the same conclusion from the other direction, by showing that a rule cannot tell a basis from any invertible recombination of it.

So what the earlier essay drew as a sequence of six designs is a sequence of six points in a set of 63: the subsets of six functions, each of them an unordered collection. The check is direct. Give one rule four functions in the designer’s order and give it the same four in another order, on the same units with the same stream of coin flips, and every one of eighty assignments comes out identical. The cost of adding the cube “fourth” was a cost of holding the cube, compared with the three functions it happened to follow.

That collapses the question to a size that can simply be counted. It also changes what the earlier essay’s curve means. Its downward slope was a fact about one chain of nested subsets. Nothing said the chain was a good one.

The chain was not a good one #

Score each of the 63 subsets the way the field has scored bases, by the largest share of a coin’s imbalance it leaves in any of four shapes an outcome might take: linear in the covariate, a threshold at one, quadratic, a median split. A share of 100% is a coin and 0% is perfect balance. The figure at the head of this essay is all of them at a hundred units, by how many functions the rule reads.

Size alone is a poor guide. At four functions the subsets run from 12.1% to 72.9%. A rule reading four functions can do six times better or worse than another rule reading four, and the worst four-function subsets are worse than the better ones holding only two. The designer’s chain, drawn through the cloud, sits above the best subset of each size at every point from three to five functions, and at four it is 31.5% against 12.1%: two and a half times what the same number of functions could have delivered.

The chain’s own shape turns out to be partly an accident of where it put the cube.

The two paths meet at two functions, where the best pair is the covariate and its square, and at six, where there is only one subset. Between them they tell different stories. The designer’s order at two hundred units falls slowly and keeps falling: 37.6%, 30.0%, 24.7%, 13.2%, which reads as more functions always helping at that size. The best subsets reach 6.8% at four functions and rise to 13.2% by six, which says the opposite: past four, more functions cost more than they buy, at two hundred units as well as at a hundred. The earlier essay’s claim that a large trial could carry the whole basis was true of the chain and of no better subset; what the constraint costs had priced the rule’s budget, and the budget is being spent badly by that order.

At fifty units the earlier essay predicted two or three functions and no cube, and it was right about the cube. The best three are the square, the threshold at one and the median split, at 31.2%; the best four add a threshold at minus one and gain only 0.1 of a point. The chain there sits at 36.2% and 36.9%. So the small-trial prediction holds, and it holds with an unexpected omission: the best small bases leave out the covariate itself. The thresholds and the square between them carry enough of its balance for free, and the slot is better spent elsewhere. A designer’s first move, the mean, is what the small trial’s best basis drops.

Choosing on one set of trials and reporting on another #

A best subset read off the draws it was chosen on flatters itself, by however lucky it was among 63 candidates. The winner’s curse is the same arithmetic on a different league table, and the scoring here is a maximum of four noisy variance ratios, which is the shape that bias likes most. So every number above was measured twice: 1,500 trials pick the subset, and a further 1,500 on separate seeds report it.

It is reassuring and it is also a small surprise how little the second set moves things. Over all sizes and all three trial sizes the mean gap between the chosen and reported readings is under half a point, with the chosen reading no more often lower than higher. The largest single change is at two hundred units and three functions, 26.4% to 19.3%, in the favourable direction; a ratio of two small variances is simply noisy. At 1,500 trials the spread among subsets is so much larger than the noise in each that the choice is not the place to look for the error.

The error is somewhere else entirely.

The list it was chosen on #

Read the subset that wins at a hundred and at two hundred units: the covariate, its square, a threshold at one, a median split.

Now read the four shapes the worst case was scored on: linear in the covariate, quadratic, a threshold at one, a median split.

They are the same four. Each shape was on the list of things the outcome might be, and each function that exactly reproduces it was available in the basis. The best subset of four, by worst case over four shapes, is the four. It is a correct answer to the question asked, and it is not an answer to anything else. The search had rediscovered its own scoring rule.

This is not a bug in the search. If a designer really knows the four shapes an outcome can take, then balancing exactly those four is what they should do, and the earlier essay’s order was wasting a slot on a function no shape needed. But nobody knows that, and the list of four was never offered as the truth; it was the set of shapes that this line of essays had happened to study. So the choice has to be read on shapes it was not told about.

The six are the cube, thresholds at −1, −0.5, 0.5 and 1.5, and an exponential in the covariate: things an outcome might plausibly do that are not on the list. Each reading is the worst share of a coin’s imbalance, on 1,500 fresh trials.

The ranking reverses:

The design that won by a factor of two and a half on its own list loses by a factor of two off it. The designer’s order, which spent a slot on a cube no shape on the list required, is by that accident the design that holds up when the list is wrong.

One shape does nearly all of it #

The off-list figure is a worst case, so it is worth asking whether the worst-case pick is bad everywhere or bad once.

It is bad once. On the cube the worst-case pick leaves 77.0%, against 28.1% for the designer’s four and 38.7% for all six. On the other five off-list shapes the three designs are nearly tied, and on the exponential the worst-case pick is the best of them, at 25.2% against 31.8% and 34.9%. So a designer can read this as a very specific warning instead of a general one. Dropping the cube costs a lot in the cube’s own direction, because nothing else in the basis resembles a cube once the linear part has been taken out of it, and it costs very little anywhere else. A threshold looks more or less like every other threshold and like the covariate. A cube, orthogonalised, does not look like anything.

The other readable thing in that figure is the row of thresholds. A threshold at 0.5 is left at between 31.8% and 38.9% under all three designs. A threshold at 1.5 is 35.4% to 37.5%. A threshold at −0.5 is 30.7% to 32.5%. Holding four or six functions buys about two thirds of the balance on a threshold none of them names, and more functions do not improve on that. The threshold at −1 is the one exception, at 14.8% under all six, and it is the one that is in the six. That sets a floor on what any finite list can do for a threshold, and it says that adding functions is the wrong way to approach an unknown cut. The covariate’s own line, by comparison, leaves between 43% and 70% in those same thresholds. So going from one function to four buys about a third of a coin’s imbalance, and going from four to six buys about nothing unless a function names the shape.

Two questions that had been one #

The earlier essay asked which basis a trial should hold, and that sentence hides two questions which now have separate answers.

Which basis is best against a list? That is answered by counting. Hold exactly the shapes on the list, and at a hundred units hold no more than four: 12.1%, falling to 6.8% at two hundred, with a further function making things worse at both.

Which basis is robust to being wrong about the list? That is what the cube is for, and on this measure the answer is a basis that holds something unlike everything else, even at some cost on the shapes it was nominally built for. The designer’s four lost nineteen points on the list and gained thirty-seven off it.

Which shapes are worth protecting asked how a basis should choose among shapes once each has a weight. This essay does not supply weights and it does not say the cube is likely. What it can say is that a comparison of bases on the worst case over a list reports, to within a small sampling error, whether the basis contains the list; and a protocol that defends its basis by such a table is defending something it could have written down without running anything.

The rule’s limit is not visible from here #

Nothing above says the rule is doing well in absolute terms at fifty units. There the best reading of any subset, 31.1%, is a third of a coin’s imbalance, and even the shapes that a design names exactly stay well above zero because four functions in fifty units are in one another’s way. What the extra function buys priced that competition for the function pair; here it means that at small sizes the gap between subsets narrows, from a factor of 2.6 at a hundred to 1.2 at fifty. The search is worth more where there are more units to use on it, which is the reverse of where a designer most wants help.

There is also the matter of what a share means for an estimate. If the shape and the noise contribute equally to the outcome, the variance of the unadjusted estimate against a coin’s is about half of one plus the share. So 12.1% and 31.5% differ by about a tenth of the variance on a shape the design was built for, and 28.1% and 77.0% differ by about a quarter of it on the cube if the outcome is genuinely a cube. Those are real sums for a trial at this size, but they are sums that an analysis adjusted on the right function would reclaim, and the wrong-shape adjustment essay has shown how little an analysis on the wrong one recovers.

How the search was run #

Six functions, each standardised on the normal so that its scale does not decide how much the rule attends to it: the covariate, its square, a threshold at one, its cube, a threshold at minus one, a median split. Every one of the 63 non-empty subsets runs on the same units within a trial, with a rule that takes the better arm with probability 0.8. The measurement is the variance across trials of the between-arm imbalance in each shape, divided by the same variance under a coin.

Subsets are chosen on 1,500 trials at each of fifty, a hundred and two hundred units, and every reported number comes from a further 1,500 trials on separate seeds. The off-list readings are on 1,500 trials as well, using the four designs that the comparison concerns. Two checks hold it in place: two orderings of the same four functions are required to return the identical allocation on every unit, and the reading that the designer’s four are about as good as the best four is refused at a hundred units, where the reported difference is more than double.

The standard errors are not small in absolute terms. A ratio of variances on 1,500 trials has a relative error of about five per cent, so 12.1% is roughly 12.1 ± 0.6, and 25.4% against 31.5% is a real difference while 31.1% against 31.2% is not.

Still open: a list with weights, and the threshold the list does not have #

A prior over shapes. A worst case over four shapes is the most pessimistic thing to do with a list, and it is what makes the search tautological. An average over a distribution of shapes would not be: with a weight on thresholds spread continuously across the range, and a small weight on the cube, the best subset would be a genuine trade rather than a mirror. That is the same ground the game over a drawn basis stood on for a mixture of bases, and asking it of a fixed subset is a short extension.

A threshold the earlier essay never tried. The earlier essay asked whether the best four at a hundred units might contain one. This search cannot answer that, because its candidates were the same six. Extending the candidates to a grid of cuts gives thresholds a chance of winning on the off-list shapes, and it would say whether two cuts placed at about ±0.75, say, do more for an unnamed threshold than either a cut at one or the cube. The floor at about a third on an unnamed threshold is what such a grid would have to beat.

What this does and does not say about the cube #

This essay takes every subset of six functions for a rule balancing one normal covariate, at fifty, a hundred and two hundred units. Its claims are that the rule depends on the subset and not on the order; that at a hundred units four-function subsets leave between 12.1% and 72.9% of a coin’s imbalance in the worst of four shapes, with the designer’s first four at 31.5%; that the best four are exactly the four scored shapes, and that on six shapes outside the list they leave 77.0% in the cube against 28.1% for the designer’s four; that the best small bases omit the covariate; and that a threshold none of the functions names stays near a third of a coin’s imbalance whichever design is used.

What stays out, as a decision: other distributions of the covariate, since every function here is standardised on the normal; more than one covariate, where the functions compete across as well as within; and any claim that a cube is a likely outcome, which is a clinical judgement. The boundary against the threshold in the tail is that it measured what one function does for the same-shaped outcome, and this one searches the subsets of functions and finds that a search on a list answers only the list.