A box with a count

Two ways of writing down which assignments a trial may use have been competing in this field. A box gives each named function a tolerance: its imbalance between the arms must be within $a$ of zero, one function at a time. An ellipse puts one bound on the Mahalanobis distance of the whole vector of imbalances, the common recommendation in the rerandomisation literature. What the ellipse spends its assignments on set them side by side on sixteen units, where every equal split can be walked. The ellipse never runs out, where the box empties at the fourth function. At the same number of admissible assignments the ellipse holds the worst named function about half as tightly, because it spends its constraint on thin combinations of the basis that barely vary and that nobody named.

That essay’s closing sentence drew the obvious conclusion. Drop those thin directions and draw the Mahalanobis region on the rest. A region that never empties and holds named functions as tightly as a box might then exist, or the two properties might trade against each other at every cutoff.

The measurement has a straightforward answer to that. It also has an awkward one, which is that the question assumed something it did not need to.

One set of units was an anecdote #

The earlier essay’s numbers came from one set of sixteen units, and that needs fixing before anything else. On that set the box had 314 assignments at three functions, the thin direction carried 0.219 of a function’s variance, and the ellipse left about twice as much in the worst named function. A single draw of sixteen covariates can have an unusually thin direction or lack one altogether. So everything here is averaged over 24 sets of sixteen units, with all 12,870 equal splits walked on each.

That moves the picture a little. At three functions the box’s mean count is 354, but it is empty on 17% of the sets, not at the fourth function: the box runs out at three functions on one set in six, and at four functions on nearly two in three. The earlier essay’s set was on the lucky side. The ratio it found survives in a milder form: read at the box’s own count on the sets where the box has one, the worst named function keeps a share 0.072 of its own variance under the box and 0.134 under the ellipse, a factor of 1.9.

The figure at the head of this essay is that comparison across the sizes, on the sets of units where a box exists at all. The box stays at about 0.07 at every size, by construction: its tolerance stays fixed, and whatever survives has every named function within it. The ellipse climbs, to 0.196 at four functions and 0.194 and 0.182 at five and six. So when more functions are named the ellipse gets further from the box, which is what it should do if the thin directions are the problem, since more functions on sixteen units means thinner directions: the smallest direction’s variance falls to 0.187 at three functions, 0.078 at four, 0.031 at five and 0.023 at six, as a share of one function’s.

Dropping the thin directions #

Now the proposed repair. Compute the imbalances’ covariance, take its principal directions, drop any whose variance is below a stated share of one function’s, and read the Mahalanobis distance on the rest.

It helps at first and then does not, and it never helps enough. At three functions, the worst share is 0.134 with no cut, 0.097 at a cut of a tenth, 0.115 at a quarter, 0.211 at a half and three quarters, and 0.529 at one. At four functions, 0.196, 0.116, 0.165, 0.179, 0.179 and 0.381. At five and six the same shape occurs, with a noisier curve. The best cut at every size is somewhere around a tenth to a quarter, and at its best it is about one and a half times the box: 0.097 against 0.072 at three functions, 0.116 against 0.081 at four, 0.134 against 0.073 at five, 0.115 against 0.069 at six.

The bad end is obvious in hindsight. A principal direction with variance near one is not a thin combination nobody named: at three functions on these units it is mostly a named function. Dropping it means that function is not constrained at all, and it keeps most of a coin’s imbalance. So the cut must be low enough to keep everything real and high enough to get rid of what is thin, and on sixteen units that window is narrow and it moves with the draw: the noise in the curves, which sometimes rise and fall twice, is that window moving between sets.

There is a more fundamental reason it cannot catch the box. Even with the thin directions gone the ellipse is still an ellipse, with each kept direction weighted by the reciprocal of its variance. So a direction with a small variance gets a large weight, and the region remains uninterested in which functions are named. Cutting has removed the most extreme case of that and left the mechanism in place.

Shrinking towards a sphere #

A second repair handles the same problem without a threshold. Add $\delta$ to the imbalances’ covariance before inverting it. No ridge gives the ellipse; a large ridge gives a sphere, a plain sum of squared standardised imbalances with every function counted alike. It never empties either, because it has just a ranking of assignments by a number.

This does what the cut was meant to. At three functions the worst share goes from 0.134 through 0.106, 0.092, 0.086, 0.080 and 0.078 to 0.076 at a very large ridge, against the box’s 0.072. At four functions, 0.196 through 0.127 and 0.094 to 0.091, against 0.081. At five, 0.194 falls to about 0.079 at a ridge of a half and 0.086 at the sphere, against 0.073. At six, 0.182 falls to 0.073 and 0.080, against 0.069.

The sphere also is not a new invention. It is what balancing more than one number was using without naming it, at larger trials, whenever a rule treated a list of functions alike, so the behaviour it shows there at a hundred units and more is the same behaviour seen on sixteen. Part of what made the ellipse look principled is that it is the form the normal theory is easiest to write in: the truncation factor has a closed form, and a sphere’s does not. On a trial small enough to walk, that advantage is worth nothing, since the count can simply be read off.

So a plain sphere on the standardised functions holds the worst named function to within about ten per cent of a box at the same count. There is no threshold to choose and nothing to fit. The repair for the ellipse’s over-attention to thin directions is not to remove the directions but to stop dividing by their variance. A reader can see why this works: a sum of squares of standardised imbalances puts its budget on each named function evenly, and an assignment with one large imbalance scores badly under it, as it does under a box.

What the sphere gives up is the invariance that a basis is a subspace made so much of. Its region depends on how the span is written down. That is no loss if the functions were named on purpose.

A box does not empty, a tolerance does #

Then comes the part that the earlier framing had hidden. The box “runs out” because its tolerance $a$ is fixed in advance, and then the number of admissible assignments is whatever it turns out to be. On sixteen units with four functions that is frequently nothing. But nothing requires the tolerance to be fixed. A protocol could instead fix the count, the number of admissible assignments it wants, and set the tolerance to whatever delivers it.

That is a one-line change. Rank every assignment by its largest standardised named imbalance, and keep the top count. At the box’s own count, that set is identically the box’s, since the box at tolerance $a$ is exactly the assignments whose largest named imbalance is below $a$. It has the same members and the same worst share, 0.072, 0.081, 0.073 and 0.069. The hero figure cannot tell them apart.

And when the fixed box is empty, this one is not.

The sets where the box is empty are not easy ones. At a twentieth of the splits the box with its tolerance widened leaves 0.411 in the worst function at three functions, 0.301 at four, 0.492 at five and 0.674 at six. The sphere leaves 0.431, 0.317, 0.500 and 0.679, and the ellipse 0.542, 0.382, 0.593 and 0.735. So on those sets all three hold far less than a tight box could have, and there is no hiding that: those sets have a function that cannot be balanced well by any assignment, in this many units, at this many functions. But a region that reports “the best one twentieth of assignments on the named functions” is at least a region, and the ranking among the three stays the same. A fixed tolerance would have reported nothing and stopped the trial from starting.

So the question the earlier essay ended on, whether “never empties” and “holds named functions as tightly as a box” trade against each other at every cutoff, has an answer it did not anticipate. There is no trade. The two were one region all along, set by a count, and all of the difficulty lay in how the protocol had specified the thing.

What the unnamed functions get #

No region here is free. Each one has only a set number of assignments to spend, so tighter on what is named has to mean looser somewhere.

The cost lands on the functions a region does not name, and it is small. At three functions held out of six, the share left in the three unnamed functions is 0.609 under the box, 0.578 under the ellipse and 0.616 under the sphere. At four, 0.661, 0.581 and 0.723, and at five, 0.823, 0.712 and 0.846. So the ellipse does hold what it was not asked about somewhat better: three to ten points, of a coin’s 100, and that is a real part of the argument for it. It is a real gain, on a smaller thing. If a trial genuinely does not know which functions matter, it should take that seriously; the list a design is chosen on found the same bargain in the choice of the functions themselves, and there it could be a large one.

Which to write down, then #

The earlier essay’s advice was a choice between a box, for a basis whose functions are the point, and an ellipse, for a basis that is only a span. It can be simplified.

Set the count, not the tolerance. A count determines the resolution of the randomisation test afterwards, and that is what a protocol actually needs to state. A tolerance is an indirect way of hoping for one.

If the functions are the point, rank by the largest named imbalance. It is the box without the failure, and it needs nothing else.

If several functions matter about equally and a sum of squares reads more naturally, use the sphere. It is within about a tenth of the box, it weights nothing by a noisy estimate, and it has one fewer way to behave strangely on a small sample.

Use the ellipse when the basis really is only a span, and only then, because it pays nearly twice as much in the worst named function for its invariance. That invariance has real uses: it means no one has to argue over which of several equivalent parameterisations to name.

And on sixteen units, count first. All three of these are computable by walking the splits in well under a second, and the count that has to be estimated covers what to do when the splits cannot be walked.

How the numbers were made #

Twenty-four sets of sixteen standard normal units, with all 12,870 equal splits of each walked. Six functions: the covariate, its square and cube in Hermite form, a median split, and two further cuts, each standardised on the sample and each imbalance standardised by its own spread across splits. Each region ranks every split by its own score and keeps a count: the box’s own count, where the box has any, and otherwise a twentieth of the splits. Reported is the share of each function’s own imbalance variance that the kept splits retain, the worst across the named functions, averaged across the sets of units.

On box-versus-others comparisons only those sets with a non-empty box enter, so at four functions the box’s numbers rest on nine sets of the 24, and at five and six on four. These are small samples of sets and the curves there are correspondingly rough: a difference of a hundredth at six functions means nothing. Three things are required of the measurement at three functions. The count-set box must agree with the fixed box to within 0.02. The ellipse must leave at least half as much again, and no cut on its directions may come within a fifth of the box. And the reading that a count-set box has nothing to report where a fixed one is empty is refused: it has a figure there, and the figure above is that figure.

Still open: more than one covariate, and whether the ranking leaks #

More than one covariate. All six functions here come from one covariate, so the correlation between them is what thins the directions. Functions of different covariates are nearly orthogonal, and an ellipse on them is nearly a sphere, so the gap between the regions should shrink as the functions spread across covariates. That is easily tested on the dictionaries of the product field, and it could well turn most of this essay’s advice into a statement about one-covariate bases.

What the set of ranked assignments does to the analysis. A region set by a count has a tolerance read off the sample of units, which a fixed tolerance does not. Nothing in a randomisation test’s reference minds that: it is the admissible set, whichever way it was found, and it is computed from the covariates alone. A normal-theory analysis is a separate matter, since its truncation factor assumes a fixed region, and whether it needs to know about the ranking on a trial this small has not been measured.

Where the two properties were #

This essay takes regions of equal splits on sixteen units, compared at the same admissible count across 24 sets of units. Its claims are that at three functions the box leaves 0.072 of the worst named function’s variance and the ellipse 0.134; that dropping the ellipse’s thin directions reaches 0.097 at its best and worsens quickly past a cut of a quarter; that shrinking to a sphere reaches 0.076; that a tolerance set by a count gives the box’s own set at the box’s count, and a usable region on the 17% to 83% of sets where a fixed tolerance leaves none; and that what the ellipse gets for its looser hold is three to ten points of a coin’s imbalance in the functions nobody named.

What stays out, as a decision: larger trials, where the walk has to be replaced by estimation; more than one covariate, deferred above; and any argument for one specific count, which a test’s resolution should decide. The boundary against what the ellipse spends its assignments on is that it measured a box and an ellipse at one set of units and one tolerance, and this one finds that the tolerance is what caused the difficulty it was reporting.