A ceiling that is one unit

A balancing rule that only admits assignments whose covariate imbalance is small can cut its own admissible set into pieces that a swap-one-pair chain cannot cross between. The walk that cannot cross found that on designs of fourteen units, enumerated, and the two-chain diagnostic built to detect it starts one chain from an assignment and one from its mirror and asks whether they disagree on a probe — a column of the design the chains are summarised by. Everything since has been about which probe.

How far to project ended on an awkward discovery about the benchmark. The probes were all measured against the separating direction: the difference between the two components’ mean assignments, standardised. It is the direction along which the two components are furthest apart on average, and it carries a median separation of 10.565 — the difference between the components’ mean probe values over the spread inside a component. But projecting it a quarter of the way off the rule’s span raised that to 11.500. A benchmark that a crude modification beats is not a ceiling. The separating direction was chosen for the numerator of the ratio and ignores the denominator.

The direction that maximises the ratio itself is a textbook object, Fisher’s discriminant, and the next measurement was to compute it and see how far above 11.5 the true ceiling sat. It does not sit anywhere. On most split designs it is infinite, and the reason is not a direction at all but a unit.

The direction built for the ratio #

The separation divides the difference between two means by a spread, so the direction that maximises it is the one that makes the difference large relative to how much the probe varies inside each component. With $\Delta\mu$ the difference of the two components’ mean assignment vectors and $S$ their pooled within-component covariance, that is $w \propto S^{-1}\Delta\mu$. Two details matter. The separation averages the two components’ standard deviations rather than pooling their variances, so Fisher’s direction is exactly the maximiser only when the two spreads agree; the ratio’s own stationarity condition is solved separately, by iteration from Fisher’s start, and both are reported. And every assignment is an equal split of the units, so the direction that adds the same amount to every unit has no spread at all and carries no information; the inverse is taken on the subspace orthogonal to it.

The setting is the one the field has used since the diagnostic was first validated: fourteen units, a covariate drawn fresh for each design, a rule that balances its first three powers and a median cut within a tolerance of 0.8, and two hundred designs of which a hundred have an admissible set that enumeration finds in more than one component. Every number below is computed from the enumerated set, so nothing in it is a chain’s estimate.

The first thing the computation reported was not a large ratio. It was a direction along which the pooled within-component covariance is zero while the mean difference is not. On such a direction every assignment in the first component reads one value and every assignment in the second reads another, the spread is zero, and the ratio has no finite maximum. That happened on 88 of the 100 split designs.

A unit that sits still #

The direction is usually as simple as a direction can be: the indicator of one unit.

The design in the figure has 192 admissible assignments in two mirror components of 96. Most of its units move freely inside either component: the unit at the bottom of the covariate’s range is treated on a quarter of the first component’s assignments and three quarters of the second’s, which is the mirror at work, and the units near the middle sit close to half in both. One unit does not move at all. The unit at 0.28 on the covariate is in the control arm on every assignment of the first component and in the treated arm on every assignment of the second. Asking which arm that one unit is in tells the components apart without error.

It is easy to see why such a unit can exist. The rule balances four columns to within a tolerance on fourteen units, which is a tight constraint, and a swap-one-pair chain moves by exchanging a treated unit with a control. A unit whose every exchange partner would break one of the four balances cannot be exchanged, from any assignment in its component; the component is then a set of assignments in which that unit is fixed, and its mirror — every assignment complemented — is a set in which it is fixed the other way. The split is the unit. The chain cannot cross because crossing would require that unit to change arm, and no single swap that moves it stays admissible.

The design in the figure is typical rather than chosen. Of the hundred split designs, 83 freeze at least one unit on opposite sides of the two components: 23 freeze exactly one, 18 two, 14 three, and 28 four or more, out to three designs that freeze all fourteen — sets whose components are each a single assignment and its mirror. Every one of the 83 is a design whose separation has no finite ceiling, which is two routes to one fact: the frozen units are counted from the masks of the enumerated assignments, and the unbounded ceiling is read off the eigenvalues of the components’ covariance, and they agree on every design. Of the 17 designs that freeze no unit, five still have an unbounded ceiling through a fixed combination of units, and twelve have a finite one.

So the question the previous essay asked — how far above 11.5 the ceiling sits — has an answer that makes the question the wrong one. On most split designs at this size there is a probe that separates the components perfectly, and it is a column of zeros with a one in it.

What the projection gave up #

The field’s first essay made a recommendation that every later one built on: project the probe off the span of the columns the rule balances, because the part of a probe inside that span is a quantity the rule has already forced near zero. On the probes it was tested on — dictionary functions of the covariate — that was right, and the dial essay found it right all the way for the fourth power.

A single unit’s indicator is not a function of the covariate in that sense, and it is not orthogonal to the rule’s span: any unit with a nonzero covariate value contributes to every balanced column. Projecting it off the span subtracts a combination of those columns, and the result is no longer constant inside a component.

Restricting the directions to those orthogonal to the rule’s span keeps an exact separator on 37 of the hundred designs and loses it on 51. Of the 83 designs with a frozen unit, 36 still have some exact separator off the span and 47 do not. The projection that turned a weak dictionary probe into a strong one is, on most split designs, the step that throws away the perfect probe. The restricted ceiling is never unbounded where the unrestricted one is finite — removing directions can only lower a maximum — so the arrow runs one way, and on half the designs it runs from infinite to finite.

That does not make the earlier recommendation wrong about the probes it made it for. It makes it a recommendation about linear probes built from the covariate, and it says the best probe is not one of those.

Where the ceiling is finite #

The twelve designs with no exact separator are where the original question can still be asked, and they answer it.

On those twelve, the ceiling is between 1.02 and 2.28 times the separating direction, with a median of 1.86. The best direction orthogonal to the rule’s span is between 0.83 and 1.30 times it. So most of the distance between the benchmark and the ceiling lies inside the span, where no fully projected probe can reach — which is what the dial essay’s partial projection of the separating direction had hinted at, and is here measured design by design. Fisher’s direction is the maximum on eight of the twelve. On the other four it falls short by between 7% and 12%, and each of those four has components of very different sizes — 49 assignments against 5, 108 against 7 — so the two spreads differ and averaging standard deviations is not the same as pooling variances.

The finite designs are also unrepresentative in a way worth stating. Eight of the twelve have one component far larger than the other, seven of them with the smaller component a single assignment or a handful. Of the 67 split designs whose two components are mirror halves of equal size, 62 freeze a unit. A finite ceiling is mostly a property of lopsided splits.

A test that reads which units moved #

A frozen unit is the kind of separator a chain can find without the enumeration. A probe is a summary a chain is read through, and choosing it well needed the components; a unit that never changes arm is visible in the raw run. So a second diagnostic suggests itself: run the same two chains from an assignment and its mirror, and for each unit record the share of draws that treat it. The verdict is split when some unit never moved in either chain and sat in opposite arms in the two. Nothing in the verdict is estimated — no standard error, no autocorrelation time — which is the property the probe-based test lacked at two hundred units, where it needed its own relaxation time to read its own statistic.

What it cannot do is tell a unit frozen by the set from a unit the run has not yet moved. Both sit still, both sit in opposite arms when the chains start from mirror images, and a replicate chain from the same start sits still in the same arm either way. Its false alarms are therefore a property of the run’s length alone, and they are counted on the 92 designs whose admissible set enumeration finds connected.

At fifty draws a chain the test fires on almost everything — 93% of split designs and 87.0% of connected ones — because in fifty swaps most units have not been touched. By 200 draws the false alarms are down to 27.2%, by 800 to 3.3%, and at 3,200 there are none on the 92 connected designs. Detection on the split designs settles at 81% at 800 draws and 79% at 3,200. That is below the 83% of designs that freeze a unit between the two components the enumeration lists first; a set split more than two ways can start its pair of chains in another pair of components, where the frozen units are not the same ones.

The probe-based test with the projected fourth power — the best feasible probe the field found — fires at 800 draws on 85% of the same split designs and on 7.6% of the connected ones. So at equal length the unit test finds slightly fewer splits and raises fewer than half as many false alarms. Neither ordering is the interesting thing. The interesting thing is which designs each finds.

Both fire on 73 of the hundred. The unit test alone finds eight more, the probe test alone twelve more, and seven are found by neither. Read together — split if either says so — the pair finds 93 of the hundred against 85 for the probe test by itself. And on the 92 connected designs every false alarm the unit test raises is on a design where the probe test is already wrong, so the pair’s false alarm rate is the probe test’s: seven designs. The second test adds eight detections and no errors.

The two are complementary for a reason that can be stated. The probe test sees a split through a column of the covariate, and is strong where the components differ in some smooth function of it. The unit test sees a split through one unit, and is strong exactly where the split is a unit — where the probe test’s linear summary must average a perfect separator in with thirteen units that move. The seven designs neither finds freeze no unit; all seven are lopsided, a large component beside one of between one and eleven assignments, every one split three or more ways, and two of them are among the five unbounded only through a combination of units, which no single unit and no smooth column will reveal.

Why this was invisible #

Every essay in this line had the enumeration in hand and read it through a probe. The separating direction is itself a summary — a difference of two means — and a mean over 96 assignments of a unit that is treated on every one of them is exactly 1, a number the direction then standardised against thirteen others and reported as a large loading. The information was in every table. What it did not look like was a probe, because the probes were all built as functions of a covariate and a unit is not one.

That is the absence-shaped defect the field keeps finding in a new place. A diagnostic designed around a summary statistic reports how well the statistic separates; it cannot report that a better summary would have separated perfectly. Only computing the ceiling — and finding it not to exist — could.

Still open: whether units freeze at two hundred #

The measurements here are at fourteen units, where the set can be enumerated and a tolerance of 0.8 on four columns is tight. The size sweep found the defect shrinking as the trial grows, and at two hundred units the probe-based test could not settle its own verdict. The unit test needs no estimated standard error, so it is the instrument that might: run it at two hundred units and a long chain, and either some unit never moves — which is a finding about the set, not about the run, once the run is long enough that connected designs stop producing false alarms — or none does. The prediction from the mechanism is that frozen units become rarer as every unit acquires more admissible exchange partners, but that a tight enough tolerance on enough columns will freeze some at any size. Where the boundary sits is not measured.

A second open question is the five designs unbounded through a combination of units. Their separators are fixed sums of several units’ arms — two units always in the same arm, say, on one side and in opposite arms on the other — and a test that read pairs of units rather than units would see them. Whether that is worth its cost, at ninety-one pairs a design of fourteen, is a measurement this design can make.

What is claimed here, and what stops it #

This essay takes the ceiling on a probe’s separation, by Fisher’s direction and the ratio’s own maximiser, over a hundred enumerated split designs of fourteen units. The claims are that the ceiling is unbounded on 88 of them, because the mean difference has a part along a direction with no spread inside either component; that 83 designs freeze at least one unit on opposite sides of the two components, and every one of those is among the 88; that restricting directions to those orthogonal to the rule’s span keeps an exact separator on 37 designs and loses it on 51; that on the twelve designs with a finite ceiling it is 1.02 to 2.28 times the separating direction, median 1.86, while the best direction off the span is 0.83 to 1.30 times it, and Fisher’s direction is the maximum on eight; and that a test reading which units moved in two chains from mirror starts fires on 81% of split designs and 3.3% of connected ones at 800 draws, 79% and none at 3,200, and together with the probe-based test finds 93 of the hundred splits with no more false alarms than the probe test alone.

What stays out, named as a decision: other sizes, rules and tolerances, deferred above; separators that are combinations of units, also deferred; and an outcome, since every quantity here is a property of the design and the rule — as the probe nobody chose established is the right thing to diagnose before a trial, and as remains true of a unit.

The boundary against how far to project is that it treated the projection as a dial on linear probes of the covariate and this essay asks what the best probe of any kind can reach. Against what a chosen probe finds, which measured how often the probe test’s verdict is right, the unit test here is a second instrument on the same designs, and the comparison between them is design by design rather than across two samples.