After the mirror is gone

A randomisation test under a balancing rule draws its reference distribution from the admissible assignments — those whose covariate imbalance the rule allows — and a chain that swaps one treated unit for one control unit is the usual way to draw them. If the admissible set is in pieces the chain cannot cross, the reference distribution is over one piece. The field’s diagnostic for that starts two chains, one from an admissible assignment and one from its complement, and asks whether they disagree. The statistic the p-value is about explained why the complement is the right second start: every rule in this collection splits the units evenly and balances a difference that changes sign when the arms are swapped, so the complement of an admissible assignment is admissible, and when the set is in two mirror pieces the complement is in the other one.

That essay’s claims section named the limit plainly. Under a two-to-one allocation the complement of an assignment is not an assignment of the design at all, the admissible set is not closed under it, and the two-chain construction has nothing to start its second chain from. This essay measures what is left of the diagnostic when the mirror is gone.

Fifteen units at two to one #

The setting is the field’s enumerable one moved to an unequal allocation: fifteen units, five treated and ten control, a covariate drawn fresh for each design, and a rule that admits an assignment when the difference between the arms’ means of each of four basis columns — the covariate, its square, its cube and a median cut — is within a tolerance of its standard error, $\sqrt{1/5 + 1/10}$. There are 3,003 ways to choose five of fifteen, so every admissible assignment can be listed, and the set’s components under the swap of one treated and one control unit can be read off exactly. Two hundred designs at each of three tolerances: 0.6, 0.8 and 1.0 standard errors.

At 0.8, 70 of the 200 admissible sets are in more than one piece; at 0.6, 96 of 192 non-empty sets; at 1.0, 30. A split is not rare at this size, which is the same finding as at fourteen units and even arms.

What is different is the shape. At even arms the split sets came in mirror pairs of equal size, because the complement maps one component onto the other. At two to one nothing forces symmetry, and nothing produces it. The median split set has 97.3% of its admissible assignments in one component; eight of the seventy have more than 99%; the least lopsided has 37.5%. A split at an unequal allocation is one large piece that holds nearly every admissible assignment and one or more crumbs that hold a handful.

How many starts a split needs #

Without a mirror, the natural second start is a second independent one: hunt for an admissible assignment by rejection, as the first was found, and start another chain there. Hunting by rejection draws uniformly from the admissible set, so a start lands in each component with probability equal to its share, and $k$ starts all land in one component with probability $\sum_c p_c^k$. Any diagnostic that compares chains from those starts — on a probe, on which units moved, on anything — can detect the split only when the starts are not all in one piece.

That ceiling is low. At a tolerance of 0.8, two starts are in different components on 12% of split designs on average; four on 21%; sixteen on 46%; 128 on 90%. At 0.6, where splits are less lopsided, two starts manage 28% and sixteen 77%. At 1.0, two manage 4% and 128 still only 64%. For a 95% chance of exposing its split, the median split design needs 21 starts at 0.6, 109 at 0.8 and 439 at 1.0. At even arms, when the split is a unit, the mirror makes two starts enough.

The arithmetic is unforgiving because the crumbs are small. A component holding 2.7% of the set is landed in by one start in thirty-seven, and two independent starts both missing it is the normal case. The diagnostic’s two chains, started without a mirror, almost always explore the same large piece and agree with each other perfectly — about a reference distribution that is missing a few per cent of the assignments the rule allows.

A unit that never moves, with nothing split #

A ceiling that is one unit found that at even arms most splits are a unit — one that sits in one arm across one component and in the other across the other — and a unit that has not moved turned that into a test that needs no second start in the right place: run a chain, and a unit that never changes arm is a split. That inference rested on something easy to miss, and at two to one it fails.

At even arms a connected admissible set can never hold a unit in one arm throughout, because the complement of every admissible assignment is admissible and, the set being connected, in the same component — so the unit is in the other arm somewhere in that component. A unit that never moves is therefore always a sign of a split, or of a chain too short to have moved it.

At two to one there is no complement, and the rule fixes units in perfectly connected sets. At a tolerance of 0.8, 79 of the 130 connected sets have a unit that no admissible assignment moves out of its arm — on the reading the even-armed essays support, a unit extreme enough on some balanced column that the rule can be satisfied only with it in one particular arm. At 0.6 it is 94 of 96 connected sets; at 1.0, 15 of 170. A chain on any of those designs would report a unit that never moved, and the set it is exploring is the whole set. At an unequal allocation a never-moving unit is evidence of nothing, and the reasoning that made it evidence was the mirror symmetry all along.

A hunt for the other arm #

The observation can be rescued by asking the question the mirror used to answer. A chain that never moves unit $i$ is holding it in one arm; the question is whether any admissible assignment has it in the other. If one does, the chain’s component cannot contain it — the chain would have reached it — and the set is split. If none does, the unit is simply fixed by the rule and the chain may be exploring everything. The search is a hunt by rejection with unit $i$ forced into the other arm: draw the remaining units at random into the remaining places, and keep the first assignment the rule admits.

Read off the enumeration, the hunt started from the largest component — where a chain almost always is — finds the split on 75 of the 96 split designs at 0.6, 41 of 70 at 0.8 and 12 of 30 at 1.0. Its cost is small: the median expected number of random assignments before one is admitted with the unit forced over is about 250 at every tolerance, against a pool of a thousand or two thousand. On the connected designs it can never fire, because no admissible assignment has a rule-fixed unit the other way; it has no false alarms by construction. Past enumeration its cost is the inverse of the share of random assignments the rule admits once the unit is forced over, which a few thousand draws estimate, and a hunt that has run for many times that without a hit is reporting that the share is small, not that it is zero — so a failed hunt is evidence of a fixed unit, never proof of one.

What it cannot find is the remainder: 21 splits at 0.6, 29 at 0.8 and 18 at 1.0 in which the largest component fixes no unit that any other admissible assignment has the other way. Those are splits no unit marks from where the chain is. The crumbs differ from the large piece in how several units are arranged together, not in where any one of them sits, and neither the hunt nor random starts at any practical number reliably reaches them.

What the even arms were doing for the diagnostic #

It is worth listing what the complement had been supplying, because each item is a property the field leaned on without naming. A second start in the right place: when the set splits into mirror pieces, the complement of a start in one piece is in the other, so two chains are enough. Pieces of equal size, and mirror images: the complement maps one component onto the other, so a reference distribution drawn from one of them is missing half the assignments in a way half a reference distribution could price exactly — no cost at all to a two-sided p-value, by the same symmetry, and the wrong side of five per cent about once in thirty for a one-sided one. A meaning for a unit that never moves: in a connected set closed under the complement, no unit can be fixed, so a fixed unit is a split.

At two to one each of these is gone, and the second loss is the subtle one. Without the mirror, the two-sided p-value loses the protection the symmetry gave it, and a reference distribution missing a crumb is missing assignments that are not the reflection of anything it kept. How much a missing three per cent moves a p-value is not measured here. It is plausibly small for most statistics, since it is three per cent, and it is the hardest kind of split to detect; the crumbs are the assignments that differ from the bulk in how several units are arranged, and a statistic that happens to depend on that arrangement is the one whose p-value they would move.

The even-armed results that do transport are the enumerative ones. The walk that cannot cross found the set in pieces by listing it, and at fifteen units a list is still possible; nothing about that depends on the arms. And the probe nobody chose found that an outcome can be blind to a split that every covariate probe sees — a finding about what a probe measures, which carries over unchanged, and which here meets a split the probes cannot reach at all because the chains never visit the piece the probe would have to distinguish.

What the diagnostic is at an unequal allocation #

The measurement splits the diagnostic’s job in two. For splits that a unit marks, a chain plus a targeted hunt is a complete instrument at two to one: the chain names the units it never moved, the hunt asks whether each can be moved, and a hit is a proof. That replaces the mirror for about three in five splits at the field’s tolerance, at the cost of a few hundred rejected draws per suspicious unit. For splits that no unit marks, there is no cheap instrument here; random starts in the low hundreds are the only route, and the large piece’s share decides how many.

So the honest summary for a trial randomised two to one under a tight balancing rule is that the field’s diagnostic does not transport. Its second chain needs a start that does not exist, its random replacement almost never lands anywhere informative, and its unit-level reading misfires on connected sets. The targeted hunt recovers most of what can be recovered cheaply and says clearly when it has found something; when it finds nothing, that is not a verdict.

How the numbers were made #

Two hundred designs at each tolerance, the covariate drawn from the field’s design generator at consecutive seeds and the four basis columns standardised and made orthogonal as everywhere else; at 0.6 eight designs admit no assignment at all and are left out, so the counts there are of 192. Every five-of-fifteen assignment is scored on each column’s difference in arm means over its standard error, and admitted when all four are within the tolerance. Components are found by a search over the admissible assignments with an edge for every swap of one treated and one control unit. The start-based ceiling is $1 - \sum_c p_c^k$ from the exact component shares; the hunt’s success and cost are read off the enumeration: for each unit the largest component holds in one arm, whether any admissible assignment has it in the other, and the number of random assignments with it forced over divided by the number of those admitted.

Checked on the measurement: at 0.8 the median split design has more than nine tenths of its set in one component; two starts expose a split on under a fifth of split designs and sixteen on under three fifths; more than two in five connected designs have a unit no admissible assignment moves; and at every tolerance the hunt finds some splits and some carry no unit it can use. The refusal is the reading the even-armed case invites — that a unit no admissible assignment in the chain’s component moves marks a split at two to one as well — and the connected designs break it.

Still open: a second start built rather than drawn #

The hunt forces one unit; the splits no unit marks need a start that differs from the large piece in an arrangement. The construction that would look for one is a second start chosen to be far from the first chain’s whole trajectory — an admissible assignment maximising its distance from every assignment the chain visited, found by a search over the admissible set — on the reasoning that the crumbs are, by definition, unreachable from the large piece and so likely to be far from it. Whether far is the right notion, and whether such a search is cheap enough at sizes past enumeration, is the measurement, and it has not been made.

What is claimed here, and what is not #

This essay takes the two-chain diagnostic at a two-to-one allocation, by enumeration at fifteen units. The claims are that at a tolerance of 0.8, 70 of 200 admissible sets split and the median split set holds 97.3% of its assignments in one component; that two uniform starts expose a split on 12% of split designs on average and 95% needs a median of 21, 109 and 439 starts at tolerances of 0.6, 0.8 and 1.0; that 79 of 130 connected sets at 0.8 have a unit no admissible assignment moves out of its arm, so a never-moving unit is not evidence of a split; and that a hunt forcing such a unit into the other arm finds 75 of 96, 41 of 70 and 12 of 30 splits at the three tolerances, at a median expected cost of about 250 draws, with no false alarms, while 21, 29 and 18 splits carry no unit it can use.

What stays out, named as a decision: sizes past enumeration, where the hunt still runs but its success rate cannot be read off a list; other allocations, such as one to one with unequal block sizes; and the built second start, deferred above.

The boundary against the statistic the p-value is about is that it built the two-chain diagnostic on the complement and named the unequal allocation as where it stops, and this essay measures what stops. Against a unit that has not moved, whose never-moving unit was a proof at even arms, this essay finds the same observation meaningless at two to one without a hunt to back it.