Field

The diagnostic after the trial

The test for whether a balanced-assignment walk can reach the whole admissible set is run on a covariate function, before any outcome exists. Run instead on the difference in arm means it is the same test — under the sharp null the outcome is a fixed column — and it is about the statistic the p-value is actually built from. Two findings: an outcome is a probe nobody chose, and on a set that is genuinely split three in ten of them see nothing at all; and the published two-sided p-value is exactly right on half a reference distribution, while a one-sided one is not.

Two diagnostics, one answer, two different moments. The two-chain statistic on a covariate probe and on the trial's own difference in arm means, at seven tolerances of a fourteen-unit rule, against the enumerated truth. Both are quiet wherever the set is one set and both fire wherever it is not, at every tolerance — which is what says the outcome probe is the same test rather than a resemblance of it. The difference between them is not accuracy and it is not power. It is when: the covariate probe can be run before a single outcome exists, when a practitioner can still loosen the rule or change the sampler, and it can be run again on a different function if it comes back quiet. The outcome probe runs after the trial, on the one column the trial produced, and what it can do with a positive verdict is repair the p-value rather than the design.

The statistic the p-value is about

The test for whether a balanced-assignment walk reaches its whole set is run on a covariate function chosen before the trial. Run on the difference in arm means it is the same test, and it is about the number the trial publishes.

The probe a trial has is the probe a trial got. What the two-chain test says when it is run on the trial's own difference in arm means, over 24 outcomes on one fourteen-unit set. The set is in 2 mirror components — that is enumerated, not inferred — so every quiet reading is a miss. 29% of them are quiet. The reason is in the enumerated set rather than in the run: how far the two components are apart on a given probe ranges from 0.001 to 4.938 of a within-component spread across these outcomes, a factor of several thousand. Both covariate probes — chosen before any outcome existed, and replaceable if they had been quiet — report the split. An outcome cannot be chosen and cannot be replaced.

A probe nobody chose

On a set that is definitively in two pieces, seven of twenty-four outcomes report nothing at all. Every covariate probe reports it. What separates them is not accuracy — it is that one of them can be chosen and the other is what happened.

The one thing a trial always reports is the one thing that survives. How wrong three p-values are when they are computed over the half of the admissible set a single walk can reach, rather than over all of it, at a fourteen-unit trial where the whole set can be enumerated. The two-sided p-value on the difference in arm means — the number a trial publishes — is wrong by exactly nothing, at every row, to machine precision. That is not luck: the two components are complement pairs and the difference in arm means is exactly negated by the complement, so the distribution of its absolute value is the same on both. A one-sided p-value on the same statistic is out by as much as 0.112, and the largest response observed in the treated arm — a safety reading rather than an effect, and the one statistic here that is not odd under the complement — by as much as 0.172. The defect survived because the commonest thing anybody computes is the one quantity it cannot touch.

Half a reference distribution

A walk that reaches half its admissible set reports the two-sided p-value exactly right, to the last digit, for ever. A one-sided one it puts on the wrong side of five per cent about once in thirty.

Where the set stops being one set. How many of the 3,432 equal splits of fourteen units a balancing rule admits, as the tolerance tightens, with the number of components single swaps leave it in. The set falls from 886 to 84 assignments, and somewhere in that fall it stops being connected: at 0.8 it is in 2 pieces and every assignment's complement is in the other one. Nothing about the rule changes at that point and nothing a chain reports changes either, which is the whole difficulty — the acceptance rate, the stationary distribution and the detailed balance are all in order on both sides of it.

Before the trial and after

The same diagnostic run at two moments answers two different questions. Before, a positive verdict changes the design. After, it changes which number gets reported — and only for the numbers the defect can reach.

How many starts it takes to land in two pieces. Averaged over the split designs at each of three tolerances, the chance that k starting assignments drawn uniformly from the admissible set — what hunting for a start by rejection gives — do not all fall in one component, which is the most any diagnostic built on comparing chains from those starts can detect. At 0.6: 28% with 2, 47% with 4, 63% with 8, 77% with 16, 88% with 32, 95% with 64, 99% with 128. At 0.8: 12% with 2, 21% with 4, 32% with 8, 46% with 16, 62% with 32, 77% with 64, 90% with 128. At 1.0: 4% with 2, 8% with 4, 13% with 8, 21% with 16, 33% with 32, 47% with 64, 64% with 128. The median split design needs 21, 109 and 439 starts at the three tolerances to expose its split with probability 95%. With an even allocation the mirror makes two starts enough whenever the split is a unit.

After the mirror is gone

The two-chain diagnostic starts its second chain at the complement of the first's start, which exists only when the arms are equal. At five treated against ten, enumerated on two hundred designs, a split admissible set is one big piece and some crumbs — the median puts 97.3% of its assignments in one component — so two random starts see the split on 12% of split designs and 95% needs a median of 109 starts. Worse, the rule fixes units in a connected set all the time, so a unit that never moves stops being evidence. A hunt that forces such a unit into the other arm finds 41 of 70 splits; 29 carry no unit that marks them.

All essays