The share a sample removes

A zero that was an assumption took the exact zero this field had built on — a rule holding every main effect of two covariates removes none of their interaction — and found it was a fact about independence. With the covariates correlated, the eight main effects, four Hermite functions of each covariate, remove $4\rho^2/(1 + \rho^2)^2$ of the interaction: nothing at zero correlation, 30.3% at 0.3, 64.0% at a half. The fourth moment that was missing derived it and three hundred thousand draws confirmed it.

Both routes are statements about the population. What a balancing rule removes of a function it does not hold is the share of that function’s variation the held functions explain, and the closed form is that share computed from the joint distribution of the covariates. A trial does not hold the joint distribution. It holds a hundred units, and what exact balance removes in that trial is the share the eight functions explain among those hundred units — an in-sample $R^2$, which has the closed form as its limit and is not equal to it in any trial.

The essay listed what it left out, and the nearest item was this: what the share does in a real trial, which needs the design rather than the geometry. This essay takes the geometry to a trial’s size and leaves the design aside, by asking what exact balance would remove. The answer moves furthest exactly where the closed form is most confident.

The share inside a trial #

A rule that balanced the eight main effects exactly would leave, of the interaction’s imbalance between the arms, only the part of the interaction the eight cannot reproduce among the trial’s own units. So the share it removes is the in-sample $R^2$ of the interaction — the product of the two centred covariates — on the eight functions, each centred in the sample. Nothing about an outcome or an assignment has to be simulated; the share is a property of the hundred units.

It is computed on four thousand samples at each of five trial sizes, 25, 50, 100, 200 and 400 units, and three correlations, 0, 0.3 and 0.5.

At independence, where the closed form is exactly zero, a trial of 25 units removes 62.1% of the interaction on average; 50 units, 40.3%; 100 units, 24.5%; 200 units, 14.9%; 400 units, 8.4%. At a correlation of 0.3, where the closed form is 30.3%, the five sizes give 69.9%, 54.2%, 44.4%, 38.4% and 34.8%. At a half, where it is 64.0%, they give 80.7%, 73.1%, 69.2%, 66.8% and 65.5%.

Every in-sample share is above its closed form, and the excess is largest where the closed form is smallest. At a hundred units the excess is 24.5 points at independence, 14.1 at 0.3 and 5.2 at a half.

Why a sample’s share is larger #

An $R^2$ computed in a sample is biased upwards: a regression on $k$ regressors explains about $k/(n - 1)$ of a response that is independent of them, because in a sample every function is correlated with every other by chance. With eight regressors and a hundred units that is 8.1%. The measured share at independence is 24.5%, three times as much, and the multiple grows with the trial: 62.1% against 33.3% at 25 units, under twice, and 8.4% against 2.0% at four hundred, over four times.

The formula assumes something the interaction does not have. At independence the product $xy$ is uncorrelated with every function of $x$ alone and of $y$ alone, but it is not independent of them: its spread given $x$ is proportional to $|x|$, so it is small where the covariates are near zero and large where they are far out — exactly where the cube and fourth power of each covariate are large too. A response whose variance grows with the regressors it is regressed on has a larger chance $R^2$ than one whose variance does not, because the sample’s chance correlations are computed mostly from the units where both are big.

That account can be tested by setting the far-out units aside. Dropping every unit beyond two standard deviations on either covariate — about one in eleven — and recomputing the share on the rest, the share at independence falls from 24.5% to 19.2% at a hundred units, from 14.9% to 9.8% at two hundred and from 8.4% to 5.2% at four hundred. So the units in the tails carry a fifth to two fifths of the excess, and the rest survives in the trimmed sample, where the product’s spread still grows with the covariates over the whole range that remains. The trimmed shares are still more than twice the formula’s $k/(n - 1)$.

The excess also falls more slowly than the formula’s: by about two fifths with each doubling of the trial — 62.1, 40.3, 24.5, 14.9, 8.4 — where a bias of eight over $n$ would halve each time. That is the rate of statistics that depend on fourth and higher moments of the covariates, which is what the interaction’s dependence on its own regressors makes this one.

How much it varies from trial to trial #

The average is a statement about many trials. A protocol is one trial.

At independence a trial of a hundred removes between 10.1% and 41.6% of the interaction, tenth to ninetieth percentile; at four hundred units, between 3.1% and 14.8%. At a correlation of a half a trial of a hundred removes between 52.9% and 83.3%, against the closed form’s 64.0%; at four hundred, between 56.2% and 74.0%.

The spread is the useful half of the result. A designer told the rule removes 64% of the interaction is being told the centre of a distribution thirty points wide at a hundred units, and nothing in the closed form says so. And a designer told the rule removes nothing at independence is being told the lower limit of a distribution that, at a hundred units, puts nine trials in ten above a tenth.

Whether it counts #

It is fair to ask whether a share removed by chance is a share removed. It is, for the trial in hand. Exact balance on the eight functions forces the arms to agree on everything those functions can reproduce among these units, and if the interaction happens to be among those things in this sample, the arms agree on it too: the interaction’s imbalance in this trial is smaller, and an unadjusted estimate of the treatment effect in this trial is less disturbed by it. Nothing about it requires the population share to be non-zero.

The same spread at a correlation of a half is narrower in proportion and still wide in points: 52.9% to 83.3% at a hundred units, 56.2% to 74.0% at four hundred. A trial at the lower end of that range has a third less protection against the interaction than the closed form promises; one at the upper end has a quarter more. The protection a design calculation quotes is the centre of that range, and the range is computable from the trial’s own covariates before anyone is assigned.

What it is not is a guarantee. The share is a function of the sample, it can be read off before randomising, and it varies from a tenth to two fifths between samples that the population closed form cannot tell apart. A designer who wants to know what balancing the main effects buys against an interaction in this trial can compute it exactly from the covariates already recorded — it is a regression on numbers the trial has before anyone is assigned. A designer who quotes the closed form instead is quoting a number about trials of infinite size.

Where the guarantee is exactly zero argued that a zero is worth more than a small number because it is a statement that cannot fail. In a finite trial this zero is not a statement about the trial. The in-sample share is never zero, and its distribution is set by the trial’s size and the covariates’ tails, which the zero does not mention.

What the closed form is for #

None of this makes the closed form wrong. It is the limit of the in-sample share, and the measurement agrees with it at every correlation as the trial grows: at four hundred units the averages are within 8.4, 4.5 and 1.5 points of it. What it describes is the part of the share a trial can rely on in advance, before its covariates are recorded — the share of an interaction that the population structure puts within the main effects’ reach, independent of which units happen to be drawn.

The in-sample share adds a second part, which depends on the units drawn and is largest where the trial is smallest. A designer choosing a dictionary before the trial exists is choosing against the first part, and the closed form is the right number for that choice. A designer analysing a trial that exists is facing the sum, and the sum can be computed.

So the two numbers answer different questions, and the field’s tables have been answering the first. The dictionary field’s own comparisons of which functions to hold are comparisons of population shares, and they stand as such; what changes is that in a trial of a hundred units every dictionary is worth something against every shape, by chance, and the ranking between two dictionaries whose population shares differ by a few points is a ranking that one trial’s covariates can reverse.

What a design calculation should say instead #

The practical form of all this is short. A protocol that says its allocation rule balances the main effects of its covariates, and quotes the share of an interaction that buys from a closed form, is quoting a number that is true of no trial. Three better statements are available, and each can be made before randomisation.

The trial’s own share. The in-sample share of the interaction explained by the functions the rule holds is a regression on the recorded covariates. It is the share this trial’s balance would remove if the balance were exact, and it is the number the protocol’s claim is about.

Its uncertainty, if the trial is being planned rather than run. Before the covariates are recorded the share is a random variable, and its distribution at the planned size — the band in the figures above — is what a designer should weigh against the cost of holding more functions. At a hundred units it is thirty points wide at a half and runs from a tenth to two fifths at independence.

And the population share, as a limit. The closed form is still the right description of what the dictionary contributes independently of chance, which is the part a designer can compare across dictionaries before any trial exists. A dictionary that is neither and the essays after it compare dictionaries on exactly that footing, and the comparison stands.

The second statement is the one most likely to change a decision. A designer choosing between two dictionaries whose population shares differ by five points should know that the trial’s realised shares will differ from them by ten to thirty, in either direction, and that the realised ordering of the two dictionaries is a property of the units drawn.

The share and the leverage of a unit #

The quantity doing the work in all of this has a familiar name in a single regression. A unit whose covariates are far out has high leverage on any fit of a function of them, and the line that one point drew is the field’s account of what one such unit does to a slope. Here every function in the dictionary is fitted at once, the interaction is the response, and the units with high leverage on the dictionary are the units where the interaction’s spread is greatest — so they move the fit most and the fit moves with the interaction most. The in-sample share is, in this sense, a leverage-weighted correlation, and a trial’s share is high when its few high-leverage units happen to lie where the interaction is large.

Two routes, and where they meet #

The closed form and the in-sample share share no arithmetic: one is three inner products of Hermite functions under a bivariate normal, the other a regression on a sample. Their agreement in the limit is the check that the in-sample computation is the quantity the closed form describes. At four hundred units the in-sample average at a correlation of a half is 65.5% against the closed form’s 64.0%, and the excess halves roughly with every doubling; at independence it is 8.4% against zero, on the same path. The two meet at infinity and nowhere a trial lives.

How many functions a trial can hold found the other side of the same coin for a covariate-adaptive rule: the functions a rule holds cost the shapes inside its span a residual that falls like one over the trial’s size. Here the shapes outside the span gain a share that falls more slowly. Both say that a trial’s balance is a property of its units, and that the population calculation is its limit rather than its value.

Four thousand samples and eight Hermite main effects #

Four thousand samples at each size and correlation. Each sample draws the covariates from a bivariate normal at the stated correlation, forms the eight main effects as the first four Hermite polynomials of each covariate, centres each in the sample, forms the interaction as the product of the two centred covariates, centres it, and computes the share of its sum of squares explained by least squares on the eight. The closed form is the field’s own. Nothing about treatment or outcomes is simulated, because the share exact balance removes is a function of the covariates alone.

A share above its closed form at every size, and not zero at independence #

The claims are that the share of an interaction that exact balance on eight main effects removes within a trial is above its population closed form at every trial size and correlation measured; that at independence it is 62.1%, 24.5% and 8.4% on average at 25, 100 and 400 units where the closed form is zero; that it varies from trial to trial by about thirty points at a hundred units; and that its excess over the closed form is three times the usual bias of a sample $R^2$ at a hundred units and a larger multiple at four hundred, a fifth to two fifths of it carried by the units beyond two standard deviations, because the interaction’s spread grows with the very functions it is regressed on.

Not claimed: anything about a rule that balances approximately rather than exactly, which removes less of everything; anything about covariates that are not normal, whose tails would move the excess; and anything about the treatment effect’s variance, which needs an outcome model.

Still open: the share a real rule removes #

Exact balance on eight functions is a limit no allocation rule reaches in a trial of a hundred: the functions a rule holds compete for the same units, and the fourth powers are the hardest to hold because their values are set by the units far out, which are also where the interaction’s spread is largest. A covariate-adaptive rule that holds the eight approximately may well remove less of the interaction than the in-sample share says, precisely because the units it cannot balance well are the units that make the share large. Running the field’s sequential rule on the eight main effects and measuring the interaction’s realised imbalance against a coin’s, at the five sizes here, would say how much of the chance share a real rule collects — and whether the far-out units that do much of the work are the ones a rule gives up on first.