A weighting that does not read its own spreads

A width rule on skewed outcomes found one effect of skew that no stopping rule causes. Run the drifting-ratio design for exactly eighteen blocks, with no stopping at all, and read each weighting’s mean error. On normal outcomes every weighting is unbiased. With a skewness of 4.75, three still are and one is not: the modelled weighting — which fits a line to the log of each block’s estimated variance ratio across the trial and weights each block by the fitted ratio, the weighting a ratio that changes between blocks recommended — is off by 0.0255, six standard errors from zero.

The essay gave the mechanism and the obvious repair. Under skew a block’s within-arm spreads move with its arm means, so a weighting that reads every block’s spreads gives each block a weight that depends on how that block’s difference came out. Fit the variance-ratio line with each block’s own spreads left out, and no block’s weight reads that block. Whether that removes the 0.0255, and what it costs, was left open.

It does not remove it. It turns it round, and makes it larger, and the reason it does is a fact about weighted means that the repair overlooked.

The design, and the three modelled weightings #

The setting is the field’s, built on the width promised for a difference: blocks of alternating composition whose variance ratio drifts along a covariate across the trial, eighteen blocks, two thousand trials on each of six outcome shapes — normal, three log-normal skews in both arms, and the strongest skew in one arm only. Every weighting is read on the same trials, at a fixed length, so nothing about a stopping rule enters.

Three versions of the modelled weighting are compared. All three fit the same line: the log of each block’s estimated ratio, less the known bias a log variance estimate carries at its degrees of freedom, against the block’s position — the correction the bias that lands in the slope showed has to be made block by block, because the allocation alternates and so do the degrees of freedom. That correction is normal theory, and on skewed outcomes it is short, which is the interval’s problem below; it is not the source of the estimate’s lean, which appears and reverses with the correction made in exactly the same way in all three weightings. What differs between the three is only which blocks each line is fitted to, and that turns out to be the whole of the difference in their errors. The modelled weighting fits its line to all eighteen blocks’ corrected log ratios and reads it at each block. The leave-one-out weighting fits, for each block, a line to the other seventeen and reads that line at the block. The cross-fitted weighting splits the blocks into two interleaved halves — nine and nine, each covering the whole range of the drift and both block compositions — fits the line on one half, weights the other half’s blocks by it, does the same the other way round, and averages the two halves’ estimates with equal weight. Beside them are the weighting told the true ratios, a single pooled ratio, and equal weights.

The repair reverses the lean #

On normal outcomes the three modelled weightings agree: 0.0031, 0.0028 and 0.0023, all within a standard error of zero. With skew they separate, and they separate more as the skew grows.

The modelled weighting’s error is +0.0092 at a skewness of 0.95, +0.0166 at 2.26 and +0.0255 at 4.75. The leave-one-out weighting’s is −0.0134, −0.0298 and −0.0455 — the opposite sign, and at every skew larger in size, by nearly a factor of two at the strongest. The cross-fitted weighting’s is −0.0014, −0.0042 and −0.0055, inside a standard error and a quarter of zero throughout and within a standard error of the weighting told the true ratios, which reads 0.0002, −0.0029 and −0.0051.

So the repair the earlier essay proposed is not a repair. At the strongest skew it is ten standard errors from zero, where the problem it was meant to fix was six.

How the two leans grow with the skew #

The two modelled weightings’ errors grow together, and their ratio settles. At a skewness of 0.95 the leave-one-out error is 1.45 times the modelled one in size, at 2.26 it is 1.79 times, and at 4.75 1.78 times. Both are driven by the same quantity — how strongly a block’s spread moves with its mean, which for a log-normal outcome rises steeply from no skew and then levels off, as the earlier essay computed in closed form — and the leave-one-out construction converts that one dependence into a lean of the opposite sign, a little under twice as large.

A ratio that steady is evidence that the two errors are one mechanism read through two weightings, not two effects that happen to coincide. If the leave-one-out bias came from something else — the noisier weights of a fit on seventeen blocks, say — it would not scale with the dependence between spread and mean, and on normal outcomes, where that dependence is absent, it would not vanish. It does vanish: 0.0028 on normal outcomes, beside the modelled weighting’s 0.0031.

Why leaving a block out is not enough #

A weighted mean is $\sum_b w_b d_b \big/ \sum_b w_b$. For it to be unbiased given the weights, the weights have to be independent of the differences — and what matters is not each weight on its own but each block’s share of the total, $w_b / \sum_c w_c$.

Leaving block $b$ out of its own line makes $w_b$ independent of block $b$. It does not make $w_b / \sum_c w_c$ independent of block $b$, because every other block’s line was fitted with block $b$ in it. When block $b$’s spreads come out large, they raise every other block’s fitted ratio, which lowers every other block’s weight — and so raise block $b$’s share. The modelled weighting gave block $b$ a smaller share when its spreads were large, through its own term; the leave-one-out weighting gives it a larger share, through everyone else’s. The link between a block’s spreads and its share is still there, with its sign changed.

The size follows from the same picture. In the full fit, block $b$ is one of eighteen points pulling on the line at its own position, and the pull on its own weight is partly offset by its pull on the others’. Leaving it out of its own fit removes the first and keeps all seventeen of the second, so the remaining effect is the larger part of what was there, and reversed.

Cross-fitting cuts both paths. Within a half, every weight comes from the other half’s line, so neither a block’s own weight nor any other weight in its half reads its spreads; each half’s estimate is a weighted mean whose weights are independent of every difference in it. And the two halves are combined with weights fixed in advance — a half each — so the combination does not read anything either. That is the whole construction: the weights a block’s difference is averaged with must not depend on that block through any route, including the normalisation.

What it costs #

Independence is paid for in precision. At a skewness of 4.75 the weighting told the true ratios has a spread of 0.1890; the modelled weighting 0.1934; the leave-one-out weighting 0.2010; the cross-fitted weighting 0.2099; a single pooled ratio 0.2281; equal weights 0.2355.

The cross-fitted weighting is 8.5% wider than the modelled one. The reason is that each half’s line is fitted on nine blocks rather than eighteen, so its weights are noisier, and noisier weights waste precision. It keeps most of what modelling the drift was for: against a single pooled ratio, which also reads no block’s own spreads in any way that leans, it is still 8.0% tighter. And it is unbiased where the modelled weighting is not, by an amount that is an eighth of the cross-fitted spread — small, but a bias, and one that grows with skew.

On normal outcomes the price is smaller: a spread of 0.1953 against 0.1934, one per cent. A trial that does not know whether its outcome is skewed pays about one per cent if it is not and buys the removal of a bias if it is.

Two weightings that never leaned #

Equal weights and the single pooled ratio are unbiased on every shape here — the pooled ratio reads −0.0067 and equal weights −0.0077 at the strongest skew, each within a standard error and a half of zero — and the reason is the same one cross-fitting relies on. Equal weights read nothing. The pooled ratio reads every block’s spreads, but it reads them into one number shared by every block, so no block’s share of the total moves with its own spreads: a block with large spreads raises the pooled ratio for everyone alike and leaves every block’s share where it was. A weighting is safe from this lean whenever a block’s spreads cannot change that block’s share, and both of these satisfy that by construction.

What they give up is the drift. The variance ratio in this design changes across the trial, and a single number cannot follow it, which is why the pooled ratio’s spread is 0.2281 where a weighting that follows the drift can reach 0.1890. The essay on the drifting ratio chose the modelled weighting to recover that difference. Cross-fitting recovers part of it — its 0.2099 sits a little under half of the way from the pooled ratio to the true ratios — without reintroducing the lean, and the modelled weighting’s extra half is exactly the part that leaned.

Why halves #

Two halves is a choice, and it is worth saying what it trades. More folds would fit each block’s weights on more of the trial — with three folds, on twelve blocks rather than nine — and so give less noisy weights and a tighter estimate. But every fold’s estimate has to be combined with weights fixed in advance, and a fold of six blocks carries a noisier estimate and a noisier variance for the interval. Two halves of nine was chosen to keep each half’s interval on eight degrees of freedom and each half’s line on nine points. Whether three or six folds does better on these designs is a measurement this essay has not made; the argument above says only that any number of folds removes the lean, provided the folds are fixed in advance and combined with fixed weights.

The interval, at a fixed length #

The cross-fitted interval is built the obvious way: each half’s weighted variance gives that half’s standard error, the two are combined for an average of two independent estimates, and the critical value is Student’s on the sixteen degrees of freedom the two halves leave. At a skewness of 4.75 it covers 93.85%, against 94.40% for the modelled weighting, 93.25% for the leave-one-out one, and 95.60% for the weighting told the true ratios.

So removing the bias does not by itself bring coverage to nominal. The cross-fitted interval’s half-width, 0.3957, is smaller than 1.96 times its estimate’s spread — the half-estimates’ own variance estimates are short on skewed outcomes, which is the separate defect the earlier essay traced to a normal-theory correction in the log-variance. On normal outcomes the cross-fitted interval covers 94.90%, which says the construction itself is sound and the shortfall at 4.75 is the skew’s.

With the skew in one arm #

The earlier essay found that a skew confined to one arm leans the stopping rule in a direction set by the arm. The weighting’s own bias depends on the arm too, but the three modelled weightings keep their pattern. With the skew in arm A only, the modelled weighting reads +0.0127, the leave-one-out −0.0262, the cross-fitted −0.0020; in arm B only, +0.0170, −0.0188 and −0.0030. In every placement the leave-one-out weighting overshoots to the other side and the cross-fitted one sits within a standard error of zero.

That is the check that the mechanism is the one described rather than something particular to two skewed arms. A skew in either arm alone ties the block’s spread in that arm to the block’s mean in that arm, which is enough to make any weight that reads the spreads lean, and enough to make a leave-one-out fit lean the other way through the normalisation.

What a practitioner should take #

Do not leave a block out of its own weight and call the weights independent. The construction is natural, it is what the earlier essay proposed, and it makes the bias worse. The share a block receives depends on every weight, and a leave-one-out fit leaves the block in all the others.

Cross-fit. Fit the variance-ratio curve on half the blocks, weight the other half by it, swap, and average with weights fixed in advance. On these designs it removes the skew bias entirely, costs about one per cent of spread on normal outcomes and about nine per cent on strongly skewed ones, and remains tighter than a single pooled ratio.

And keep the two defects apart, as the interval after a stop it chose kept a stopping rule’s two failures apart. The bias in the estimate and the shortfall in the interval are separate effects of skew with separate repairs. Cross-fitting repairs the first and not the second; the second belongs to the variance estimate, and the pooled ratio — which averages sums of squares before taking any logarithm — is the existing rule that avoids it, at a larger price in spread.

Still open: cross-fitting inside the stopping rule #

Everything here is at a fixed eighteen blocks, deliberately, so that the bias is the weighting’s and not a stopping rule’s. The trial the field is about stops when its predicted width reaches a target, and that prediction reads the weights too. A stopping rule that predicts its width from cross-fitted weights has to split the blocks enrolled so far into halves that change as blocks arrive, and whether the halves can be assigned in advance — every fourth pair of blocks to the same half, say — without breaking the blinded rule’s independence is a design question this essay has not answered. The leave-one-out weighting’s sequential bias is already measurable, and at a skewness of 4.75 in arm B it reads −0.0055 against the modelled weighting’s +0.0255 with the stopping rule running — so the stopping rule interacts with the reversal, and a sequential version of the cross-fit is the measurement this points to.

The claim, and where it stops #

This essay takes the modelled weighting’s bias on skewed outcomes and two repairs. The claims are that at a fixed eighteen blocks the modelled weighting’s mean error is +0.0092, +0.0166 and +0.0255 at skewnesses of 0.95, 2.26 and 4.75; that fitting each block’s weight with that block left out reverses it to −0.0134, −0.0298 and −0.0455, because a block’s share of the total weight still depends on its spreads through every other block’s fit; that cross-fitting on two interleaved halves, combined with fixed weights, gives −0.0014, −0.0042 and −0.0055, within a standard error of the weighting told the true ratios; that it costs a spread 8.5% wider than the modelled weighting at the strongest skew and remains 8.0% tighter than a pooled ratio; and that its interval covers 93.85% there and 94.90% on normal outcomes.

What stays out, named as a decision: the sequential rule, deferred above; other splits than two interleaved halves; and other drifts in the variance ratio. The boundary against the earlier essay is that it found the bias and named a repair, and this essay measures that repair, finds it reversed, and replaces it.