Field

When the stratified answer and the pooled one disagree

Simpson's reversal is normally taught once, on one table, as a warning about confounding. None of the reversals here is confounded. A trial randomised by a coin, on a population where the treatment helps in both groups, reports it losing overall on 3.40% of trials at eighty units — and stratifying the randomisation takes that to zero at every size. An odds ratio of exactly 2.5 in all five strata pools to 1.789, because an odds ratio is not a weighted average of odds ratios, while the risk difference on the same table is exactly its stratum value. And where the grouping variable is something the treatment caused, both answers are right and answer different questions.

How often a randomised trial reports the reversal, advantage 6 points. Simple randomisation against randomisation stratified by group, 4,000 trials at each size. The simple design reverses on 3.40% of trials at its worst size and 0.50% at 1280 units; the stratified design reverses on none of them, at any size.

The reversal a coin cannot prevent

Randomisation removes Simpson's reversal in expectation, which is not the same as removing it. A correctly randomised trial of eighty units, on a population where the treatment helps in both groups, reports it losing overall on 3.40% of trials — and stratifying the randomisation takes that to zero at every size.

One odds ratio in every stratum, and a different one marginally. Five strata with baseline risks from 5% to 85%, a treatment allocated by a coin in each, and a conditional odds ratio of exactly 2.5 throughout. The marginal odds ratio is 1.789. Nothing is confounded; the odds ratio is simply not a weighted average of odds ratios.

The change that is not confounding

Five strata, a treatment allocated by a coin in every one, and an odds ratio of exactly 2.5 in all five. The odds ratio computed on the pooled table is 1.789. Nothing is confounded — an odds ratio is not a weighted average of odds ratios, and the risk difference, on the same table, is exactly its own stratum value.

What stratifying on a mediator estimates, and what it does not. The direct effect stays at -0.109 throughout, because it is defined with the mediator held fixed. The total effect runs from -0.218 to 0.027 and crosses zero at 1.5. On 15% of the sweep the two have opposite signs, and both are correct answers.

Conditioning on what the treatment caused

When the grouping variable lies on the path from treatment to outcome, the stratified answer is the direct effect and the aggregate is the total effect. Both are correct. Over 15% of a sweep of the indirect path they have opposite signs, and no arithmetic on the table says which question was being asked.

The ratio of the arms' hazards over time, when treatment halves every patient's hazard. Every patient's hazard ratio is 0.5. With two risk groups — half at 0.1 a year, half at 1 — the ratio of the arms' hazards rises from 0.5 to 0.929 at 3.5 years and falls back towards 0.5. With a gamma frailty of variance 1 it rises steadily, to 0.857 at ten years.

Every patient at one half

A randomised trial in which the treatment halves every patient's hazard, and patients differ only in their baseline risk. The unadjusted Cox hazard ratio is 0.524 after six months of follow-up, 0.635 after three years and 0.679 after seven, and the ratio of the arms' hazards at a single moment climbs from 0.50 to 0.929 and falls back. Nothing is confounded and nothing is wrong. Survival removes high-risk patients from the control arm faster, and a hazard ratio compares whoever is left.

How often a randomised trial of eighty shows a Simpson reversal on some baseline variable, against how many were tabulated. The treatment helps every patient equally. On one nominated, strongly prognostic variable the reversal appears in 4.1% of trials. Tabulating 2, 5, 10, 20, 50, 100 variables of mixed prognostic value, some variable reverses in 4.1%, 6.5%, 10.8%, 16.1%, 26.1%, 34.1% of trials; stratifying the randomisation on the most prognostic one gives 0.2%, 2.9%, 7.7%, 12.2%, 23.4%, 31.9%.

Twenty variables nobody stratified on

A randomised trial of eighty patients in which the treatment helps every patient by the same six points shows Simpson's reversal on one nominated, strongly prognostic baseline variable in 4.07% of trials. Tabulate twenty baseline variables of mixed prognostic value and some variable reverses in 16.07% of trials; a hundred, in 34.13%. Stratifying the randomisation on the strongest variable removes its reversal and leaves 12.17% and 31.93%: the protection is exactly as wide as the list it was given.

How often a trial of 80 shows a Simpson reversal on its strongest baseline variable, against how many cut points are tried. The treatment helps every patient equally. Cut at its median, the variable's table reverses in 4.4% of trials; trying 2, 3, 5, 9 cuts, in 6.4%, 8.7%, 10.8%, 13.2%. Independent chances would give 33.3% at nine.

Five places to cut one variable

A continuous baseline variable has to be cut before it makes a subgroup table, and the cut is chosen. In a randomised trial of eighty where the treatment helps every patient equally, the strongest baseline variable cut at its median shows a Simpson reversal in 4.4% of trials; tried at five candidate cuts, some cut reverses in 10.8%, and at nine, 13.2%. Nine correlated cuts are worth about five independent chances, not nine. Across twenty continuous variables cut at five points each, some table reverses in 32.2% of trials — twice the rate with each variable cut once, and about thirty-one independent chances where the table holds a hundred.

How often a Simpson reversal appears on the strongest baseline variable when its cut is slid along it, by the trial's size. The treatment helps every patient by the same six points. At 40, 80, 160, 320, 640 patients: some cut in the middle 80% reverses in 18.8%, 23.4%, 24.3%, 19.2%, 10.6% of trials; some decile in 12.2%, 13.2%, 14.1%, 11.0%, 6.2%; the median in 3.9%, 4.4%, 4.2%, 3.1%, 1.7%.

A cut that slides

Slide the cut on a trial's strongest baseline variable along every gap between observed values in its middle eighty per cent, and a trial of eighty whose treatment helps everyone equally shows a Simpson reversal somewhere in 23.4% of trials — against 13.2% for the nine deciles and 4.4% for the median. Sixty-five cut positions are worth about 9.5 independent cuts, and at 640 patients, with 513 positions, about 7.2: a table that moves one patient at a time is a continuous scan, and its count is set by the range scanned rather than by the number of cuts. The reversing cuts spread along the whole range, in 2.62 separate runs at eighty patients and 8.36 at 640.

How often a trial with nothing to find shows a Simpson reversal somewhere, by how many variables are searched and how. Trials of eighty whose treatment helps every patient by six points, twenty baseline variables tracking the risk with correlations from 0.95 to 0.10; 4,000 trials. Searching the strongest 1, 2, 5, 10, 20: every cut, 23.4%, 30.7%, 41.0%, 46.9%, 49.0%; the deciles, 13.2%, 19.4%, 27.9%, 33.3%, 36.4%; the median, 4.4%, 6.9%, 11.3%, 14.4%, 16.2%.

Every variable, every cut

Slide the cut along every one of twenty baseline variables in a randomised trial of eighty whose treatment helps every patient equally, and some variable shows a Simpson reversal at some cut in 49.0% of trials — a coin toss with nothing to find. The twenty scans do not add like independent chances: if they did they would reverse 92.4% of the time. They read one patient risk between them, and the weak variables seldom reverse at all, so the strongest five supply 41.0 of the 49.0 points. A trial that shows one reversal usually shows several: 4.8 variables on average.

All essays