Concept

Effective number of tests — where it appears

The number of independent tests that would carry the same chance of at least one false positive as a family of correlated ones. Overlapping subgroups or repeated looks at one sample count as fewer tests than their raw number, and a correction can be set against that smaller count.

Named by 3 essays across 2 fields — each of them below, with the objects they name alongside it.

How often a randomised trial of eighty shows a Simpson reversal on some baseline variable, against how many were tabulated. The treatment helps every patient equally. On one nominated, strongly prognostic variable the reversal appears in 4.1% of trials. Tabulating 2, 5, 10, 20, 50, 100 variables of mixed prognostic value, some variable reverses in 4.1%, 6.5%, 10.8%, 16.1%, 26.1%, 34.1% of trials; stratifying the randomisation on the most prognostic one gives 0.2%, 2.9%, 7.7%, 12.2%, 23.4%, 31.9%.

Twenty variables nobody stratified on

A randomised trial of eighty patients in which the treatment helps every patient by the same six points shows Simpson's reversal on one nominated, strongly prognostic baseline variable in 4.07% of trials. Tabulate twenty baseline variables of mixed prognostic value and some variable reverses in 16.07% of trials; a hundred, in 34.13%. Stratifying the randomisation on the strongest variable removes its reversal and leaves 12.17% and 31.93%: the protection is exactly as wide as the list it was given.

reversal · Simpson

How often a trial of 80 shows a Simpson reversal on its strongest baseline variable, against how many cut points are tried. The treatment helps every patient equally. Cut at its median, the variable's table reverses in 4.4% of trials; trying 2, 3, 5, 9 cuts, in 6.4%, 8.7%, 10.8%, 13.2%. Independent chances would give 33.3% at nine.

Five places to cut one variable

A continuous baseline variable has to be cut before it makes a subgroup table, and the cut is chosen. In a randomised trial of eighty where the treatment helps every patient equally, the strongest baseline variable cut at its median shows a Simpson reversal in 4.4% of trials; tried at five candidate cuts, some cut reverses in 10.8%, and at nine, 13.2%. Nine correlated cuts are worth about five independent chances, not nine. Across twenty continuous variables cut at five points each, some table reverses in 32.2% of trials — twice the rate with each variable cut once, and about thirty-one independent chances where the table holds a hundred.

reversal · Simpson

How often a trial whose effect is the same for everyone shows a significant subgroup interaction, by the size of its subgroup table. Four hundred patients, 80% power overall, characteristics whose latents correlate 0 (their split indicators 0.000). Some interaction test reaches 5% in 5.10% at 1, 9.13% at 2, 17.43% at 4, 34.63% at 8, 47.13% at 12, 56.23% at 16; the equicorrelated-normal integral gives 5.00%, 9.75%, 18.55%, 33.66%, 45.96%, 55.99%. A Bonferroni reading of the eight holds 5.57%.

Sixteen subgroups and one effect

A trial of four hundred with 80% power tabulates its result by eight binary characteristics — sixteen subgroups — and the treatment's effect is the same for every patient. Among trials significant overall, 93.5% report at least one subgroup where it is not, and the median one reports seven. Some subgroup-against-the-rest interaction reaches 5% in 34.6% of trials — eight independent chances — and when the characteristics overlap as two age thresholds do, 18.6%, about four. The effective count is set by the correlation between the characteristics' split indicators, (2/π) arcsin of their latent correlation, and a Bonferroni reading of the eight holds the rate at 5% whatever the overlap.

alongside · Repetition

Named alongside it

The objects these essays reach for when they reach for this one.

Multiple comparisons Subgroup analysis Randomisation Simpson's paradox Baseline imbalance Bonferroni Correlation Dichotomisation Familywise error rate The garden of forking paths Interaction Prognostic factor

All concepts