Concept

Researcher degrees of freedom — where it appears

The choices in an analysis that the data could have decided — which outcome, which exclusions, which covariates, which transformation. Each one multiplies the chances of a significant result, and the family they form is rarely reported and often not enumerable.

Named by 2 essays across 2 fields — each of them below, with the objects they name alongside it.

Naming the analysis in advance, against correcting for all 20 of them. The prespecified analysis detects an effect that is in it 52% of the time at two standard errors and an effect elsewhere 5% of the time. The corrected slate detects it 23% of the time wherever it is. The two are worth the same when the chance of having named the right analysis is 38% — and that figure rises to 91% at four standard errors.

What naming it in advance costs

Preregistration is argued for as free. Against an effect of two standard errors hiding in one of twenty analyses, naming the right one detects it 51.5% of the time and naming the wrong one detects it 4.7% of the time; correcting all twenty detects it 22.5% wherever it is. The two are worth the same when the chance of having named correctly is 38%.

paths · Forking

The false-positive rate against the number of analyses, on pure noise. The data has no effect in it. Each analysis is correct and each p-value is honest. With 20 correlated outcomes available, something reaches p below 0.05 58% of the time.

Twenty analyses of nothing

Twenty honest, correct analyses of data with no effect in it find something significant 57% of the time. Nobody p-hacked, every individual p-value is right, and the reported one is the smallest of twenty.

testing · Forking

Named alongside it

The objects these essays reach for when they reach for this one.

False positive The garden of forking paths Multiple comparisons Bonferroni Correlation Effect size Familywise error rate p-value Preregistration Statistical power

All concepts