Normal Distribution
nothing is called 95% until it has been counted
Concept

Preregistration — where it appears

Naming in advance, before any data are seen, which analysis will be run and how its result will be judged. It restores a test's stated error rate at the cost of power against effects that turn up somewhere the plan did not look.

Named by 2 essays across one field — each of them below, with the objects they name alongside it.

Naming the analysis in advance, against correcting for all 20 of them. The prespecified analysis detects an effect that is in it 52% of the time at two standard errors and an effect elsewhere 5% of the time. The corrected slate detects it 23% of the time wherever it is. The two are worth the same when the chance of having named the right analysis is 38% — and that figure rises to 91% at four standard errors.

What naming it in advance costs

Preregistration is argued for as free. Against an effect of two standard errors hiding in one of twenty analyses, naming the right one detects it 51.5% of the time and naming the wrong one detects it 4.7% of the time; correcting all twenty detects it 22.5% wherever it is. The two are worth the same when the chance of having named correctly is 38%.

paths · Forking How much naming more analyses in advance is worth, at an effect of 2 standard errors, for four analysts' beliefs. Twenty analyses correlated at 0.6, one of which holds the effect. Each curve is an analyst whose belief falls by a fixed ratio from one choice to the next: 0.2, 0.5, 0.7, 0.9. The best number to name is 1, 3, 5, 15 respectively, and each best set holds between 80% and 90% of the analyst's belief. Every curve ends at 22.7%, the power of correcting for all twenty.

Naming a handful in advance

A preregistration that names three of twenty analyses sits between naming one and naming none, and whether it beats both depends on one thing: how much of the analyst's belief the three hold. An analyst who is 50% sure of a first choice, and whose belief halves down the list, gets 34.8% expected power from naming three, 28.3% from naming one and 22.7% from exploring all twenty. Spread the same 50% evenly over the rest and naming three is the worst of the three plans. The rule for adding one more to the list needs no belief at all: the next analysis must be at least 20.8% as likely as the first to be worth naming.

paths · Forking

Named alongside it

The objects these essays reach for when they reach for this one.

False positiveFamilywise error rateThe garden of forking pathsMultiple comparisonsResearcher degrees of freedomStatistical powerCorrelationEffect size

All concepts