Theme

The thread: The rule is part of the result

The same observations mean different things depending on how they were going to be collected and when they were going to be looked at. A stopping rule changes a p-value without changing a number in the dataset, and a prior changes an interval's endpoints while leaving the likelihood untouched.

One experiment, with the blocks getting smaller as the target comes into range. A single run at a requirement of 0.25, with the block sizes 5, 5, 11, 25, 11, 8, 3, 2 and a total of 70 observations in 8 blocks. The rule stops when the observations in hand reach z²σ̂²/d², with σ̂² pooled from the within-block contrasts — an estimate that moves as the run goes on, so the target moves too. Early blocks are large because the target is far away and cannot be overshot; late ones are small because a block is the granularity of the answer. The interval afterwards is built from the 8 block means and from nothing the rule looked at, and it has 7 degrees of freedom against the rule's 62.

The block size as a schedule

A block size that changes

The blinded rule's exactness never needed the blocks to be the same size. Letting the size be chosen from the contrasts as the run goes on leaves the coverage exactly where it was — and runs straight into an identity that says what a schedule can and cannot buy.

What each rule gives up against an oracle that is arithmetic. Expected squared error of the candidate each rule selects, minus the expected squared error of the best candidate in the table, over 500 draws of 120 rows. Both quantities are closed forms — σ_S²(1 + q/(n − q − 1)) — so the only Monte Carlo here is over which candidate got picked. The hold-out spends half its rows measuring what the criterion computes, and pays 1.8 times as much for it. Schwarz's criterion is worst because it is answering a different question: which candidate contains the truth, rather than which one forecasts best.

Scoring a search without spending data

A criterion is a prediction of the hold-out

A rolling hold-out spends half the sample measuring what a criterion computes from all of it. Against an oracle that is arithmetic rather than an estimate, the criterion gives up 0.01701 and the hold-out 0.03200 — and the number the hold-out reports for its own winner is optimistic by more than either.

A design chosen rather than looked up

A design is a number

A standard design is taken from a catalogue and then measured. Turn the arithmetic round and a design becomes the answer to an optimisation — and over 121 candidate settings the search keeps nine of them, which are exactly the nine a catalogue would have offered, at weights nine equal runs cannot express.

The correction is not a property of the sample. tr(HΩ)/q for each of fifteen candidates, at ρ = 0.7. Two candidates that fit the same number of coefficients need corrections that differ by as much as 1.49, because one of them is fitting the persistent predictors and the other is not — so no single number can be right for both, and the scalar n/n_eff = 5.537 is above every one of them. The four predictors carry persistences 0.9, 0.6, 0.3, 0; at one persistence for every column the whole spread collapses and a scalar looks exactly as good as the trace.

Counting what is independent

A penalty is a trace

Akaike's 2q is not a count of coefficients. It is the answer a trace collapses to when the rows are independent — and once they are not, the trace is still the right object and is no longer the count.

A proposal that moves more, refused more often. The two halves of the trade, both exact, on the 410 admissible assignments of twelve units. The integrated autocorrelation time of an imbalance the rule was never handed falls from 7.30 at one swap to 3.97 at three, and the acceptance rate falls with it, from 58.8% to 40.8%. A rejected proposal costs one evaluation and leaves the chain where it was, so acceptance is not the price of anything and the ranking by acceptance is the reverse of the ranking by cost. Past three the family folds: exchanging k of six from each arm is the complement of exchanging six − k, so k = 5 has the same 36 proposals as k = 1 and k = 6 has 1.

What a block may vary

A proposal that moves more than two units

The walk's autocorrelation is a fact about its step size and not about its acceptance rate. Exchanging three units from each arm mixes nearly twice as fast as exchanging one, and is refused a third more often.

The stopping rule costs more than the weighting does. Coverage over 2000 runs of a trial whose variance ratio drifts by a factor of twenty, at three ways of deciding when to stop. Twelve blocks fixed in advance is the top line and reproduces what a trial of fixed length delivers. Stopping when the reported interval is short enough is the bottom line, and it costs between 3.0% and 5.5% of coverage — including for the rule that is told every block's true ratio, which is what says the shortfall belongs to the stopping and not to the weights. Stopping on a width predicted from the within-arm sums of squares is the middle line, and it is back at the fixed-length values. The standard error on each point is 0.49%.

When a fixed width is reached

A width the trial has to stop for

The weighting that covers at 94.9% on twelve blocks covers at 91.5% when the trial stops as soon as its interval is short enough — and so does the rule that is told every block's true variance ratio. The shortfall is the stopping, not the weights.

$Four rules, three shapes, and no ordering that survives. The variance of the unadjusted treatment estimate under each rule, as a fraction of the variance a coin gives, over 450 trials of 200 units each. Against a covariate that enters linearly the rule that reads the number nearly halves it. Against a threshold at 1 it removes about a fifth. Against a quadratic every rule here is at or worse than a coin — they are all optimising a criterion that is one over the variance of an estimate in a model this outcome does not obey, and a constraint that helps nothing still costs something. Nothing in a trial says which column it is in.$ The shape the covariate enters by

Balanced on the wrong function

A rule that reads a covariate's numbers halves the variance of the treatment estimate, if the covariate enters the outcome as a straight line. If it enters as a threshold the rule is worth a fifth of that, and if it enters as a curve every rule here is worse than a coin.

What the interim sees, at an effect of 1. The same 60 observations, estimated two ways. Keeping the arms separate gives 0.995, which is σ. Pooling them without separating the arms — the price of staying blind to the comparison — gives 1.114, against the identity √(1 + Δ²/4σ²) = 1.118. The sample size is proportional to the variance, so a blinded design at this effect asks for 25% more units than it needs, and it does so systematically rather than by chance.

Designs that change while they run

Choosing n after looking

Re-estimating the sample size from an interim is the one adaptation with a defence, and the defence is exactly what it costs: an analyst kept blind to the arms measures a spread that contains the effect, so the design overshoots by 1 + Δ²/4σ². Re-estimating the effect instead breaks the error rate.

The allocations the rule could have made, from these exact patients. One 200-patient trial allocated by response-adaptive randomisation, re-randomised 999 times. No outcome is redrawn anywhere in this figure: each re-randomisation runs the same rule over the same patients in the same order, so what is drawn is the set of experiments that could have happened rather than a sampling distribution. The observed |z| is 1.417, 258 of the 999 re-randomisations reach it, and the p-value is (1 + 258)/(1 + 999) = 0.2590. The curve is the half-normal the ordinary analysis reads the same statistic against; its 5% point is 1.96 and this distribution's is 2.101.

The reference distribution the design supplies

The experiments that could have happened

An adaptive trial's allocation is a function of the outcomes it will later be compared against, so the ordinary analysis rejects a true null 9.2% of the time. Hold the outcomes fixed, re-run the rule that assigned them, and count — the same statistic against a reference distribution the trial could actually have drawn from is back at 4.0%.

The fourth-order expectation, by two routes. Every inner product in an eight-term slice of the dictionary at ρ = 0.5, computed from the linearisation and Mehler's formula and counted from two hundred thousand draws of a correlated pair. The entries that matter are the ones off the main effects: ⟨f(X)u(Y), g(X)v(Y)⟩ is a fourth-order expectation, which the independent-covariate field could not write down. The worst departure is 1.99 standard errors over 36 pairs, measured in each pair's own error because the entries differ in size by two orders of magnitude.

When the two are not independent

The fourth moment that was missing

Mehler's formula makes the main effects exact at any correlation and stops there, because the interactions need an expectation of four Hermite functions rather than two. A linearisation turns the four into two, and the whole geometry becomes closed again.

Six groups of 10, each fitting its own slope, then borrowing. Each faint line is one group's own least-squares slope through its own centre; each solid line is that slope after pooling towards the population slope of 0.79. Every group has the same 10 observations. The group whose x values span 0.4 has a slope standard error of 1.86 and moves 91% of the way in; the group spanning 2.0 has a standard error of 0.37 and moves 28%.

Hierarchy past one number

The slope that borrows

Pooling a mean makes it look as though how much a group borrows depends on how much data it has. Pool a slope instead and the illusion breaks — ten groups with ten observations each can borrow anything from 28% to 91%, decided entirely by where those ten observations were placed.

Two diagnostics, one answer, two different moments. The two-chain statistic on a covariate probe and on the trial's own difference in arm means, at seven tolerances of a fourteen-unit rule, against the enumerated truth. Both are quiet wherever the set is one set and both fire wherever it is not, at every tolerance — which is what says the outcome probe is the same test rather than a resemblance of it. The difference between them is not accuracy and it is not power. It is when: the covariate probe can be run before a single outcome exists, when a practitioner can still loosen the rule or change the sampler, and it can be run again on a different function if it comes back quiet. The outcome probe runs after the trial, on the one column the trial produced, and what it can do with a positive verdict is repair the p-value rather than the design.

The diagnostic after the trial

The statistic the p-value is about

The test for whether a balanced-assignment walk reaches its whole set is run on a covariate function chosen before the trial. Run on the difference in arm means it is the same test, and it is about the number the trial publishes.

Where one rule becomes three. Every arrival in 200 simulated trials is put to all three scores, and the picture is how often they would send that patient to different arms. The range and the pairwise sum are the same rule at two arms and at three — for sorted counts the pairwise sum is twice the range, so the arm that minimises one minimises the other — and they part company at four, where the pairwise sum is 3(d − a) + (c − b) and the range still sees only d − a. The variance disagrees with both from two arms onwards, on 5.4% of arrivals at two and 27.0% at five, because the scores are summed over 3 factors and a sum of squares does not order the candidates the way a sum of absolute values does. All three are called minimisation.

More arms than two

Three arms and three scores

Minimisation balances a trial by keeping the arms' counts even inside every prognostic factor. With two arms there is one way to measure how uneven two counts are. With three there are several, they are all called minimisation, and they send different patients to different arms.

Three series and one relation between them. Above, three series generated from Δy = Πy₋₁ + ε with Π of rank 1. Below, the combination y1 −y2. It stays inside a band of 9.5 while the series themselves travel 28.4. The count of combinations that behave this way is the rank of Π, and it is what every method in the field sets out to estimate.

Three series, and a count

Three series and a count

A pair of series is either tied together or it is not, so its whole inference is one test with one answer. Three can carry none, one or two relations at once — and the thing being estimated stops being a slope and becomes an integer, read off the gap in a spectrum whose top eigenvalue holds at 0.25 while the rest fall like 1/n.

Exact in the corner, where nothing was. Coverage of a nominal 95% interval on five designs, at a required half-width of 0.3. The first four are the two-arm field's own and the fifth is its corner — two variances, block sizes that swing by eight, and an allocation that alternates between five to one and one to five — where neither of that field's two conditions holds. The effective-size weights over-cover there at 98.40%; the weights h_b(λ) = (1/m_A + λ/m_B)⁻¹ cover at 94.84%, and at 94.84% when λ is estimated from the within-arm contrasts rather than known. Nothing here is supposed to move.

The weights the corner needs

Weights that need only a ratio

A fixed-width interval about a difference is exact under either of two conditions and under neither in the corner. It is exact there too, and the only thing it needs is how much larger one arm's variance is than the other's.

weakly informative — Beta(2, 2), updated by 5 of 20. The prior is worth 4 observations. With 20 observations the posterior mean is 0.292, against a data proportion of 0.250 and a prior mean of 0.500.

The prior, doing visible work

What a prior is worth

A prior is not a philosophical position, it is a component with a stated size. For a proportion it is worth exactly a + b observations, which turns "how much does the prior matter" from an argument into a subtraction.

A search with no fixed point

When the benchmark is a candidate

A specification search with a benchmark nailed down is the case with a closed form. Take the nail out — let the model that would have been reported be one of sixteen, chosen by the same data as its rivals — and the same true null is read three ways, at 2.0%, 7.8% and 76.2%.

Testing at 0.05 every time the data is looked at. The null is true in every one of these trials and the test is correct every time it is run. Looking once rejects 4.9% of the time, as it should; looking ten times rejects 19.2% of the time. Nothing changed except permission to look.

Stopping rules

When the looking happens

A p-value is defined relative to a sampling plan, so the same data means different things under different stopping rules. Testing five times at the nominal level rejects a true null 14% of the time, and no observation in the dataset changed.

The value that is not there

Three mechanisms and one dataset

Four rules for which outcomes go missing, each calibrated to lose the same 35% of the rows and each leaning on what it reads with the same coefficient. Three leave the fitted slope exactly where it was, and the one that reads the outcome moves it by 0.163531.

The power trials actually have when sized for 80% from a pilot of 10. Four thousand pilots of 10 observations, each sizing a trial for 80% power at half a standard deviation from its own standard deviation. 55.9% of the trials have less than 80% power and 11.1% less than 50%; the median trial has 76.8%.

What a sample-size calculation was given

The spread a pilot supplies

A trial sized for 80% power from a pilot's standard deviation is sized from an estimate that is too small more often than not. With a pilot of ten, 55.9% of the trials it sizes have less than 80% power and 11.1% less than 50%, although the planned sample is right on average. Sizing from the pilot's 80% upper confidence limit instead leaves 19.8% short, at 1.65 times the sample; from its 90% limit, 10.0% short at 2.12 times.

Searching among fitted models

A null with a model in it

The distribution to read the winner of a table against cannot be resampled from the data, because the data does not contain the null. It has to be generated from a model — which is the assumption the resampling was chosen to avoid.

Three priors on the spread, at a true τ of 0.5. The posterior for τ under a flat prior (mean 1.66), a half-Cauchy of scale 1 (1.32) and one of scale 0.25 (1.15). The three answers differ by 30% of the widest. The prior does visible work when eight groups cannot separate a small spread from none, and almost none when they can.

The spread, and its own uncertainty

A prior on the spread

Integrating over the population spread means putting a prior on it, which sounds like the objection rather than the repair. The prior's effect is measurable, it is invisible where the groups are clearly different, and the reflex choice for a scale parameter turns out not to have a posterior at all.

How far each reference distribution's 95% point falls short. Seven constructions on rows that repeat each other, at a block length of 5 and 200 draws, against the statistic's own 95% point of 3.0224 computed from three thousand draws of the same world. Reading down: a multiplier on every row keeps no dependence at all and is 44% short; a multiplier shared along a block keeps the triangle; a fixed block keeps the same triangle and is 8% closer, which is the pair that says a taper is not what decides this; the stationary bootstrap; the two tapered blocks, both further short than the untapered one at this block length; and errors generated from a fitted model, which is the only construction here not bounded by what the residuals report.

A block, weighted inside itself

A taper and a critical value

Two constructions whose tapers visibly differ give the same critical value, and two that share a taper exactly do not. Adding a construction whose taper is a decision rather than an accident says which half of that is true.

The zero was a fact about independence. What a balancing rule handed every main effect of both covariates removes of a pure interaction, as the covariates are allowed to move together. At ρ = 0 it is exactly nothing — at machine precision, at any number of main effects — which is the independent-covariate result and is correct. It is not small anywhere else: the product of the two covariates loses 64.0% of itself by ρ = 0.5, because h₁h₁ = h₀ + √2·h₂ and Mehler pairs h₂ with h₂ at ρ². Four interactions are drawn and none of them keeps the zero.

When the two are not independent

A zero that was an assumption

A rule handed every main effect of both covariates removes exactly none of a pure interaction. That is true at machine precision, it is a fact about independence, and it dies as the square of the correlation.

More arms than two

Balancing towards unequal targets

A three-arm trial allocating two to one to one is the ordinary case, and a balancing rule built from raw counts does not know it. It balances the arms towards equality inside every factor level, delivers a third to each arm, and reports that it minimised imbalance.

How wrong the ratio is allowed to be. λ enters only through the weights, so misstating it leaves the estimate unbiased and moves two things — the interval's calibration and its efficiency — both of which are closed forms of the design. Coverage stays at its level over a factor of two in either direction (94.93% at half the truth, 94.27% at twice it) and starts to go at a factor of five. An estimate on hundreds of within-arm degrees of freedom is never wrong by anything like that, which is what makes the feasible rule usable rather than merely definable.

The weights the corner needs

Blinded, and still exact

The one number the exact interval needs is a ratio of within-arm spreads, which is a contrast and contains no mean — so a rule forbidden to look at the effect may compute it, on more degrees of freedom than the interval itself has.

Every way of splitting 16 units into two halves. All 12,870 assignments, enumerated. The spread of the standardised imbalance is exactly 2/√n = 0.500, whatever the covariate's own distribution, and 33.3% of assignments differ by more than 0.5 standard deviations. Randomisation does not deliver balance; it delivers a known distribution of imbalance.

Decided before the data

Randomisation is not balance

A third of all ways to split sixteen units leave the two halves more than half a standard deviation apart on a covariate. What randomisation delivers is not balance but a known reference distribution — and it makes a test exact with no assumption about the data's shape at all.

Designs that change while they run

Randomising towards the winner

Allocating more patients to the arm that is doing better is the humane thing to want and it buys nothing statistically: at a fixed total it costs thirty points of power. And because the allocation is a function of the outcomes, the ordinary test on it rejects a true null 7.8% of the time before any time trend is applied — and 58% after one.

Stationary is not the same as convergent. How far each k-swap walk is from uniform after t steps, started at the least balanced admissible assignment of 410. Every one of these chains has a symmetric proposal and rejects by standing still, so every one of them is doubly stochastic and every one preserves the uniform distribution exactly. Only five of the six get there. Exchanging all six units of each arm is a single proposal — the complement — and the admissible set is closed under complement, so the walk takes it every time and oscillates between two assignments for ever: after 160 steps it has visited 1 state and sits 0.9976 from uniform. Its stationary distribution is a fact about the matrix; its limit does not exist.

What a block may vary

Stationary is not convergent

A walk that exchanges every unit in each arm preserves the uniform distribution exactly and never gets near it. Every doubly stochastic matrix has the same stationary distribution; only some of them have a limit.

Two promises, and no rule here keeps both. A fixed-width procedure promises two things: that the interval covers at its nominal rate, and that it is no wider than the width asked for. Over 1500 runs of the modelled weighting, a rule that stops when the interval it will report is short enough keeps the width — only 2.0% of runs come out wider than 0.34 — and covers at 91.13% against a nominal 95%. A rule that stops on a width predicted from the within-arm sums of squares covers at 94.80% and comes out wider than promised on 42.3% of runs. The two promises are in conflict because keeping the second one exactly requires conditioning on the very quantity that has to be independent of the stopping time for the first.

When a fixed width is reached

Stopping on the arms

The width a trial will report is predictable from quantities the interval is not about. A rule that stops on the prediction covers at 94.5% where one that stops on the interval covers at 91.5, and it costs two blocks and half of the width promise.

The rate and the size of a disagreement

The quarrel that changes the winner

A disagreement about the tuning parameter costs 0.031 when it changes which candidate the table selects and −0.0007 when it does not. The distance between the values disagreed about has nothing to do with it.

Balancing on what was recorded first

The rule that can be guessed

A balancing rule improves as it becomes more deterministic, and a deterministic rule can be worked out in advance from information the person enrolling the patient already has. At full determinism 87.6% of assignments are guessable, and an investigator who acts on the guess produces a treatment effect of three quarters of a standard deviation where the truth is zero.

The reference distribution the design supplies

The test that needs the rule

A randomisation test assumes almost nothing about the data and one thing about the experiment. Tell it a fair coin produced an allocation that an adaptive rule produced — which is what every off-the-shelf permutation routine does — and it rejects 8.0% of true nulls where knowing the rule gives 4.0%.

Where the residual test's statistic actually falls, at n = 200. Four thousand pairs of unrelated random walks, each regressed on the other and each residual tested for a unit root. The statistic is computed as a t and its distribution is not a t: five per cent of it falls below -3.38, where the ordinary one-sided 5% point of a t on 198 degrees of freedom is -1.65. Everything left of -1.65 — 70.2% of the whole distribution — is a pair of unrelated walks that a t table calls cointegrated.

Series that move together

The test with no table

The statistic that separates a real long-run relation from a spurious one is computed as a t and is not a t. At two hundred observations its 5% point is −3.38 where the t table says −1.65, and reading it against the table calls two unrelated random walks cointegrated 70.5% of the time.

What a dictionary buys and what it costs

What the extra function buys

A rule balancing the mean of each covariate has a worst case of exactly zero. Adding the median split — the other thing every trial balances — leaves it at exactly zero, and one square moves it.

The crossing is in the dependence, not in the split. Regret of each rule as the design and the errors are made persistent at the same coefficient, scored on fresh rows because the closed form assumes exactly what is being taken away. An optimism theorem counts rows; when the rows repeat each other there are fewer of them than there are rows, the penalty is too small for the fit it is correcting, and the criterion starts buying coefficients it should not — its average winner grows from 3.31 coefficients to 3.90. The hold-out never used the theorem and overtakes at ρ ≈ 0.81. Schwarz's criterion, worst of the three on independent rows, is best on repeating ones — its heavier penalty is right for the wrong reason.

Scoring a search without spending data

Where the two searches cross

The obvious dial between a criterion and a hold-out is how much of the sample to hold out, and moving it never changes the answer. The dial that does is one nobody chooses — how much each row repeats the one before it — and the two rules change places at about 0.81.

The same data, one regression per choice of left-hand side. The two-step procedure has to put one series on the left, and with 3 series there are 3 ways to do it. Each returns a relation and a residual test; the 5% point is -3.71, simulated. Here they do not agree: 2 of 3 reject, and the relations they report are written with a 1 in the position of whichever series was on the left, so they can be compared. Nothing in a printed output records which regression was run.

Three series, and a count

Which series goes on the left

The two-step procedure has to pick a series to regress the others on, and nothing in its output records which. With a pair that choice never changes the verdict. With three series and one relation between them, the three choices disagree about whether the system is cointegrated at all 98.0% of the time.

The guarantee, as the basis is allowed more functions. The lower line is the best worst case over the six named shapes for a basis of each size, found by scoring every subset of the dictionary — an exact answer, since the problem is finite. One function guarantees 2.3%, which is nearly nothing; three guarantee 59.0% and the basis that does it is the covariate, its square and its cube, with no indicator in it. The upper line is the same problem with the basis drawn rather than fixed, which is worth 2.09 times as much at two functions and 1.32 at three. The two lines converge because a basis large enough to protect everything has nothing left to randomise over.

Choosing what the rule reads

Which shapes are worth protecting

Choosing a basis by its worst case is a finite problem with an exact answer. The answer has no tie in it, which a maximin optimum is supposed to have — and the tie comes back, along with twice the guarantee, when the basis is drawn rather than chosen.

One forecaster, one sample, and the bins it was read in. A reliability diagram of 500 forecasts in 10 equal-count bins, drawn over the curve the same forecaster has in population. The forecaster is honest, so its population curve is the diagonal exactly and every departure the points show is sampling. The sample's reliability term reads 0.002262 and its expected calibration error 0.0358, against a true reliability of 0.000000. Both are properties of this binning as much as of this forecaster: at 5 bins and at 50 the same honest forecaster reports 0.001429 and 0.014100.

A forecast that is a probability

A curve that is a binning

A forecaster with no miscalibration in it at all reads 0.001429 at five bins and 0.014100 at fifty, on the same five hundred forecasts. The closed form is K/n times the forecaster's own irreducible score, and subtracting it returns zero.

Naming the analysis in advance, against correcting for all 20 of them. The prespecified analysis detects an effect that is in it 52% of the time at two standard errors and an effect elsewhere 5% of the time. The corrected slate detects it 23% of the time wherever it is. The two are worth the same when the chance of having named the right analysis is 38% — and that figure rises to 91% at four standard errors.

The analyses that were available and not run

What naming it in advance costs

Preregistration is argued for as free. Against an effect of two standard errors hiding in one of twenty analyses, naming the right one detects it 51.5% of the time and naming the wrong one detects it 4.7% of the time; correcting all twenty detects it 22.5% wherever it is. The two are worth the same when the chance of having named correctly is 38%.

The operating characteristic, and the point that minimises harm at 0.10% prevalence. The published pair — 90% sensitive, 95% specific — is the open mark. With a miss costing 100 times a false alarm and a prevalence of 0.10%, the threshold that minimises expected cost sits at 74.7% sensitivity and 98.81% specificity, with a predictive value of 5.9%.

Two tests, a threshold, and the rate they are read against

The test is a point somebody chose

A test reported as 90% sensitive and 95% specific is not two properties of a test. It is one property read at a threshold, and the threshold that minimises harm runs from 3.05 standard deviations of the score at a prevalence of one in ten thousand to −0.12 at one in two — 45% of cases detected at one end and 99.9% at the other.

What a sample-size calculation was given

The chance a trial succeeds

A trial of sixty-four per arm has 80% power at an effect of half a standard deviation. If the effect is only believed to be about half a standard deviation, give or take a quarter, the chance the trial reaches significance is 69.2%; give or take a half, 61.4%. Reaching 80% then takes 113 per arm, or 1,268 — and when the belief is uncertain by three quarters of a standard deviation no number of patients reaches 80%, because the chance can never exceed the 74.8% probability that the effect is positive at all.

The best of 8 arms, tested as though it were the only one. 8,000 trials with no effect in any arm. Stage one runs 8 arms at 60 each, the best is carried forward, and stage two adds 30 more to it and to the control. The histogram is where the final statistic lands and the curve is the standard normal it is being read against — shifted right, because the arm was chosen for being ahead. 10.5% of these trials clear 1.96 against a claimed 5%, and the value that actually holds the rate for this design is 2.313.

Designs that change while they run

Dropping the losers

Carrying the best of eight arms forward and testing it at 1.96 rejects a true null 10.3% of the time — the hypothesis was chosen by looking at the data, so the statistic is a maximum wearing a single comparison's clothes. The value that holds the rate is 2.313, and it has to be solved for.

Four criteria on 5 designs, on a square region. Each design scored as an efficiency — its value over the best attainable — so four criteria in as many different units sit on one scale where 1 is the optimum. Rows are ordered by D. D picks 13-run exchange; A picks face-centred composite; G picks 13-run exchange; I picks face-centred composite. Every design has been scaled to just fit the region first, because a design run at settings the region does not contain is not a competitor on it. The disagreement is the point: the letter is a choice, and it is almost never reported as one.

A design chosen rather than looked up

Four letters and two camps

D, A, G and I are four ways of turning one matrix into one number, and they do not agree. The design that wins on D is the worst thing here on I. And the same two designs swap places entirely when the region changes from a square to a disc — on all four criteria at once.

The one thing a trial always reports is the one thing that survives. How wrong three p-values are when they are computed over the half of the admissible set a single walk can reach, rather than over all of it, at a fourteen-unit trial where the whole set can be enumerated. The two-sided p-value on the difference in arm means — the number a trial publishes — is wrong by exactly nothing, at every row, to machine precision. That is not luck: the two components are complement pairs and the difference in arm means is exactly negated by the complement, so the distribution of its absolute value is the same on both. A one-sided p-value on the same statistic is out by as much as 0.112, and the largest response observed in the treated arm — a safety reading rather than an effect, and the one statistic here that is not odd under the complement — by as much as 0.172. The defect survived because the commonest thing anybody computes is the one quantity it cannot touch.

The diagnostic after the trial

Half a reference distribution

A walk that reaches half its admissible set reports the two-sided p-value exactly right, to the last digit, for ever. A one-sided one it puts on the wrong side of five per cent about once in thirty.

A search with no fixed point

Residuals that keep their own variance

A reference distribution for a search has to be generated from a fitted model, and the generator draws residuals. Four ways of drawing them keep four different things — and the one this site has reached for three times repairs nothing at all here.

One rule keeps its promise and the other keeps its budget. Both stopping rules at five requirements, 1,500 experiments each, with a first stage of 5. The upper curve is the two-stage rule: 97.1%, 96.2%, 95.9%, 96.1%, 96.0% — at or above 95% at every point, which is a theorem rather than a tendency, because its interval is built from a spread estimated before the stopping point was chosen. It pays 2.06×, 2.01×, 2.00×, 1.99×, 1.99× the observations that knowing σ would need. The lower curve is the rule that re-estimates after every observation: 94.5%, 89.3%, 91.0%, 91.5%, 94.3%, on 0.98×, 0.87×, 0.88×, 0.93×, 0.96×. The second rule is the one anybody would run and the first is the one whose claim is true.

What the design is asked to guarantee

Stopping when it is precise enough

An experiment that runs until its estimate is precise enough is the natural design and the one with a theorem against it. Its two-stage cousin keeps its promise exactly, for every unknown spread, and pays twice the observations for it.

Four analyses of the same 3-arm trials, under a true null. 250 trials of 150 patients, 3 arms, minimisation with p = 0.85, 99 re-randomisations for each exact test. Two statistics — an F on the arms alone and an F on the arms after the balanced factors — against two reference distributions: the table the statistic is named for, and the distribution the allocation rule itself generates when the outcomes are held fixed and the rule is re-run. Only the first cell is wrong, and it is wrong in the direction that costs power rather than the one that manufactures findings: 0.0% where 5% is claimed. Either repair works — adjusting for what the rule balanced, or asking the rule what it would have done.

More arms than two

The analysis after three arms

An unadjusted analysis after a two-arm balancing rule rejects 0.6% of true nulls where it claims 5%. With three arms and a deterministic rule it rejects none at all — and the repair is the same repair, which is a sentence and a column in the model.

The weights may not read the block they weight. A weighted least squares decomposition needs weights that are constants, or at least independent of the differences they multiply. One λ̂ pooled across the trial is estimated on hundreds of degrees of freedom and is effectively a constant; a λ̂ estimated inside each block is estimated on that block's own two or three, and is correlated with the difference it weights. Coverage falls from 94.68% to 82.76% — and the interval gets wider while doing it, 0.5163 against 0.3024, which is the signature of weights that are noise.

The weights the corner needs

The condition that cannot be dropped

The weights may not read the block they weight. Estimate the variance ratio inside each block rather than across the trial and the coverage falls to 83% — on an interval that is at the same time seventy per cent wider.

The criterion, and what it assumes

The design that needs the answer

Every design this site has computed is optimal whatever the experiment turns out to say, because X′X does not contain the parameters. For a non-linear model it does, so the best place to take a measurement is a function of the number the measurement exists to find — and guessing it three times too low costs two and a half times more than guessing it three times too high.

The correction, at a generating α of -0.2. Each point is one step: the gap at the end of yesterday against the change in y today. The fitted slope is -0.202 against the -0.2 the data was generated from, which means 20% of any disagreement between y and its long-run relation with x is undone in a single step. A shock therefore has a half-life of 3.1 steps. Neither series is stationary; the relation between them is.

Series that move together

The model that corrects its error

A cointegrated pair can always be written as a mechanism — today's change in y depends on yesterday's disagreement between y and its long-run relation with x. The coefficient of that disagreement is recovered from data that never saw it — and on unrelated series the same fit produces one a t table would call real 41% of the time.

τ̂ across 2,000 datasets of 12 groups, true τ = 1.5. The population spread is not supplied to a hierarchical model — it is estimated from how far apart the group means are, after subtracting the noise that would separate them anyway. It averages 1.41 here against a true 1.5, and comes out exactly zero on 5% of datasets.

Groups that borrow

The prior the data estimates

A hierarchical model needs a population spread, and it does not ask for one. It reads τ off the distance between the group means — biased six per cent low, exactly zero on 53% of datasets where the groups are identical — and the prior stops being a belief.

Exact coverage, at every block size. Coverage of the interval each rule reports, at a nominal 95%, over 2,500 runs each with a standard error of 0.44 points. The blinded rule stops on the within-block contrasts and reports an interval built from the block means, and those two are independent whatever the rule does — so the interval is an ordinary t interval on b − 1 degrees of freedom and its coverage is exact. It is exact at every block size drawn. The interval a practitioner writes at the purely sequential rule's stopping time covers 91.72%, and Stein's two-stage rule is exact for the same reason as the blinded rule and spends 2.10 times the observations to be so. The bars are truncated at 86% so the differences can be seen.

What the procedure may not read

The rule that cannot see the mean

A sequential rule stops when its own estimate of the spread is small, which is more often on the samples whose spread came out low — so the interval afterwards is short. There is a way to keep updating the estimate and stop being able to see the mean at all.

Whichever dial made the set thin, the crossing is at the same thinness. Each curve is one dictionary, swept over eight tolerances at two hundred units; a point above the line is a set thin enough that walking beats hunting. The curves lie nearly on top of one another, which is the answer to whether the crossing is a fact about the tolerance or about the thinness it produces: the crossings sit between one admissible assignment in 176 and one in 268 for dictionaries of 3 to 6 functions. The mechanism is that a hunt costs exactly 1/p and a walk costs almost the same everywhere — between 82 and 394 evaluations per usable draw across the whole table — so the crossing is wherever 1/p reaches a number that does not move.

What a dictionary buys and what it costs

The set a dictionary leaves

A rule constrained on six functions at a loose tolerance leaves a set as thin as one constrained on three at a tight one. Both sampling methods cross over at the same thinness, and the tolerance where that happens moves by a factor of three.

How long a walk has to be given. Every equal split of twelve units is enumerated, the 410 admissible ones are found, the transition matrix is built, and the distance from uniform is computed exactly at each step — no simulation anywhere. The walk is started at the least balanced admissible assignment, which is the state a rejection sampler is least likely to have handed it and the one a burn-in has to cover. It is 0.0849 away after twenty steps and 0.00008 after ninety. A real cost, and a small one, and naming it is what stops it being assumed to be zero.

When the two are not independent

Walking the admissible set

A rerandomisation test hunts for admissible assignments and throws away the rest. A walk visits them instead — and it is exactly uniform only because it stands still when a proposal fails, which is the step that looks like waste.

The crossing barely moves. Both methods' costs in one unit — assignments evaluated per usable draw — as the tolerance tightens. A hunt costs 1/p and rises without limit: from 2.22 at a tolerance of 1.2 to 357.14 at 0.18. A walk costs its autocorrelation time and barely moves. The two cross at a tolerance of 0.190 at one swap and 0.195 at eight — the whole family of proposal sizes crosses inside a band of about two hundredths, because where the crossing is, the large proposal has already lost its advantage. A multi-swap proposal is worth a factor of 5.65 in the regime where the walk should not be used at all.

What a block may vary

Where the gain is, and where the decision is

A bigger proposal is worth a factor of six at a loose tolerance and nothing at a tight one. The tolerances where it helps are the ones where a hunt costs two evaluations a draw, and the crossing barely moves.

A normal mean with a flat prior — one interval, two readings. Both intervals are [1.878, 3.446]. The frequentist reading is that the procedure captures the truth 95% of the time; the Bayesian reading is that the parameter is in this interval with probability 0.95. The endpoints are identical to machine precision.

The prior, doing visible work

Where the two schools agree

With a flat prior on a normal mean, the credible interval and the confidence interval are the same interval, endpoint for endpoint. Knowing exactly when that stops being true is more useful than either camp's general argument.

Forty O'Brien–Fleming trials at a true effect of 0.16, with the boundary written as an effect. The dashed line is the smallest effect a trial can report and still stop at each look: 0.510 at 80 observations, 0.255 at 160 observations, 0.170 at 240 observations, 0.128 at 320 observations, 0.102 at 400 observations. The true effect is 0.16, so at 3 of the five looks a trial cannot stop without reporting more than it. 29 of these forty trials stop before the last look, each marked where it stopped.

Stopping rules

The effect a stopped trial reports

An O'Brien–Fleming trial at 88.45% power holds its error rate exactly and reports an effect 9.6% too large on average. The 11.39% of trials that stop at the second look report 1.83 times the truth, the ones that cross at the last look report 0.80 times it, and pooling every trial by its size gives the truth back to the last digit.

What a fixed-width interval covers, by the number of blocks the trial ran before it stopped. Two thousand runs of each rule, the modelled weighting, a promise of 0.34. Reading its report: 4–8 blocks, 22.3% of runs, 78.2%; 9–12 blocks, 16.6% of runs, 90.4%; 13–16 blocks, 18.4% of runs, 96.2%; 17–20 blocks, 17.4% of runs, 96.0%; 21–28 blocks, 17.9% of runs, 96.4%; 29–36 blocks, 7.4% of runs, 99.3% — 91.45% overall. Reading the arms: 4–8 blocks, 0.0%, none; 9–12 blocks, 0.9%, 94.4%; 13–16 blocks, 30.4%, 95.6%; 17–20 blocks, 50.0%, 93.9%; 21–28 blocks, 18.0%, 94.4%; 29–36 blocks, 0.7%, 92.3% — 94.50% overall.

When a fixed width is reached

The trials that stopped early

A fixed-width trial that stops when its own interval is short enough covers 91.45% — an average of 78.2% among the 22.3% of runs that stop within eight blocks and 96% to 99% among those that run longer. Widening every interval by 17.1% brings the average to 95% and leaves the early stops at 85.6%, while 92.8% of runs now report an interval wider than the width they promised. Even doubling every interval leaves the early stops short.

What stratifying on a mediator estimates, and what it does not. The direct effect stays at -0.109 throughout, because it is defined with the mediator held fixed. The total effect runs from -0.218 to 0.027 and crosses zero at 1.5. On 15% of the sweep the two have opposite signs, and both are correct answers.

When the stratified answer and the pooled one disagree

Conditioning on what the treatment caused

When the grouping variable lies on the path from treatment to outcome, the stratified answer is the direct effect and the aggregate is the total effect. Both are correct. Over 15% of a sweep of the indirect path they have opposite signs, and no arithmetic on the table says which question was being asked.

How much of a normal outcome's information survives cutting it into two, by where the cut is. For a small shift, a cut at the mean keeps 63.7% of the information, so the trial needs 1.57 times the sample. A cut at the top tenth keeps 34.2% and needs 2.92 times; a cut two standard deviations out keeps 13.1%.

What a sample-size calculation was given

An outcome cut in two

Replacing a measured outcome with whether it crossed a threshold keeps 63.7% of the information when the cut is at the mean, 34.2% at the top tenth and 13.1% two standard deviations out. A trial that needs 63 patients per arm on the measured outcome needs 102 cut at the mean and 185 cut at one and a half standard deviations. The responder rates that result read as a share of patients who respond — 50.0% against 69.1% — when every patient moved by the same amount; and a cut chosen after looking turns a 5% test into a 17.7% one.

What a schedule is allowed to read, and what happens when it reads more. The construction allows the block sizes to be anything at all as long as they are functions of the within-block contrasts, which are independent of every block mean. A schedule that shrinks the block whenever the between-block spread is running above what the contrasts say is a direct attempt to hold down the quantity the interval will be built from, and it succeeds: the estimate lands at 0.8373σ² against the honest 0.9831, and the coverage goes with it. Growing the block on the same spread instead pushes the other way and over-covers — which is not a repair, it is the same violation with the sign reversed, and the level is no longer a property of the procedure at all.

The block size as a schedule

A schedule that reads the mean

The block sizes may be anything at all provided they are functions of the contrasts. Two natural schedules break that, in opposite directions — and the most natural mistake of the three is not a schedule at all but a stopping rule, at 86.87% coverage and fewer observations.

Where the set stops being one set. How many of the 3,432 equal splits of fourteen units a balancing rule admits, as the tolerance tightens, with the number of components single swaps leave it in. The set falls from 886 to 84 assignments, and somewhere in that fall it stops being connected: at 0.8 it is in 2 pieces and every assignment's complement is in the other one. Nothing about the rule changes at that point and nothing a chain reports changes either, which is the whole difficulty — the acceptance rate, the stationary distribution and the detailed balance are all in order on both sides of it.

The diagnostic after the trial

Before the trial and after

The same diagnostic run at two moments answers two different questions. Before, a positive verdict changes the design. After, it changes which number gets reported — and only for the numbers the defect can reach.

What a guesser gets, and what a guesser gets for nothing. 600 trials of 150 patients under a fully deterministic rule, with an investigator who knows the rule, the factors and every assignment so far. The guess rate falls with the number of arms — 87.8% at 2, 86.2% at 3, 81.0% at 4 — which reads like a trial getting safer and is not: what a guesser can trade on is the excess over the 50%, 33%, 25% they would get by naming an arm at random, and that goes the other way, from 1.76× chance at two arms to 3.24× at 4. The gap a guesser manufactures between the best and worst arm under a true null is 0.759, 0.733, 0.711 standard deviations — nearly unchanged.

More arms than two

Guessing one arm in three

A balancing rule is guessable because it is balancing. With three arms the next assignment is worked out less often than with two — and by more, relative to what a guesser gets for nothing, and the damage they can do is almost unchanged.

A bias against a variance, with the answer in between. How wrong one sample's reference distribution is, split into the two things it is wrong by. Sharing the multiplier over more rows keeps more of the dependence and closes the bias from 1.688 to 0.835; every row it is shared over also removes an independent sign from the 101 the sample started with, and the spread of the resulting quantile rises from 1.307 to 2.172. The distance a practitioner with one sample is actually exposed to is the two together, and it is smallest at ℓ = 5.

Scoring a search without spending data

How long a block a multiplier shares

Sharing a sign over more rows keeps more of the dependence and leaves fewer independent signs to build a distribution from. The bias falls from 1.6885 to 0.8479 and the spread rises from 1.3073 to 2.1716, and the rejection rate walks straight through its nominal level on the way from 11.3% to 1.3%.

What each analysis does at a true null, by shape. Four analyses of the same trials — 500 of them at each shape, 120 units, assigned by the rule that reads the covariate. Every rejection is false. The unadjusted analysis is the one that moves: 1.60% against a linear outcome, where the design removed a great deal that the standard error still prices, and 5.20% against a quadratic, where it removed nothing and the standard error is right. Adjusting holds the level in all three columns, and so does the design's own reference distribution, which needs to be told the rule and nothing else.

The shape the covariate enters by

The analysis and the shape

An unadjusted analysis after a rule that read the covariate is too cautious — by a third against a linear outcome, by nothing at all against a quadratic. And an adjustment for the wrong function recovers almost none of the precision the right one would.

What a positive test means, sensitivity 90%, specificity 95%. At a prevalence of one in a thousand, 98 of every hundred positives are false. At one in 10, 33 are. The test has not changed.

The prior, doing visible work

The base rate was always Bayes

The screening arithmetic everybody finds counter-intuitive is a posterior update with a prior of one in a thousand. Naming it that way turns a famous puzzle into an instance of a rule, and makes the sequential version obvious.

What the corner costs when the table is full of hopeless candidates. The benchmark holds two predictors, one of which is worth 1; a third predictor, worth the amount on the horizontal axis, is held only by candidates the benchmark does not contain. At the left the null is true and both procedures hold their level. To the right there is a genuinely better candidate, and the uncorrected reality check finds it 2.7% of the time while the same test with the clearly bad columns recentred finds it 51.0% of the time. The columns doing the damage are the ones nobody would have looked at twice: they are so far behind that they cannot win, and calibrating as though they might is what makes the test blind.

A search with no fixed point

The corner the test is calibrated at

"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.

A design that is right once, against one that is never wrong by much. Two designs for the same two-parameter model, scored at every true value of K across a range of 16-fold. The local design is the two-point optimum for a guess of K = 1: it reaches 100% there and 66.7% at the worst point of the range. The hedged design maximises the average of log|M| over a prior spanning a factor of 4 either side, uses 3 settings rather than two, and is never below 75.4%. What it costs is 10.4 points at the one value the local design was built for — which is the whole trade, and it is only available to somebody willing to say how wrong the guess might be.

The criterion, and what it assumes

The design that hedges

A locally optimal design is right at one value of the unknown and 23.9% efficient at the edge of a sixteenfold range. Averaging the criterion over a prior instead buys the worst case back to 56.3% — and buys it by adding support points, at spreads the arithmetic decides rather than the experimenter — a third setting at a factor of 3.36 and a fourth at 8.86.

Four intervals at one stopping time. 3,500 experiments under the sequential rule with a first stage of 5 and a required half-width of 0.4, which is a demand that knowing σ would meet with 24.0 observations and which the rule meets with 20.8. The rule's own interval covers 90.3%. Replacing the fixed width by a t interval on the same data gives 92.0%. Keeping the rule's own random sample size and drawing a fresh sample of that size gives 89.8% at the fixed width and 95.5% for a t interval — so the sample size being random costs nothing, and the sample size being chosen by the data the interval is built from costs the rest. The spread estimated at the stopping moment is 17.1% below the truth, which is the same fact one level down.

What the design is asked to guarantee

The interval after a stop it chose

A rule that stops when the estimated precision is good enough stops on the samples whose estimate was small. Its interval covers 90% and claims 95%, and a fresh sample of the same random size covers 95.4%.

What the interval covers once the order is chosen as well. 1200 series of 40 observations from an AR(1) with φ = 0.7, at each horizon, on one set of seeds. The upper line is the interval computed at the true parameters — it covers 95.5% on average, which is the check that σ²Σψ² is the right formula rather than a claim about anything a forecaster can do. The lower line is the same formula fed σ̂² and φ̂: 93.6% at one step and 90.8% at 6. The third line chooses the order by AIC from the same data before computing the interval, which costs a further 0.8 points at h = 6.

The observation that has not happened

The interval after the choice

Estimating the coefficients of a known model costs a 95% forecast interval about two points of coverage. Choosing which coefficients to estimate, from the same forty observations, costs another four and a half — so the step nobody records in the output is the more expensive of the two.

The allocations this trial could have made, and the ones it could not. One 120-patient trial allocated by minimisation at p = 1, re-randomised 399 times. No outcome is redrawn anywhere in this figure: each re-randomisation runs the rule again over the same patients in the same order with the same recorded factors, so what is drawn is the set of experiments that could have happened. The bars are that set; the outline is what shuffling the labels gives, which is the reference distribution of a coin and is what every off-the-shelf permutation routine assumes. The coin's is wider — its 5% point is 1.95 against the rule's 1.09 — because a coin's allocations are less balanced and a less balanced allocation gives a larger statistic. Reading this trial against it makes the test conservative rather than anti-conservative, which is the opposite error from the outcome-adaptive case and for the same structural reason.

Balancing on what was recorded first

The reference the covariates supply

Hold the outcomes fixed, re-run the rule that assigned them, count. The same construction cost nineteen points of power in the adaptive field, because its rule chased outcomes and its critical value depended on a rate nobody has. Here the rule reads only what was recorded before anything happened, and the same unadjusted statistic goes from 20.3% power to 55.0% by being read against the right distribution.

A thin enough set is not one set. Every admissible set of 14 units this table can enumerate, by how much of the assignment space it admits and how many pieces it falls into under single swaps. A walk is uniform on the piece it starts in and never leaves it. The pieces are not fragments: at 522 admissible assignments the set splits into 3 halves of exactly 520 each, and every assignment's complement is in the other half — no sequence of admissible single swaps takes an assignment to its own mirror image. Two-swap proposals reconnect four of the six disconnected sets here — the two they do not are the thinnest, where a two-unit move rarely lands anywhere admissible either — which makes a bigger proposal a correctness repair rather than the speed dial it was measured as.

What a dictionary buys and what it costs

The walk that cannot cross

A thin enough admissible set is not one set. It splits into an assignment and its mirror image, no sequence of admissible single swaps joins them, and the walk that samples it is uniform on half the reference distribution for ever.

A design that assumes less

What a design chosen from the data costs

Two earlier measurements found what happens when a rule reads the data, and the error rate broke both times. A design that reads the data to decide where to put its runs breaks nothing — and the control that proves it also finds what the real shortfall is.

When the set is too large to walk

What a reference distribution costs to sample

A randomisation test on a trial too large to enumerate has to sample its reference distribution, at 1/p attempts per draw and a p-value resolved to 1/(B + 1). Six constraints cost 9,878 attempts per thousand draws, and a thousand draws resolve p to 9.99·10⁻⁴ and not one digit finer.

Free until the sums stop seeing what the differences see. Coverage with and without the block sums pooled into the interval's variance estimate. With one effect and one level they are free. With an effect that varies between blocks they are still free, because a block's sum picks that variation up exactly as its difference does. With a level that varies they make the interval 37% wider and conservative. And where the effect falls as the level rises — a ceiling, and not an exotic thing to suppose — the sums carry none of the between-block variation while the differences carry all of it, the pooled estimate is short, and the interval that uses it covers 88.75% on a width 20% narrower than the honest one.

A promise about two arms

What a two-arm rule may not pool

A spread computed "within the block" without the arm label carries a share of the effect, so the trial runs 173 observations at a null and 282 at an effect of 1.5. The stopping rule is reading the thing it exists to measure, and the phrase that produced it is one word long.

The cheap repair needs a number nobody has. The obvious alternative to re-randomising is to simulate the design under its null once and use the critical value that comes out — which is what the arm-dropping design does, where the critical value has to be solved for and is 2.313. It does not transfer here. The rule chases outcomes, so how imbalanced the allocation gets depends on how often anything succeeds, and the critical value moves from 1.668 at a success rate of 0.05 to 2.718 at 0.8. Calibrated at 0.3 and used at 0.8 the test's real size is 12.4%; used at 0.05 it is 0.12%. The randomisation test needs none of this, because it conditions on the outcomes that happened rather than on a rate they were supposed to come from.

The reference distribution the design supplies

What the exactness buys

Against a z test calibrated to reject exactly 5% of true nulls on this design, the randomisation test loses nineteen points of power. What it buys is that the calibration needs the success rate — which moves the critical value from 1.668 to 2.718 and is the quantity the trial was run to find out.

Where the constraints exhaust the randomisation. At 16 units there are 12,870 equal splits, so the ones meeting a stated tolerance can be counted rather than estimated. With each of the first k standardised imbalances required to be within 0.4 of a coin's own spread, the admissible count runs 3874 → 1006 → 314 → 0 → 0 → 0 — and at 4 functions there is no admissible assignment at all. The count is the number of distinct answers a randomisation test can give: at 3 functions its finest attainable p-value is 1 in 314. Balance improves with every constraint and the reference distribution shrinks with it, and the two run out at different rates.

Choosing what the rule reads

When the constraints run out

Every function added to a basis is a constraint the assignment has to satisfy with the same units. At sixteen units and a stated tolerance the admissible assignments run 3,874, then 1,006, then 314, then none — and the count is exact, because the assignment space is finite.

Four numbers, and only one of them is the question. Four quantities in a population where the effect is not the same for everybody: 40.0% compliers, 25.0% always-takers and 35.0% never-takers, with the compliers carrying an effect 1.0000 larger than everybody else's. The population average effect is 0.5000. The compliers' average effect is 1.1000. A valid instrument converges on 1.1000 — the second of those, not the first, and the two differ by 0.6000. Comparing the treated with the untreated as they stand gives 1.4182, wrong by 0.9182 in the same direction, because always-takers start 1.5000 above never-takers before any treatment happens. The instrument removes the selection and changes the question at the same time, and only one of those is reported.

A variable that moves one thing only

Whose effect it is

With a perfectly valid instrument and no violation of anything, the estimate converges on 1.1000 where the population average effect is 0.5000. The gap is exactly θ(1 − p_c), the always-takers and never-takers cancel out of both halves of the ratio, and five per cent defiers move the answer to 1.2667.

Two groups, a baseline and a follow-up, and nothing happening in between — baseline reliability 0.6. 600 units in two pre-existing groups whose true means are 1.00 apart, read once at baseline and once at follow-up, with no change for anybody. The two groups' mean changes are −0.075 and −0.032, so the change-score analysis reports a group difference of 0.043. The regression of follow-up on baseline and group reports 0.409, against a closed form of (1 − λ) × 1.00 = 0.400: at any one baseline reading the two groups' lines sit that far apart, because each group's units regress towards their own group's mean. The pooled slope in this sample is 0.614, the baseline's reliability.

Reversals that are not errors

Two analyses of one baseline

Two groups read at baseline and again at follow-up, with no change for anybody. Subtracting the baseline reports a group difference of −0.0014 and adjusting for it reports 0.4008 — and each analysis is exactly right about one reason the groups started apart and wrong by 0.40 about the other.

Two worlds, one Kaplan–Meier curve, two truths. World A gives each subject a frailty with mean one and variance 1, and multiplies both its event hazard (0.35) and its dropout hazard (0.5) by it, so the subjects likeliest to leave are the ones likeliest to fail. World B has independent event and dropout times whose hazards are world A's crude hazards. Kaplan–Meier over 1000 studies of 400 gives the same curve from both — 0.7750 and 0.7763 at t = 1; 0.6627 and 0.6641 at t = 2; 0.5924 and 0.5932 at t = 3; 0.5047 and 0.5042 at t = 5 — and that curve is world B's truth, 0.5052 at t = 5. World A's truth is 0.3636 there. The dashed lines are the two bounds that assume nothing, from every dropout failing on leaving (0.1905 at t = 5) to none ever failing (0.6667).

When the data stops early

A dropout the data cannot see

Two worlds leave the same record to the last detail a study can write down — the same times, the same share ending in the event, the same share leaving first — and a log-rank test between them rejects at its own 5% level at every sample size from a hundred to sixteen hundred. Kaplan–Meier converges on 0.5052 at t = 5 from both. The truth is 0.5052 in one and 0.3636 in the other, and what is left to argue about is where between two bounds to stand.

Ordered stagewise: the outcomes at least as extreme as stopping at 160 observations with z = 3.3. Each column is one look of an O'Brien–Fleming trial; above the boundary a trial stops there. Highlighted are the outcomes that count as at least as extreme as the observed one when outcomes are ordered stagewise: at 80, z ≥ 4.56 (probability 2.54 × 10⁻⁶ with no effect); at 160, z ≥ 3.30 (probability 4.82 × 10⁻⁴ with no effect); at 240, none; at 320, none; at 400, none. The two-sided p-value is 9.69 × 10⁻⁴.

Stopping rules

The outcomes a trial could have stopped with

A trial that stops at its second look with z = 3.3 has a two-sided p-value of 0.000969, 0.000987, 0.00187 or 0.0421, depending on how the outcomes it could have stopped with are ordered. One of the four orderings does not change when the looks the trial never reached are replanned, and the same one gives a trial that ran to the end with z = 6 a p-value of 0.0256.

The fixed-width trial's coverage when the outcomes are not normal, for both stopping rules. normal: stopping on the arms 94.05% after 18.1 blocks, on the report 89.95%; log-normal, skewness 0.95: stopping on the arms 94.70% after 18.5 blocks, on the report 90.80%; log-normal, skewness 2.26: stopping on the arms 94.15% after 19.3 blocks, on the report 90.25%; log-normal, skewness 4.75: stopping on the arms 94.45% after 18.7 blocks, on the report 90.50%; t, five degrees of freedom: stopping on the arms 94.35% after 18.3 blocks, on the report 90.30%; skewness 4.75, arm A only: stopping on the arms 93.80% after 26.0 blocks, on the report 89.90%; skewness 4.75, arm B only: stopping on the arms 93.60% after 14.2 blocks, on the report 89.95%; equal variances, normal: stopping on the arms 94.75% after 11.4 blocks, on the report 90.90%; equal variances, skewness 4.75: stopping on the arms 94.05% after 11.1 blocks, on the report 92.00%.

When a fixed width is reached

A width rule on skewed outcomes

The blinded fixed-width rule rests on a within-arm spread being independent of the arm means, which only normal samples guarantee. On outcomes with a skewness of 4.75 the independence fails and the overall coverage barely notices — 93.60% to 94.70% across every shape counted, against 94.05% on normal outcomes. What skew moves is the runs that stop by twelve blocks, which cover about 90% with the skew in one arm, and the trial's length: a variance ratio corrected on normal theory lengthens it from 18.1 blocks to 26.0 with the skew in the first arm and shortens it to 14.2 with the skew in the second.

A discrepancy of 4 σ at the top of the range: one bad run, or a line that bends there, in two designs of twelve runs. The same twelve errors in every panel. With one run at the top (upper panels) a bad observation and a bent line produce identical data, point for point. With three runs at the top (lower panels) the bad observation leaves one run far from its two neighbours and the bent line moves all three together. The dashed line is the true line; the grey line is the least-squares fit.

What a diagnostic plot is showing

Three runs at the end of the line

With one run at the top of a regression's range, a bad observation there and a line that bends there produce data with exactly the same distribution, so no residual, influence measure or test can say which happened. Move six of twelve runs to the ends, three at each, and a discrepancy of four σ is named correctly as a bad run 84.1% of the time and as a bend 91.6% — while the slope's standard error falls from 0.0836 to 0.0709. The design, not the diagnostic, decides whether the question has an answer.

The ratio of the arms' hazards over time, when treatment halves every patient's hazard. Every patient's hazard ratio is 0.5. With two risk groups — half at 0.1 a year, half at 1 — the ratio of the arms' hazards rises from 0.5 to 0.929 at 3.5 years and falls back towards 0.5. With a gamma frailty of variance 1 it rises steadily, to 0.857 at ten years.

When the stratified answer and the pooled one disagree

Every patient at one half

A randomised trial in which the treatment halves every patient's hazard, and patients differ only in their baseline risk. The unadjusted Cox hazard ratio is 0.524 after six months of follow-up, 0.635 after three years and 0.679 after seven, and the ratio of the arms' hazards at a single moment climbs from 0.50 to 0.929 and falls back. Nothing is confounded and nothing is wrong. Survival removes high-risk patients from the control arm faster, and a hazard ratio compares whoever is left.

How much naming more analyses in advance is worth, at an effect of 2 standard errors, for four analysts' beliefs. Twenty analyses correlated at 0.6, one of which holds the effect. Each curve is an analyst whose belief falls by a fixed ratio from one choice to the next: 0.2, 0.5, 0.7, 0.9. The best number to name is 1, 3, 5, 15 respectively, and each best set holds between 80% and 90% of the analyst's belief. Every curve ends at 22.7%, the power of correcting for all twenty.

The analyses that were available and not run

Naming a handful in advance

A preregistration that names three of twenty analyses sits between naming one and naming none, and whether it beats both depends on one thing: how much of the analyst's belief the three hold. An analyst who is 50% sure of a first choice, and whose belief halves down the list, gets 34.8% expected power from naming three, 28.3% from naming one and 22.7% from exploring all twenty. Spread the same 50% evenly over the rest and naming three is the worst of the three plans. The rule for adding one more to the list needs no belief at all: the next analysis must be at least 20.8% as likely as the first to be worth naming.

The bottom ten of a league table of a hundred when larger groups do better, size–effect slope 0.6. Groups of 4 to 400, true effects rising with log size at a slope of 0.6 population widths. The true bottom ten is 80.6% small groups. Ranked by their own means the bottom ten holds 4.28 of the true ten; pooled to one centre, 3.19, with 42.6% small groups; pooled to a centre that rises with size, 4.35, with 83.0%.

What partial pooling does to one group, to the set, and to a ranking

The small groups one centre protects

In a league table of a hundred groups where larger groups do better on average, pooling every group towards one centre overstates the small groups by half a population width — 0.501 — because it pulls them towards an average they are not part of. The top ten barely notices. The bottom ten, the list that flags poor performers, holds 3.19 of the true bottom ten when pooled to one centre and 4.28 when the groups are ranked by their own noisy means. Pooling towards a centre that rises with size restores 4.35 and puts the small groups back at the bottom where 80.6% of the true bottom ten sit.

One pooled ratio, two periods. The calibration factor of weights built from one variance ratio pooled across the trial — the factor by which the interval's own scale estimate overstates the estimate's variance — when arm B's ratio is F times higher in one kind of period than the other, closed form on the corner's block compositions. With the small A-heavy blocks the noisier, the factor rises: 1.00 at F = 1, 1.50 at F = 1.5, 2.07 at F = 2, 3.12 at F = 3, 3.91 at F = 4, 4.86 at F = 6, 5.44 at F = 8. With the large B-heavy blocks the noisier, it falls: 1.00 at F = 1, 0.74 at F = 1.5, 0.65 at F = 2, 0.58 at F = 3, 0.55 at F = 4, 0.53 at F = 6, 0.53 at F = 8. A ratio estimated per period gives weights that are the inverse variances again, and its factor is exactly 1 at every F.

The weights the corner needs

A ratio that changes with the period

The corner's exact interval needs one variance ratio, and the corner's own alternating allocation makes two kinds of period that need not share one. When they differ the pooled ratio is precise and wrong: at a fourfold difference the interval covers 100.0% one way round and 85.1% the other. A ratio estimated per period is exact again, blind to the effect, and costs under a tenth of the width — and a ratio that merely wanders at random costs width and no coverage at all.

The line is the sample, and the sample is the finding. 900 draws of two independent standard normal causes, with the 453 of them past a threshold of 0.00 marked and the 447 that fall short left pale. In the population the two are independent by construction. Inside the selected sample the correlation is -0.4669 in closed form and -0.5050 counted on these 453 rows, and the least-squares line through them has a slope of -0.545. The mechanism is visible in the picture rather than argued: the threshold removes one corner of the cloud, and a cloud with a corner missing is a cloud whose two coordinates carry information about each other.

What conditioning on a variable does

The sample is a condition

Two independent standard normals, selected on their sum exceeding its median, read a correlation of exactly −1/(π − 1) = −0.4669 inside the sample. Nothing is measured badly and nothing is missing — and both halves of that split read it, in the same direction, while the population containing both reads zero.

$The damage does not stay in the term that was left out. Where each coefficient lands when the model that fills the missing outcomes and the model that analyses them disagree, over 1500 studies of 200 rows at 35.0% missing and 20 imputations. An imputer that omits a covariate the analysis fits attenuates that covariate's coefficient by exactly the missing fraction — -0.1405 counted against a closed -0.1400 — and pushes the coefficient it did impute on the other way by exactly the product of the omitted coefficient, the covariates' correlation and the missing fraction: 0.0402 counted against 0.0420. Both closed forms come out of the same two-by-two solve. Matching models leave both alone, and so does an imputer that knows more than the analysis.$ The value that is not there

An imputation model the analysis does not contain

A model that fills the gaps without a covariate the analysis fits attenuates that covariate's coefficient by exactly the missing share, 0.4 to 0.26, and moves the one it did carry by exactly γρf, 0.6 to 0.642. The reverse case is supposed to inflate the interval, and at four strengths of the extra knowledge it does not.

The price of insurance is noise, not coverage. What a robust standard error costs under a constant error variance — the case where the model-based one is exactly right — at five sample sizes over 20000 draws at the small end. It is not coverage: the leave-one-out interval read against a t on n − 2 covers 95.52% at 20 rows against the model-based 95.06%. It is a wider interval, by a factor of 1.0689 at 20 rows falling to 1.0049 at 250, and it is a variance estimate 2.57 times as variable at 20 rows and 1.85 times at 250 — against a denominator that is exactly V²·2/(n − 2), so only the numerator is counted. The uncorrected estimate is the cheaper of the two at small samples and the dearer at large: 1.31 against 1.76.

A standard error for a model that is wrong

Right for the wrong reason

A robust standard error costs no coverage where the risk is absent — 95.52% against 95.06% at twenty rows. It costs a 6.89% wider interval and a variance estimate 2.572 times as variable, and the pre-test that would avoid paying recovers 15.9% of what the insurance is worth.

A cohort screened once, the top tenth enrolled, and followed up with nothing given — correlation 0.6. 2000 people read once at screening and once at follow-up, with a test–retest correlation of 0.6 and no treatment. The 215 above a cut at the top ten per cent of one reading (1.282 standard deviations) are enrolled. Their mean screening reading is 1.744 and their mean follow-up reading 0.982, a fall of 0.762 ± 0.055 with nothing done to anyone. The closed form for the fall is (1 − ρ) times the truncated-normal mean, (1 − 0.6) × 1.755 = 0.702.

Reversals that are not errors

The measurement that got them enrolled

Enrol the top tenth of one screening reading and give them nothing, and they fall by 0.702 standard deviations at follow-up. Measured from a fresh reading taken after enrolment they fall by nothing. Averaging ten screening readings still leaves 0.101, and it takes twenty-one to get under 0.05.

Twenty runs simulating an exactly 95% interval, checked every 250 replications. Each line is one run's running estimate; the dashed band is where the Wilson interval of the running estimate still contains 95%, and a run stops, marked, the first time it leaves the band. 8 of these twenty stop before 10,000 replications. The exact probability of stopping, from the recursion over the count, is 29.54%.

What makes it checkable

A simulation that stops when it looks settled

A simulation of an interval that covers exactly 95%, checked every 250 replications for a significant departure and stopped when it finds one, flags that correct interval on 29.54% of runs. Stopped instead as soon as its estimate reaches 95%, it reports an interval that covers 94% as meeting its level on 37.21% of runs. Stopped when the estimate stops moving, it reports the right number — and has quietly chosen to run about fifteen hundred replications.

Forty trials at a true effect of 0.16, under the rule "power at the trend < 10%". The upper line is the benefit boundary (4.56, 3.23, 2.63, 2.28, 2.04); the lower line is where the rule stops a trial for futility (0.40 at 80, 0.66 at 160, 0.95 at 240, 1.31 at 320). Of forty trials with a real effect, 29 cross for benefit and 11 are stopped for futility.

Stopping rules

A boundary for giving up

Adding "stop if z is below zero" to an O'Brien–Fleming trial costs 5.20 points of power at the effect it was designed for and halves the observations a trial with no effect uses. Stopping when conditional power at the observed trend falls under 10% costs 13.23 points and stops 21.28% of trials with a real effect. Making that rule binding lowers the benefit boundary from 2.040 to 1.901, and a binding rule that is then ignored rejects a true null 3.523% of the time instead of 2.5%.

The damage and the warning, against the same dial. Two readings at each persistence. In the darker colour, how often a regression between two independent series of 200 steps is called significant at 5%: 4.9% at φ = 0, 34.2% at 0.8, 52.4% at 0.9, 83.4% at a unit root. In the lighter, how often the standard unit-root test refuses a unit root on one of those series — the chance the analyst is told the series is stationary and may be regressed: 87.2% at φ = 0.9 and 31.9% at 0.95. At φ = 0.9 both are high at once, which is a correct diagnostic licensing a regression that is wrong half the time.

When the observations repeat each other

The cliff that is a slope

A regression between two independent series is called significant 4.9% of the time at no persistence, 52.4% at a lag-one correlation of 0.9, and 83.4% at a unit root. The rule the field offers asks whether the last of those holds, and at 0.9 the unit-root test correctly refuses one 87.2% of the time.

Thirty winners of twenty analyses, effect 2 standard errors in one of them, with two intervals each. Each row is the largest of twenty statistics correlated at 0.6, reported because it cleared the family-wise threshold of 2.846. The thin grey bar is the ordinary interval, z ± 1.96; the heavy bar is the interval conditional on the statistic having won; the tick is the true mean of the analysis that won. Of these thirty the ordinary interval misses 1 and the conditional one 2; over 4,468 reported winners they cover 76.0% and 94.9%.

The analyses that were available and not run

An interval for the winner

The largest of twenty correlated analyses, reported because it cleared the family-wise threshold, carries an ordinary 95% interval that covers its true mean 75.96% of the time at an effect of two standard errors and never at all when there is no effect. An interval built on the fact that it won covers 94.90% — and it is 2.48 times as wide, excludes zero for only 14.10% of the winners, and has no lower limit at all for 14.32% of them. It restores the promise by declining to say most of what the ordinary interval said.

What a diagnostic plot is showing

A number for the shape

Five one-number summaries of a quantile plot's departure from its line, each a correct 5% test on forty normal observations, and no two agree about what matters. Straightness catches a skewed source 83% of the time and light tails 31%; kurtosis catches light tails 72% and skewness 0.2%. Reading all five and reporting whichever looks bad rejects 13.9% of genuinely normal samples. Correcting that search to 5% costs almost half the power against the departure it would have caught best — and a line-up of twenty panels gets the same correction for nothing.

How often a randomised trial of eighty shows a Simpson reversal on some baseline variable, against how many were tabulated. The treatment helps every patient equally. On one nominated, strongly prognostic variable the reversal appears in 4.1% of trials. Tabulating 2, 5, 10, 20, 50, 100 variables of mixed prognostic value, some variable reverses in 4.1%, 6.5%, 10.8%, 16.1%, 26.1%, 34.1% of trials; stratifying the randomisation on the most prognostic one gives 0.2%, 2.9%, 7.7%, 12.2%, 23.4%, 31.9%.

When the stratified answer and the pooled one disagree

Twenty variables nobody stratified on

A randomised trial of eighty patients in which the treatment helps every patient by the same six points shows Simpson's reversal on one nominated, strongly prognostic baseline variable in 4.07% of trials. Tabulate twenty baseline variables of mixed prognostic value and some variable reverses in 16.07% of trials; a hundred, in 34.13%. Stratifying the randomisation on the strongest variable removes its reversal and leaves 12.17% and 31.93%: the protection is exactly as wide as the list it was given.

Two tests, a threshold, and the rate they are read against

A threshold that jumps

When a test's scores are equally spread in the healthy and the diseased, the harm-minimising threshold slides smoothly as the prevalence changes. Make the diseased scores twice as spread — the ordinary shape of a group that mixes mild and severe cases — and keep the published 90/95 pair, and the best threshold flags 81.1% of healthy people at a prevalence just under 25.7% and every healthy person just above it. At three times the spread it jumps from 45.7% to everyone at 13.0%. A threshold that jumps cannot be given as a formula; it has to be given with the second-best point beside it.

How often a trial that borrows its control from an earlier one declares a treatment with no effect a success, by the earlier control's drift. Fifty patients an arm, an earlier control of two hundred, a one-sided 2.5% threshold. Without borrowing the rate is 2.5% at every drift. Pooling outright reaches 73.1% at a drift of half a standard deviation. A half-normal prior of scale 0.05 on the between-trial spread peaks at 35.2% and is still at 31.8% at a drift of 1.5; a half-Cauchy of the same scale peaks at 11.0% near a drift of 0.4 and falls back to 3.4%.

The spread, and its own uncertainty

A control borrowed from the last trial

A trial of fifty patients an arm can borrow its control group from an earlier trial of two hundred by treating the two control means as draws from one population with a spread between trials. With two trials there is one difference to learn that spread from, so the prior on it decides how much is borrowed — and its tail decides whether the borrowing stops when the two trials disagree. A half-normal prior of scale 0.05 buys 17.4 points of power and, at a drift of 1.5 standard deviations, still declares a treatment with no effect a success 31.8% of the time. A half-Cauchy of the same scale buys 13.6 points, peaks at 11.0% and falls back to 3.4%.

How far each estimate of the small groups sits from their truth, when the size relation is flattening below the median size. The small groups' average true effect is −0.259. Their estimates are off on average by −0.012 (own means), +0.276 (one fitted centre), −0.173 (fitted line in log size), −0.025 (fitted quadratic in log size), −0.003 (the true relation).

What partial pooling does to one group, to the set, and to a ranking

The relation the table has to estimate

Pooling a league table's groups towards a centre that rises with size repaired the small groups' bias — when the size relation was known. Fitted from the table itself, a straight line in log size loses almost nothing: the small groups' error is 0.750 against 0.719 with the true relation. When the relation bends below the median size, the fitted line extrapolates the large groups' slope into the region where it is barely observed and misplaces the small groups by −0.173 if the relation flattens and +0.264 if it steepens — smaller than one centre's error, of either sign. A quadratic in log size removes nearly all of it, −0.025 and +0.014, and costs 0.025 in error when the relation was straight.

A lean, its reflection, and none. The mean error of each weighting's estimated difference at a fixed eighteen blocks, 2,000 trials a shape, with two standard errors. normal: the modelled ratio, every block 0.0031, the modelled ratio, each block left out 0.0028, the modelled ratio, cross-fitted halves 0.0023, the true ratios 0.0037; skewness 0.95: the modelled ratio, every block 0.0092, the modelled ratio, each block left out -0.0134, the modelled ratio, cross-fitted halves -0.0014, the true ratios 0.0002; skewness 2.26: the modelled ratio, every block 0.0166, the modelled ratio, each block left out -0.0298, the modelled ratio, cross-fitted halves -0.0042, the true ratios -0.0029; skewness 4.75: the modelled ratio, every block 0.0255, the modelled ratio, each block left out -0.0455, the modelled ratio, cross-fitted halves -0.0055, the true ratios -0.0051.

When a fixed width is reached

A weighting that does not read its own spreads

On skewed outcomes a block's spreads move with its means, so a weighting fitted to every block's spreads leans: +0.0255 at a skewness of 4.75. The obvious repair — fit each block's weight with that block left out — does not remove the lean. It reverses it and nearly doubles it, to −0.0455, because a weighted mean divides by the sum of the weights and every other block's weight still reads the one left out. Splitting the trial into two halves and weighting each by the other's fit does remove it: −0.0055, indistinguishable from the weighting told the true ratios, for a spread about a tenth wider.

What a guesser gets from less than everything. 1,000 three-arm trials of 150 patients under minimisation on three factors with the preferred arm taken with probability 1, and five investigators guessing each next assignment from different information. Knowing the rule, the factors and every assignment so far: 86.3%. Knowing everything about the patients at their own centre and nothing about the others, and applying the rule's own score to that: 52.8%. Knowing only how many of their own centre's patients went to each arm: 55.5%. Knowing only the trial's arm totals, which a drug-supply system shows anyone who reorders: 54.5%. A guess at random gets 33.3%. The cruder the information, the less the guesser gets — except that the rule's own score, applied to a partial record, does worse than simply counting.

More arms than two

A guesser who sees the totals

The bound on how guessable a three-arm minimisation is assumes an investigator who sees everything. One who sees only the arm totals — what a drug-supply screen shows — names the next arm 54.5% of the time against a chance 33.3%, and manufactures 45% of the full guesser's bias.

The block size as a schedule

How much of the forbidden spread a rule may read

A stopping rule that reads the spread its interval will report breaks the interval. Blend that spread into what the rule reads, a share at a time, and the cost is not proportional to the share — at a tight requirement a rule can read four fifths of it before anything is lost, and the last fifth costs nearly four points.

A third stage, between the two rules. Four rules for one requirement: a half-width of 0.4 at 95%, which knowing σ would meet with 24.0 observations, after a first stage of 5. Each bar is the mean number of observations as a multiple of that, and the note beside it is the coverage. The two-stage rule spends 2.03× and covers 95.42%, computed exactly. The three-stage rule — a second stage to half of what the first spread asks for, the spread re-read on everything, a third stage to the rest — spends 0.86× and covers 88.42%; given 8.50 more observations in every experiment's last stage it covers 95.00% on 1.20×. The rule that re-estimates after every observation spends 0.88× and covers 90.5% over 4,000 drawn experiments; it is the only row that has to be drawn, because it is the only one whose stopping point shares observations with the mean.

What the design is asked to guarantee

The fifth a third stage never reaches

A three-stage stopping rule keeps the two-stage rule's independence and the sequential rule's budget. With a first stage of five it covers 88.4% — worse than the rule it was meant to repair — because in a fifth of experiments the second stage buys nothing.

What a dictionary buys and what it costs

Two functions a covariate

Every constraint costs admissible assignments at the same rate, so the decision a trial faces is a budget: which two functions of each covariate. Over twenty-eight pairs the worst case never picks the mean. At a correlation of a half it picks the square with a cut at one, beating the mean and the square by sixteen hundredths of a point on a floor held by a shape no pair removes a tenth of — and giving up 41% of a linear outcome to do it. Over the 127 lists of shapes a designer could write, twenty-three different pairs are best somewhere.

The shape the covariate enters by

How many functions a trial can hold

A rule that reads more functions of one covariate protects more shapes, and every function it reads costs the shapes it already held: the imbalance it leaves in a linear outcome falls like one over the trial size, with a constant that runs from about 3 for one function to about 16 for six. At fifty units that cost eats the protection, and the worst case over four shapes sits between 37% and 42% of a coin's from two functions to six. At four hundred it falls from 42% to 6%. By six functions the worst shape, at every size, is one the rule is reading.

Four intervals for the effect, minimisation at p = 0.8, 120 patients. Under minimisation at p = 0.8, 120 patients, over 600 trials with an additive effect of 0.45 and 199 re-run allocations per trial, the average width of four 95% intervals for the effect, and how often each covers it: unadjusted t interval 1.229 wide, covering 99.7%; unadjusted, test inverted 0.816 wide, covering 95.3%; adjusted t interval 0.723 wide, covering 96.0%; adjusted, test inverted 0.722 wide, covering 96.2%.

Balancing on what was recorded first

The interval the test inverts to

A randomisation test of no effect becomes an interval for the effect by testing every value at once: subtract the hypothesised effect from the treated outcomes and ask the same question again. Because the rule reads only covariates, one set of re-run allocations answers every value. Under minimisation the unadjusted t interval covers 99.7% at a width of 1.229; the inverted test of the same unadjusted difference covers 95.3% at 0.816, a third narrower, with no model of the outcome. Under complete randomisation the two are the same interval — which is what says the third was the rule's.

What each stopping rule spends. The average number of observations the corner design takes before it stops, with the stopping rule reading pooled within-arm spreads (the upper bar) and each kind of period's own spreads (the lower), at a promised half-width of 0.3, 3,000 trials each. one ratio: 1,656 and 1,649; 2-fold, small blocks noisier: 1,020 and 1,015; 2-fold, large blocks noisier: 2,976 and 2,614; 4-fold, small blocks noisier: 751 and 669; 4-fold, large blocks noisier: 5,642 and 3,368; 8-fold, small blocks noisier: 711 and 490; 8-fold, large blocks noisier: 10,989 and 3,019. With one ratio the two rules are the same rule. With the large blocks noisier the pooled rule over-spends by a factor that grows with the difference, to 3.6 times at eightfold; with the small blocks noisier it over-spends less.

The weights the corner needs

A stopping rule that reads the periods

A fixed-width trial in the corner stops when its estimated precision reaches the promise, and it estimated that precision from spreads pooled across a trial whose two kinds of period need not share a variance ratio. Read each kind of period's own spreads instead and the trial stops on 3,019 observations where it took 10,989, at an eightfold difference, with the interval it reports still covering 95.1%. The stopping time reads nothing but within-arm spreads, per period now as well as pooled, and that is all the exactness needed.

$An estimate reported as a function of an assumption. What the slope really is, against a shift in the outcomes nobody saw — line from the closed form, dots counted over 2000 studies of 200 rows at 35.0% missing. Every point on this line produces exactly the same observed data, and the complete-case estimate is the flat line at 0.5996 regardless. The truth moves at -0.2845 per unit of shift, which is a function of the missingness model and the missing fraction and of nothing that can be estimated: across the swept range the true slope runs from 0.8845 to 0.3155, a span of 0.5691 against a value of 0.60 in the world where the shift is zero. Reporting the line is the honest form of the answer.$ The value that is not there

The mechanism the data cannot see

Two worlds produce identical covariates, identical patterns of what is recorded and identical recorded outcomes, to the last bit. Their true slopes are 0.6 and 0.315452, and the truth moves at 0.284548 per unit of an assumption nothing in the data can inform.

One treatment, a different hazard ratio at every follow-up. The hazard ratio a Cox model converges to, found as the root of its expected score by numerical integration, as the trial runs longer; dropout at 0.1 throughout. The proportional treatment reads 0.5 at every τ. The waning treatment reads 0.5000 at τ = 1, 0.6362 at τ = 3 and 0.7890 at τ = 8 — the same two arms, the same effect in the same first year, and a number that drifts towards one as later, effect-free events are added to the average. The dots are the mean of 400 Cox fits with 400 subjects an arm: 0.5014 at τ = 1, 0.7020 at τ = 2, 0.7635 at τ = 3, 0.8034 at τ = 5, 0.8208 at τ = 8. The crossing treatment reads 0.3429 at τ = 1, exactly 1 at τ = 3 by construction, and 1.0611 at τ = 8: beneficial, null or harmful according to when the trial stopped.

When the data stops early

The hazard ratio the follow-up chose

A treatment that halves the hazard for one year and then does nothing has a Cox hazard ratio of 0.5000 if the trial stops at one year, 0.7617 at three and 0.8194 at eight. Nothing about the treatment differs between those numbers. When hazards are not proportional the hazard ratio is an average, and the length of follow-up and the dropout rate choose its weights.

Three ways to reject with two studies, drawn where the two z statistics live. Two one-sided studies, each summarised by its z statistic. Fisher's combination rejects outside a curve that runs parallel to both axes, so one study past z = 2.378 decides it alone; Stouffer's rejects above the straight line z₁ + z₂ = 2.326; Tippett's rejects when either z passes 1.955. Each region holds exactly 5% of the standard bivariate normal — Fisher's in closed form, e^(−c/2)(1 + c/2) at c = 9.488 — and of 100,000 counted null pairs they catch 4.95%, 4.88% and 5.04%. Two alternatives carry the same Stouffer evidence: one study at 2.326 and the other at nothing, where the powers are 62.7%, 50.0% and 65.4%; and both at 1.163, where they are 47.7%, 50.0% and 38.3%.

Tests, and the second number

Two ways to combine p-values

Fisher's and Stouffer's combinations are both exactly right when every null is true, for the single reason that each p-value is flat. Under a real effect they disagree about which evidence counts: with Stouffer held at 50% power across ten studies, Fisher is the more powerful while the signal sits in six or fewer of them and the less powerful from seven.

The chance of crossing later from each interim z, under six schedules with O'Brien–Fleming-type spending boundaries. Exact. At an interim |z| of 1.0: end only 3.64%, +0.6 3.41%, +0.75 3.20%, +0.9 3.41%, every 0.125 3.00%, every 0.05 2.88%. At 2.5: end only 38.30%, +0.6 45.80%, +0.75 45.36%, +0.9 41.70%, every 0.125 50.08%, every 0.05 53.24%. The heavy line is the largest of the six at each z.

Stopping rules

A look the trend asked for

Under an O'Brien–Fleming-type spending function, every schedule of looks fixed in advance spends exactly 5.0000%. A committee that adds a look at three quarters of the trial whenever the interim z is 1.5 or more spends 5.2323% — 5.315% counted over a hundred thousand trials — and the most a committee choosing among six schedules could spend is 5.4390%.

Twenty hypotheses tested in a declared order, the ten real effects listed first. Effects of three standard errors, ten real, familywise 5%. fixed sequence: 85.3% at position 1, 45.0% at 5, 20.4% at 10; overall power 46.10%; fallback: 49.1% at position 1, 56.4% at 5, 57.9% at 10; overall power 55.87%; Holm: 52.5% at position 1, 52.2% at 5, 52.5% at 10; overall power 52.53%.

Corrections, and what each controls

An order that spends the error rate

Test twenty hypotheses in a declared order, each at the full 5% and each only if every one before it was rejected, and the first is found 85.3% of the time where Holm finds it 52.5%. The tenth is found 20.4% of the time, the product of the powers before it. Move one true null to the head of the list and every real effect behind it is found no more than 4.3% of the time.

Expected loss of three upper limits against the level each is set at, fifteen exponential observations, penalty 20. A failure costs 20 standard deviations of margin. At the conventional 97.5% the t limit's loss is 2.143, Hall's 1.922 and the fitted family's 1.530. Each at its own best level: the t limit 1.263 at 99.95%, Hall's 1.862 at 98.50%, the family 1.308 at 99.25%. The best fixed multiple of the standard error costs 1.255.

Shape, and what it does to a two-sample test

The level a limit should be set at

When a failure costs twenty times a standard deviation of margin, a t limit on fifteen exponential observations set at the conventional 97.5% has an expected loss of 2.143. Set at 99.95%, the same formula's loss is 1.263 — within 1% of the best any fixed multiple of s/√n can do, and ahead of both Hall's transformation and the fitted gamma family at their own best levels. Hall's is the worst at every level on every source. Choosing the level does more than choosing the construction, and the correction that reads the sample is the one no level rescues.

How often a trial borrowing from 16 earlier trials that agree declares a treatment with no effect a success, by its own control's drift. Fifty patients an arm, each earlier control of two hundred, a one-sided 2.5% threshold. With 16 earlier trials, the half-normal prior peaks at 88.1%, the half-Cauchy at 80.7% near a drift of 0.6, and the half-Cauchy with a fifth of the prior on a vague component at 16.9%. At a drift of 1.5 the three read 73.3%, 25.5% and 3.7%.

The spread, and its own uncertainty

A history that agrees with itself

Borrowing a control from sixteen earlier trials that agree with each other should be safer than borrowing from one, and it is the opposite. Sixteen agreeing trials estimate the spread between trials as small, and a small spread estimated confidently is a licence to pool: under a half-Cauchy prior that let go of a single disagreeing trial, a current control 0.6 standard deviations from a sixteen-trial history turns a treatment with no effect into a success 80.7% of the time, against 10.5% at worst with one earlier trial. Moving a fifth of the prior onto a vague component brings the worst case to 16.9% and keeps power at 91.0%, against 70.5% without borrowing.

How often seven 95% intervals cover the small groups of a league table whose size relation is steepening below the median size. Groups of twenty patients or fewer, 600 tables of a hundred groups. Coverage: one centre, plugged in 58.0% (average width 2.81); one centre, its uncertainty added 58.5% (average width 2.83); a fitted line, plugged in 83.1% (average width 1.58); a fitted line, its uncertainty added 88.4% (average width 1.77); a fitted quadratic, plugged in 85.5% (average width 1.53); a fitted quadratic, its uncertainty added 93.7% (average width 1.92); the true relation and spread 94.8% (average width 1.55).

What partial pooling does to one group, to the set, and to a ranking

The interval a small group is given

A league table's 95% interval for a small group, pooled towards a size relation fitted from the table, covers 92.8% of the time when the relation is straight and 83.1% when it steepens below the median — and 77.4% for the groups of eight patients or fewer. Adding the fitted centre's own uncertainty, weighted by how much the group borrows from it, brings the line to 88.4% and a fitted quadratic to 93.7%, and the quadratic's interval covers 94.6% of the smallest groups at every shape tried. One centre, plugged in, covers the smallest groups 35.2% of the time when the relation steepens, and no added variance repairs a centre in the wrong place.

How often a winner's interval covers when it is conditioned on the analyses admitted to rather than the twenty run, effect 2. The largest of twenty analyses reported when it clears the family-wise threshold for the k admitted. At k = 1, 2, 5, 10, 20: correlation 0.6, conditioned on the admitted family 95.3%, 94.9%, 94.9%, 95.1%, 95.2%; the ordinary interval 67.7%, 71.1%, 75.2%, 76.5%, 76.1%; correlation 0, conditioned on the admitted family 94.4%, 95.6%, 95.3%, 94.8%, 95.6%; the ordinary interval 47.2%, 52.0%, 58.9%, 62.4%, 64.7%.

The analyses that were available and not run

An interval for the analyses admitted to

An analyst runs twenty correlated analyses, admits to one, and reports the winner with an interval conditioned on having cleared 1.96 — the selection as written down rather than as it happened. The interval covers 95.3% of the time, against 95.6% for one conditioned on all twenty, because a winner's truncation point is set by the threshold far more than by its unseen rivals: hiding nineteen analyses moves it by 0.061 on average. What the hiding changes is how often there is a winner at all. With no effect anywhere, a family of twenty produces a reported winner 35.5% of the time against 4.8% when all twenty are admitted, and one whose interval excludes zero 1.48% of the time against 0.13% — the interval keeps its promise and the report does not.

The shape the covariate enters by

The list it was chosen on

Six functions of one covariate looked like 720 designs, one per order of addition. The rule reads a span and not a sequence, so they are 63, and at a hundred units the best four leave 12.1% of a coin's imbalance against the designer's first four at 31.5%. Those best four contain no cube and are, function for function, the four shapes the worst case was scored on. Read on six shapes it was not told about, the same design leaves the cube 77.0% of a coin's imbalance, and the designer's four, which lost by a factor of two and a half, leave it 28.1%.

$Three repairs at one requirement. Coverage against spend at a half-width of 0.4 and a first stage of 5, all exact. A floor on the second stage at 5, 8, 10, 12, 15, 18, 21, 24 observations covers 88.4%, 91.4%, 92.4%, 93.1%, 93.9%, 94.6%, 95.3%, 96.0% at 0.86, 0.94, 0.97, 1.00, 1.02, 1.05, 1.09, 1.14 times the oracle's spend. The second stage reading with t, with its fraction dialled from 0.2 to 1, covers 88.1%, 89.0%, 90.6%, 92.1%, 93.4%, 94.4%, 95.2%, 95.9%, 96.4% at 0.84, 0.91, 1.02, 1.17, 1.34, 1.52, 1.70, 1.89, 2.08 times. A constant at the end covers 88.4%, 89.4%, 90.4%, 91.3%, 92.1%, 92.9%, 93.6%, 94.2%, 95.0%, 95.7% at 0.86, 0.90, 0.94, 0.98, 1.02, 1.06, 1.10, 1.14, 1.20, 1.26 times.$ What the design is asked to guarantee

A floor that reads the first stage

A three-stage stopping rule fails at a first stage of five because a low first reading asks for nothing more, and the only floors that repair it were set near the answer, by somebody who already knew roughly what the spread was. Inflating the first reading by Stein's own factor needs no such knowledge, and it does cut the experiments left stuck from a fifth to a fifteenth. But those still stuck cover less than before, the rule needs a different dial at every requirement, and a floor of 26 observations covers 96.5% at the spend where the rule covers 92.1%, even met by a spread it was not set for.

Coverage lost against the honest rule. Coverage lost in points against the rule that reads only within-block spreads, on the same observations, blocks of 2, 6000 runs, at half-widths 0.5, 0.35, 0.25, 0.15. the reported spread in full: 0.32 ± 0.27, 2.53 ± 0.37, 5.33 ± 0.41, 3.75 ± 0.35. a fixed half: 0.72 ± 0.26, 2.43 ± 0.29, 0.98 ± 0.26, -0.25 ± 0.20. κ = (b − 1)/(N − 1): 0.37 ± 0.21, 1.18 ± 0.25, 0.58 ± 0.23, -0.18 ± 0.19. the same, phased in as b/(b + 20): 0.22 ± 0.10, 0.35 ± 0.16, 0.23 ± 0.18, -0.10 ± 0.19. the same, read in only after twenty blocks: 0.00 ± 0.00, -0.02 ± 0.04, 0.17 ± 0.19, -0.17 ± 0.19.

The block size as a schedule

A rule that sets its own κ

A stopping rule may read the spread inside its blocks and may not read the spread between them, and the dial between the two costs nothing for a long stretch and then a lot. The share an efficiency argument recommends is each estimate's share of the degrees of freedom, and on blocks of two at a half-width of 0.15 that loses nothing and steadies the experiment by 30%. At 0.35 it loses 1.18 points. Deferring the forbidden reading until twenty blocks costs nothing measurable at any requirement, and it keeps most of the steadiness where steadiness can be had.

What the interval covers as the effect stops being one number. Coverage at 120 units under minimisation, 600 trials and 199 re-run allocations each, as the effect's slope on the prognosis rises from 0 to 1.5. the inverted test, unadjusted, for the population's mean: 96.7%, 96.2%, 95.7%, 95.7%, 94.8%. the same, for the sample's mean: 96.7%, 96.0%, 96.2%, 97.3%, 98.3%. the inverted test, adjusted, for the population's mean: 96.5%, 95.5%, 95.0%, 95.3%, 96.0%. the same, for the sample's mean: 96.5%, 95.7%, 96.5%, 98.0%, 99.5%. the regression with an interaction, for the population's mean: 96.2%, 95.5%, 94.0%, 91.0%, 84.0%. the same, for the sample's mean: 96.2%, 95.8%, 96.2%, 95.7%, 95.2%.

Balancing on what was recorded first

An effect that is not one number

A randomisation test becomes an interval for the effect by testing every constant effect in turn, so when the effect grows with the prognosis every hypothesis it tests is false. It does not fall apart. It covers the population's mean effect within a point of its level at every slope measured, and its width grows by exactly what the sample's own composition adds. A regression that models the interaction stays narrow at 0.72, covers the sample's mean at 95%, and covers the population's at 84%. The extra width was never a defect. It was the difference between two questions.

The guess rate through a trial. Guess rates in each tenth of a trial of 150 patients on three arms with the rule taking its preferred arm with probability 0.8, over 1500 trials. knowing everything: 65.9%, 72.2%, 72.0%, 72.8%, 72.4%, 72.2%, 72.5%, 72.7%, 72.7%, 72.4%. a joint table learned in earlier trials: 53.4%, 55.4%, 54.8%, 55.1%, 54.3%, 53.8%, 54.9%, 54.8%, 54.1%, 54.8%. the centre's totals: 45.4%, 50.3%, 50.2%, 50.8%, 49.8%, 50.2%, 50.9%, 50.6%, 50.4%, 50.7%. following whichever has been right: 46.5%, 50.2%, 49.3%, 49.6%, 48.9%, 48.7%, 49.3%, 49.0%, 49.0%, 48.7%. a joint table learned in this trial: 36.4%, 43.8%, 45.0%, 46.2%, 46.7%, 46.8%, 48.1%, 48.5%, 47.9%, 48.9%. the centre's own factors, scored: 46.8%, 51.4%, 49.7%, 49.9%, 48.4%, 48.6%, 48.5%, 47.7%, 47.6%, 46.7%. Chance is 33.3%.

More arms than two

The guesser who learns

An investigator who watches a three-arm minimisation can tally which of several partial readings has been right so far and follow the leader. It does not help. Following the leader names 48.9% of assignments where the best single reading names 49.9%, and a table of what the three readings say together, learned inside the trial, starts at 36% and has not caught up by a hundred and fifty patients. Learned over earlier trials, that table reaches 54.5%. The full guesser's 71.8% stays out of reach of all of them, because the gap to it is information rather than practice.

Choosing what the rule reads

A box with a count

The ellipse never runs out of assignments and holds the named functions loosely; the box holds them tightly and can come up empty. Dropping the ellipse's thin directions was meant to give both, and it does not: no cutoff comes within a third of the box. But the box's emptiness was never a property of the box. It belongs to a tolerance fixed in advance. Set the tolerance by a count instead, and at that count the region is the box, on every set of units, with nothing left to run out.

A guarantee that needed a symmetry

The covariate on its ranks

A skewed covariate spoils every parity argument a balancing rule relies on, and a rule that balances ranks instead of values looks like the way out, since a rank has the same uniform law on any marginal. The measurement says the skew really does belong to the scale for one kind of shape: a rank holds a median split at exactly a quarter of a coin's imbalance whatever the covariate. But it pays with a cap on any shape linear in the value, at nearly half a coin's imbalance on the strongest skew. And a rule holding values and median splits together holds both, beating the rank on the shape a rank gives up and matching it on the one it gains.

How often each combination, and each union of them, rejects ten studies of nothing. Each combination alone rejects exactly 5% of null sets. Counted on 1,000,000 sets of ten null studies: Fisher or Stouffer 6.63%, Fisher or Tippett 8.05%, Stouffer or Tippett 8.96%, any of the three 9.66% — enclosed on a two-dimensional lattice between 9.18% and 10.09% — and all three together 1.05%. The three sizes add to 15%.

Tests, and the second number

The smallest of three combinations

Reporting whichever of Fisher's, Stouffer's and Tippett's combinations is smallest is a test of its own, and on ten studies of nothing it rejects 9.66% of the time — not 5%, and nowhere near the 15% the three sizes add to, because the statistics are correlated at up to 0.903. Read at 2.448% each it is exact, and then it trails the best single combination by at most 7.45 points and leads the worst by at least 10.30.

The difference in restricted mean survival at every horizon, in three worlds. Treatment minus control, in closed form, with dropout irrelevant to the truth. The proportional treatment's difference grows to 0.4766 at τ = 3 and the waning treatment's to 0.2675. The crossing treatment's rises to 0.1776 at τ = 2, near where the two survival curves cross, and falls back to 0.1366 at τ = 3. The ticks along the bottom are the eleven horizons, from 0.5 to 3 in quarters, at which a trial below reads its differences.

When the data stops early

A horizon chosen after looking

A difference in restricted mean survival read at whichever of eleven horizons looks most convincing rejects 11.24% of trials in which the treatment does nothing, against 4.70% at a horizon fixed in advance. The correlation of the differences across horizons is closed, and the Gaussian process it defines prices the choice at a critical value of 2.317 — which brings the counted size back to 4.99% and keeps 96.92% of the power that a horizon nobody could have known to fix would have had.

The value that is not there

The assumption that identifies the mechanism

A selection model estimates how strongly an outcome decides whether it is recorded — the quantity two identical datasets showed no statistic can see — and it does so by assuming the outcome is normal. Where that holds and the outcome does decide, it repairs a slope complete cases put at 0.4318 to 0.5795. Where the missingness is at random and the residual is merely skewed, it reports selection that is not there, moves the slope from 0.5971 to 1.0319, and rejects missingness at random in 72.5% of studies.

How often a trial borrowing from 8 earlier trials on a trend declares a treatment with no effect a success. Fifty patients an arm, each earlier control of two hundred, the current trial one step after the last. At slopes 0, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1: exchangeable 2.7%, 4.4%, 6.6%, 8.3%, 9.9%, 8.3%, 6.7%; with a fifth of the prior vague 2.4%, 3.6%, 5.5%, 6.9%, 7.6%, 6.5%, 5.7%; with a line in time 2.9% at every slope.

The spread, and its own uncertainty

A history that has been moving

Earlier control arms borrowed as exchangeable draws are pooled towards the history's middle, and when the control mean has been rising the current trial sits at its end. A drift of 0.02 standard deviations a trial — a step between neighbouring trials of under a third of either one's standard error — turns a treatment with no effect into a success 9.9% of the time from sixteen earlier trials, and a drift of 0.05 overstates the effect by 0.130 standard deviations from eight. A line in time removes the bias exactly, at every slope, and pays for it in borrowing: from eight earlier trials it borrows 175 control patients where the exchangeable model borrows 439.

Every group's 90% rank interval in one league table whose size relation is steepening below the median size, with its true rank. A hundred groups from four to four hundred patients, ranked from their posterior under a fitted quadratic size relation whose uncertainty is drawn once per table. The groups of twenty patients or fewer have rank intervals 30.1 places wide on average, and 34 of their 36 true ranks fall inside; the groups of over a hundred, 32.1 places, with 27 of 30 inside.

What partial pooling does to one group, to the set, and to a ranking

A ranking with its uncertainty

A league table's reader wants a rank, and a rank interval inherits a fitted size relation's errors differently from an effect interval. When the relation steepens below the median size, a fitted line's 95% interval for a small group's effect covers 83.1% of the time; its 90% interval for the same group's rank covers 95.6%, because the line's error lifts the small groups into the crowded middle of the table, where rank intervals are wide. The error surfaces instead in the large groups' ranks, 87.6%, and in width: 36.9 places against the 27.3 the true relation needs. A quadratic's uncertainty has to be drawn once per table, since it moves every small group together — added group by group it over-covers at 92.8%.

How often a diagnostically chosen winner's interval excludes zero, against the diagnostic's correlation with the statistic, no effect anywhere. Twenty analyses correlated 0.6; the one with the best diagnostic is reported if it clears 1.96. A winner is reported in 2.52%, 4.16%, 6.81%, 13.07% of families at γ = 0, 0.3, 0.6, 0.9. Its interval excludes zero: conditioned on clearing 1.96, 1.69%, 2.28%, 3.30%, 4.69%; conditioned on the diagnostic as well, 1.69%, 1.86%, 1.83%, 2.31%; rebuilt from the report's three numbers, 1.69%, 1.80%, 2.13%, 2.52%.

The analyses that were available and not run

A winner chosen by its diagnostic

An analyst runs twenty correlated analyses and reports the one whose diagnostics look cleanest, if it clears 1.96 — and a diagnostic is rarely independent of the statistic it checks. With no effect anywhere, a diagnostic correlated 0.9 with its statistic lets a winner through in 13.07% of families against 2.52% for an unrelated one. The interval conditioned on clearing 1.96 still covers 93.27%, but its misses have moved: it excludes zero in 4.69% of reported winners, where a 95% interval should do so in 2.5%. Conditioning on the diagnostic as well repairs it, and so does something a reader can do alone — weight the reported statistic by the chance an analysis like it wins the diagnostic contest, from three numbers a report can state.

Expected loss of three upper limits against the level each is set at, when a failure is charged by its size, fifteen exponential observations, penalty 20. A failure costs 20 times the distance, in standard deviations, by which the mean exceeds the limit. At the conventional 97.5% the t limit's loss is 0.728, Hall's 1.136 and the fitted family's 0.835. Each at its own best level: the t limit 0.726 at 97.00%, Hall's 0.805 at 90.00%, the family 0.735 at 92.50%.

Shape, and what it does to a two-sample test

A failure charged by its size

Charge an upper limit's failure by how far the true mean exceeds it rather than by a fixed amount, and the level a limit should be set at falls further than any other change in this comparison moves it: on fifteen exponential observations, a penalty of twenty per standard deviation of shortfall asks for a t limit at 97% where a penalty of twenty per failure asked for 99.95%. The failures are small whichever limit produces them — about an eighth of a standard deviation — so a charge by size is a small charge per failure. The ranking hardly moves: the t limit stays within about 1% of the fitted family, whose failures are no smaller than its own, and Hall's transformation stays last by 11%.

Stopping rules

A look added by a rule

A monitoring committee's rule — add a look at three quarters of the trial when the interim |z| is at least 1.5 — spends 5.2323% under O'Brien–Fleming-type boundaries. Written into the protocol it can be given boundaries that spend exactly 5%, three ways: raise the added look's boundaries by 2.05%, raise every later boundary by 0.99%, or preserve at each interim value the chance of crossing the plan would have given. All three cost the same, 0.38 points of power, from 88.94% to 88.56%, because the power the uncorrected rule seemed to keep was the excess error. Made exact, the rule is a design between the two fixed schedules — and at every trigger slightly better than randomising between them.

How often reporting whichever contrast is most significant rejects when the arms do not differ, by the split. 20,000 trials a point. With both arms at 0.05, 0.1, 0.3: at shares 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7 the chosen contrast rejects 6.5%, 5.6%, 5.0%, 5.0%, 5.1%, 5.1%, 5.2%, 5.2%, 6.3%; 6.0%, 5.4%, 5.3%, 4.9%, 4.8%, 4.9%, 5.2%, 5.4%, 5.8%; 5.8%, 5.6%, 5.2%, 5.1%, 5.2%, 5.1%, 5.3%, 5.4%, 5.7%.

Splitting the units

A contrast chosen after the data

A two-arm trial with a binary outcome that reports whichever of the risk difference, the log risk ratio and the log odds ratio comes out most significant has tested three statistics that ask one question. At an equal split the choice is no choice at all: the difference has the largest z in every one of twenty thousand trials, so the selected test is the difference's, at its 5%. At a 30–70 split the choice is real and costs a point and a half of false positives, 6.5% at the rarest outcome. Pre-registering the difference and the odds ratio as a pair with Bonferroni's correction — the cautious alternative — costs eight to twelve points of power at every split.

How often an adjusted permutation test rejects a true sharp null when the covariate is chosen after randomisation, held fixed or re-chosen inside every shuffle. 150 units, half treated, six baseline covariates of which three are prognostic; 2,000 experiments and 99 re-randomisations each, read on the adjusted coefficient over its sandwich standard error. Choosing the covariate most correlated with the outcome: 5.0% held fixed, 5.0% re-chosen. Choosing the covariate with the smallest residual variance: 5.5% held fixed, 5.3% re-chosen. Choosing the covariate most imbalanced between the arms: 3.0% held fixed, 4.8% re-chosen. Choosing the covariate with the smallest adjusted p-value: 10.4% held fixed, 5.1% re-chosen. Naming the most prognostic covariate in advance: 5.5%.

The reference distribution the design supplies

The covariate chosen afterwards

A trial that chooses which baseline covariate to adjust for after randomisation has two ways to run its permutation test: hold the chosen covariate fixed across the shuffles, which is what an analysis that reports adjusting for it actually does, or re-make the choice inside every shuffle, which is exact by construction. Holding it fixed, a choice made on significance rejects 10.4% of true sharp nulls, a choice made on imbalance only 3.0% — and pays seven and a half points of power for it — while a choice that never reads the arm labels is the same test either way. The interacted, studentised statistic keeps its weak-null behaviour only when the covariate chosen is the one the effect varies with.

How often two orders of selection report a winner from twenty analyses, no effect anywhere. Twenty analyses correlated 0.6, each with a diagnostic correlated γ with its own statistic, over 20,000 families a point. Clean then significant — the best diagnostic, reported if it clears 1.96 — reports a winner in 2.55%, 4.25%, 6.91%, 13.15% of families at γ = 0, 0.3, 0.6, 0.9. Significant then clean — the best diagnostic among those that cleared 1.96 — reports one in 18.04%, 18.04%, 18.04%, 18.05%: in every family where any analysis cleared 1.96, whatever γ is.

The analyses that were available and not run

The cleanest of the significant

An analyst with twenty correlated analyses can choose in two orders: take the one whose diagnostics look cleanest and report it if it clears 1.96, or take the analyses that cleared 1.96 and report the cleanest of them. The second is the one described as prudent, because it never reports an analysis whose diagnostics look poor — and with no effect anywhere it reports a significant analysis in about 18% of families whatever the diagnostic is, against 2.55% to 13.15% for the first. The diagnostic decides which significant analysis is shown and never whether one is. Each report's own interval can still be made exact, by conditioning on a selection set that is a union of pieces rather than a half-line; the family's claim count cannot.

Stopping rules

The variance that contains the effect

A monitoring committee that adds an interim look because the variance came in higher than planned is reading a nuisance quantity, and a look added on a nuisance quantity should leave the spending function's 5% alone. It does, exactly, when the variance is the within-arm one, and when the trigger is the calendar. It does not when the variance is the one computed with the arm labels hidden: that variance is the within-arm sum of squares plus the interim statistic squared, so it is high more often when the trend is strong. The excess is 0.069 points at a twenty-patient interim and falls as one over the square root of the interim's size — a tenth of what a trigger on the trend itself spends, and never zero.

How far the published estimate of the chosen arm sits from the best arm's effect, by how six hundred patients are split between choosing and estimating. Eight arms and a control, six hundred patients: a stated number an arm in the first stage and the rest split equally between the chosen arm and the control in the second. Root mean squared error of the published estimate about the best arm's true effect, over 5,000 trials a split. All eight alike: best at 16 an arm (0.092), 0.183 at the familiar sixty; One modestly ahead: best at 56 an arm (0.151), 0.153 at the familiar sixty; One clearly ahead: best at 48 an arm (0.146), 0.154 at the familiar sixty; Two clearly ahead: best at 40 an arm (0.128), 0.151 at the familiar sixty; Spread evenly: best at 40 an arm (0.131), 0.146 at the familiar sixty. The split that guards every world puts 32 an arm before the choice and costs at most 21.3% over each world's best.

Designs that change while they run

Patients spent before the choice

A trial that picks the best of eight arms and then confirms it splits its patients between choosing and estimating, and the familiar shape — sixty an arm before the choice, thirty after — spends ninety per cent of a six-hundred-patient budget on the choice. Measured against the effect a reader is told about, the best arm's, that split is nearly right when one arm is modestly ahead and twice as wrong as it needs to be when the arms are alike. A split of thirty-two an arm before the choice is never more than 21.3% worse than the best split for whichever world holds; sixty an arm can be 98.8% worse. The two targets an estimate could serve — the chosen arm's effect and the best arm's — pull the split in opposite directions, and only one of them is what gets published.

What a trial's own first patients buy when they choose the follow-up schedule for the rest, powered for an effect of 0.2. Internal pilots measured on two follow-ups, the schedule that needs no pilot, whose correlation curve chooses the schedule for every later patient. Extra cost over the cheapest schedule at stable shares 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7: no pilot, two follow-ups throughout, 11.1%, 9.8%, 8.1%, 5.9%, 2.9%, 0.0%, 0.0%, 3.2%; an internal pilot of 20, 10.5%, 9.8%, 9.3%, 7.9%, 5.9%, 3.7%, 4.2%, 5.2%; an internal pilot of 40, 5.9%, 5.7%, 5.6%, 4.7%, 3.4%, 2.0%, 2.4%, 2.5%; an internal pilot of 80, 3.6%, 3.4%, 3.2%, 2.8%, 1.9%, 1.0%, 1.6%, 1.6%. Worst cases: no pilot 11.1%, 20 patients 10.5%, 40 patients 5.9%, 80 patients 3.6%.

What a sample-size calculation was given

A schedule the trial chooses for itself

A separate pilot cannot pay for itself in choosing a trial's follow-up schedule, because its patients are recruited, measured and discarded. A trial can instead measure its own first patients on the schedule that needs no pilot — two follow-ups — fit the correlation curve to them, and give everyone after them the schedule that curve makes cheapest. Those first patients are counted in the analysis and cost no extra visit. Forty of them cut the trial's worst-case excess cost across every curve with adjacent correlation 0.7 from 11.1% to 5.9% in a trial of two hundred an arm, and eighty cut it to 3.6%; twenty buy almost nothing. And because the curve is read from within-arm deviations, the choice cannot see the effect: the pilot's effect estimate is the same whichever schedule it chose.

The textbook (Wald) interval's coverage among studies whose count of 40 lands in the tails. Coverage of the nominal 95% textbook (Wald) interval at 40 trials, conditional on the count being at most k or at least 40 − k, computed exactly over the counts for every true proportion. With k = 2 its highest value anywhere is 88.96%; With k = 5 its highest value anywhere is 94.90%; With k = 8 its highest value anywhere is 95.57%. A set whose highest value stays below 95% is a set of counts on which the label is wrong whatever the proportion is.

Intervals, counted

A count that bets against its interval

A reader who sees only the count can bet against an interval's 95%, and win whatever the proportion is, exactly when some set of counts carries less than 95% at every proportion. Out of forty trials the textbook interval has one — a count of five or fewer, or thirty-five or more, covers at most 94.90% — and the score interval has none: a prior exists under which every one of its forty-one intervals is 96.46% credible, and that prior is the proof that no bet can be found.

Ten studies every study shares the control: the three combinations read as though independent, by the correlation. Counted on 400,000 null sets of ten one-sided studies at each correlation, every study shares the control. Fisher: 5.00%, 9.54%, 12.83%, 15.01%, 16.63%, 17.88%, 18.80%. Stouffer: 5.00%, 11.51%, 16.29%, 19.54%, 22.20%, 24.23%, 25.86%. Tippett: 4.98%, 4.86%, 4.68%, 4.45%, 4.17%, 3.72%, 3.27%. Smallest of three: 9.66%, 14.44%, 17.94%, 20.41%, 22.55%, 24.34%, 25.88% — at correlations 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6. Each combination is read at its critical value for independent studies.

Tests, and the second number

Studies that share a control

Ten comparisons against one control arm of the same size are correlated at exactly one half, and combined as though they were independent, Stouffer's sum rejects 24.23% of sets in which nothing works, Fisher's 17.88%, while Tippett's smallest p-value falls to 3.72%. Repaired with the correlation the design states, each is exact again — and the smallest of the three is then inflated less than for independent studies, 7.05% against 9.66%, because sharing makes the three combinations agree. The two inflations the dependence was expected to stack do not stack.

A cross-entropy fit of a normal proposal to P(Z > 5), stage by stage, fitting mean and spread or the mean alone. Each stage draws a thousand points from the current proposal, keeps the top tenth, and refits the normal to them, each weighted by the target density over the proposal's. Fitting the mean alone, the levels are 1.24, 2.99, 4.59, 5.00 and the fit reaches 5 at stage 4, ending at N(5.199, 1). Fitting mean and spread from the same seed, the spread goes 1.000, 0.456, 0.244, 0.096, 0.054, 0.043 over the first six stages, roughly halving each time, and after 30 stages the level has stalled at 3.728 with a spread of 1.0e-6, never reaching 5.

What makes it checkable

A proposal fitted to its own draws

The cross-entropy method fits an importance-sampling proposal from its own pilot draws, and asked to fit a normal's mean and spread to P(Z > 5) it fails before the question of variance arises: the spread halves at every stage, the level stalls near 3.7, and 95% of runs never reach the threshold, so the interval covers 1.1% of the time. Its ideal end point, the normal closest to Z conditioned past 5, is N(5.187, 0.181²), which has an infinite variance inside its own draws' reach and covers 93.7% at a thousand draws and 90.8% at ten thousand. Fitting the mean alone lands within a tenth of the best shift and covers 94.5%.

The bias each model puts into the treatment effect when sixteen earlier controls stepped once. Sixteen earlier trials of two hundred, the first eight a step below the last eight, which sit at the current control's level; the current trial has fifty patients an arm and no treatment effect; 1,500 trials at each step. Exchangeable trials: -0.003, 0.020, 0.041, 0.056, 0.063, 0.065, 0.054. A line in time: -0.003, -0.016, -0.029, -0.041, -0.052, -0.067, -0.075. A change point: -0.003, 0.003, 0.003, -0.001, -0.002, -0.003, -0.003 — at steps of 0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5.

The spread, and its own uncertainty

A history that stepped

When sixteen earlier control arms stepped once — the first eight a fifth of a standard deviation below the last — borrowing them as exchangeable flatters a treatment with no effect by 0.063 and stops borrowing, and fitting a line through them penalises it by 0.052. A model with a change point is unbiased at every step size, borrows more the larger the step, 512 patients at a fifth, and puts two thirds of its belief on the right place for the change. Against a steady drift it is the one that fails, so each of the three is safe only against the shape of history it assumes.

What the sixteen-run factorial's surviving runs estimate when about 1.3 runs are lost, by why they were lost. A 2⁴ factorial fitting eleven coefficients; true A = 1, B = 0.5, AB = 0.5, everything else 0, noise 1; 20,000 datasets. Runs are lost when the response exceeds 2.5, or at random at the high corner of A and B, or at random anywhere, each losing 1.26 runs on average. Bias of the fit on the surviving runs — lost at random: A 0.001, B 0.003, AB 0.000, C -0.002; lost at the high corner: A 0.003, B 0.001, AB 0.000, C -0.000; lost when the response passes the threshold: A -0.131, B -0.120, AB -0.117, C -0.000.

Decided before the data

A run lost for a reason

A sixteen-run factorial that loses a run at random loses precision and nothing else. One that loses a run because the process failed — the response passed a threshold — loses about the same number of runs and biases exactly the effects that pushed the response there: an interaction of 0.5 reads 0.383, a main effect of 1 reads 0.869, and the interval around each still covers about 94%. Treating the lost runs as censored above the threshold overshoots with eleven coefficients and works with six, because a run's own leverage decides how much a censored observation pulls its own estimate.

Seven ways to report the height at the setting a response-surface analysis recommends, at noise 2. Bias and root mean squared error of the reported height about the true height at the chosen setting, over 2,000 studies with a central composite design of thirteen runs, or the same design run twice. 13 runs, the fit's own height: bias 0.884, error 1.537. 13 runs, optimism bootstrapped away: bias 0.183, error 1.396. One confirmation run alone: bias -0.005, error 2.005. Corrected fit and one confirmation run: bias 0.056, error 1.121. 26 runs, the fit's own height: bias 0.475, error 0.998. 26 runs, optimism bootstrapped away: bias 0.050, error 0.924. 26 runs split: choose on 13, estimate on 13: bias -0.014, error 1.213.

The surface between the corners

The height at the chosen setting

The height a response-surface fit predicts at the setting it recommends reads high, because the setting was chosen where the fit was highest. The obvious repair — choose on some runs, estimate on the rest — is impossible in the usual thirteen-run design, which has nine distinct settings where two separate fits need twelve. A parametric bootstrap of the optimism removes four fifths of it at no cost in runs, 0.884 down to 0.183 at twice the noise. With the design run twice, correcting the full fit beats splitting it: an error of 0.924 against 1.213, at a better setting.

Testing for flatness at 5% before choosing the interval for a squared mean. Coverage of four nominal 95% intervals for μ², exact, against the true mean's distance δ from the flat point in standard errors. The tangent interval Z² ± 2z|Z| has its worst, 85.98%, at δ = 1.45. The second-order interval read off χ²₁ is right only at δ = 0 and falls to 34.22% at δ = 2. The two-stage rule, which uses the second-order interval when a 5% test does not reject flatness, has its worst, 49.72%, at δ = 1.96. The squared exact interval never falls below 95%.

The distribution itself

Testing for a flat point first

Test whether the gradient is zero, and if the test does not reject, read the interval off the second-order law that holds at a stationary point. For a squared mean that two-stage rule covers about 50% at its worst whatever the pretest's level — 49.72% with a 5% test, where the tangent interval it was meant to rescue never falls below 85.98% — because its worst sits exactly where the true distance equals the test's critical value, half the samples are declared flat there, and the second-order interval cannot reach the truth from any of them.

Reversals that are not errors

A correction built on a retest

Tweedie's correction for regression to the mean needs the error variance, and a study gets it from a few people measured twice. Twenty of them are enough: with normal errors the estimated correction then beats the correlation's rule on 73% to 77% of studies of a thousand readings from a heavy-tailed population, though its variance is uncertain by a third. What twenty cannot reveal is the error's shape. Laplace errors of the same variance make the correction overstate the lead a top group keeps by about 0.17, the plain rule then wins on most studies, and a study needs about two hundred retest pairs before a check on its own differences notices.

How often a league table's own data choose a bent centre over a straight one. Over 400 tables of a hundred groups each. The quadratic's curvature coefficient is significant at 5% on 5.3% of tables whose relation is straight, 14.5% of tables whose relation is flattening below the median, 29.3% of tables whose relation is steepening below the median; its average AIC weight is 0.374, 0.456, 0.558.

What partial pooling does to one group, to the set, and to a ranking

A centre chosen by its own table

A league table that picks its centre's shape from its own data — a line unless a curvature test says otherwise — picks the line on 85.5% of tables whose relation flattens below the median, because the bend lives where the groups are smallest and noisiest. Its bottom ten then flags like the line's: 2.61 innocent small groups a table against 2.19 for a quadratic told the shape. Always fitting the quadratic costs 0.07 of a correctly flagged group when the relation is straight and matches the told answer when it bends, and a flag set by probability carries the choice through a mixture of the two.

How many units a range needs to let at most r of the next hundred outside, fixed, learning or forgetting. 95% of the time, for any continuous population. At most 1: fixed 637, window 564, learning 512; At most 2: fixed 300, window 236, learning 197; At most 5: fixed 107, window 64, learning 36; At most 10: fixed 50, window 27, learning 4. The window counts are from 20,000 lots of uniform draws; the other two are exact.

The interval that holds observations, not a mean

A range that forgets

A band set from the range of the last sixty-four units flags each unit of a stable lot with probability exactly 2/65, for ever, and its flags arrive as if independent — so a promise of at most five outside in the next hundred costs sixty-four units, between the learning range's thirty-six and the fixed range's hundred and seven. What it was supposed to buy, a flag that lasts as long as a step and then stops, it cannot give: it absorbs a two-sigma step as fast as the learning range, 1.89 flags in ten units against 1.93, because it learns from what it flags. Refusing to learn from a flag makes it a fixed range that costs 141.

How far each estimator errs when instruments are cells of unequal size and the confounding moves with them, μ² = 8. Thirty-three cells of 2 to 15 units, thirty-two instruments, 1,000 draws a point. At η = 0, 1, 2: two-stage least squares 0.286, 0.387, 0.593; likelihood, Bekker's interval 0.049, 0.534, 0.830; jackknife instrumental 0.146, 0.325, 0.689; jackknife likelihood, robust interval 0.066, 0.066, 0.195.

A variable that moves one thing only

Cells of unequal size and an error that follows them

Thirty-two instruments that are the indicators of cells holding 2 to 15 units each, and confounding whose variance is larger in the small cells. Limited-information maximum likelihood, the estimator that came out least biased with many instruments, then errs by 0.534 at a concentration of 8 — further than two-stage least squares — and Bekker's interval covers 74.8%. The jackknife version of the likelihood, which never lets a row's own leverage reach its own fitted value, errs by 0.066 and its robust interval covers 90.6%; at eight times the strength, 94.8% against Bekker's 87.3%. On normal instruments the same unequal variance costs almost nothing, because their leverages are nearly equal.

What a diagnostic plot is showing

Residuals of a model the data chose

Five columns picked by a forward search from twenty candidates that have nothing to do with the outcome leave residuals that show a skewed error 36.8% of the time, where five columns fixed in advance show it 65.4% — the power of a fit of 13.4 fixed columns. Against light-tailed errors the chosen fit's plot is blind, 8.5% against 34.5%. Simulating the search on normal errors gives a band that holds 5%, which the ordinary band already nearly did; the level was never what broke. A reader told the final model has five columns is reading a plot with the power of thirteen.

The interval for a finding's effect given that it cleared its threshold, against where its estimate landed. Threshold c = 2.8. At z just past it the selective 95% interval runs from −0.58 to 0.95; at z = 3.8, from 0.18 to 5.73; at z = 6.3, from 4.31 to 8.26, against the ordinary z ± 1.96.

Corrections, and what each controls

An interval that knows its finding was found

An interval for one Benjamini–Hochberg finding can be read from the distribution of its estimate given that it cleared the threshold it had to clear. It covers the finding's effect 94.8% of the time with real effects of two standard errors, against the ordinary interval's 87.0%, and it covers the most prominent finding 91.3% of the time where the ordinary one covers 72.4%. The price is width — a median 5.06 standard errors against 3.92 — and something larger: for three findings in four at that effect size the interval reaches back past zero, so a finding that cleared the line is, once the line is accounted for, not distinguishable from nothing.

Two tests, a threshold, and the rate they are read against

A threshold reported with its interval

Resample a validation study of a hundred diseased and three hundred healthy people, refit the binormal model each time, and read off the threshold each resample recommends. Far from the prevalence at which the best threshold jumps to flagging everyone, the interval is tight — flagging 12% to 24% of healthy people at a prevalence of 5%. Near the jump, the median study's interval runs from flagging under half of the healthy to flagging everyone, and 91% of intervals cross the jump. The two ends differ by fifteen false alarms' worth of harm per hundred people, and no rule for choosing inside the interval beats the study's own choice at every prevalence.

How often a test of a supplied cointegrating relation rejects it, by how wrong it is and how long the sample is. 800 systems a point. At 100, 200, 400, 800 observations the right relation is rejected 9.4%, 5.4%, 5.4%, 6.4% of the time; one with c = 0.9, 19.9%, 37.8%, 78.5%, 98.5%; c = 1.1, 16.8%, 37.9%, 76.8%, 97.5%.

Three series, and a count

A restriction the data can check

A rank-two system of three series is identified by exactly two restrictions on its relations, and any two that describe the estimated plane fit it equally well — their forecasts agree with the unrestricted fit's to the last digit, so the data can neither confirm nor refuse them. Supply one relation in full instead, one restriction more than identification needs, and the surplus becomes a test: a relation a tenth wrong is rejected 37.8% of the time at two hundred observations and 98.5% at eight hundred, the error found half the time halving as the sample doubles. A right relation buys the forecast almost nothing; a fifth wrong costs 13% of it at two hundred observations and 25% at eight hundred.

The bias a tolerance on the second moments leaves where a square drives both the assignment and the outcome. 600 samples a point. Bias at δ = 0, 0.02, 0.05, 0.1, 0.2, 0.5: −0.0074, 0.0500, 0.1353, 0.2782, 0.5473, 0.6983; coverage 89.7%, 89.5%, 72.4%, 26.5%, 1.0%, 1.0%. The tolerance spent in full at the outcome's slope on the square would give 2·δ·√2: 0.0566, 0.1414, 0.2828, 0.5657 at the four smaller values; balancing the means alone gives 0.6973.

Weighting one sample into another

A balance allowed a tolerance

Weights that balance the means exactly and the second moments only to within a tenth of a standard deviation were supposed to keep the second-moment repair and recover the samples exact balance loses. Where a square drives both the assignment and the outcome they keep neither half of the repair: the bias is 0.2782 against −0.0074 with exact balance, and the interval covers 26.5%, because the weights spend the whole tolerance — every second-moment gap sits at the edge of its band — and the outcome converts a gap of a tenth into bias at its own slope, 2·δ·√2 = 0.283. At thin overlap the same tolerance does recover samples: 89.0% have weights at all, against 62.0% with exact balance.

All themes