Normal Distribution
nothing is called 95% until it has been counted
Theme

The thread: The second number

A p-value cannot be interpreted alone. The same 0.04 corresponds to a large effect at ten observations and a negligible one at two thousand, and it means nothing at all until you know how many analyses were available to produce it.
One margin rises; the other turns over. The two halves of the table's margin across the sweep: a lower-tail copula's own leak with a symmetric covariate, and a covariate skewed at 0.95 under a Gaussian copula. The marginal's leak rises at every step, from 3.727% to 36.056%. The copula's does not: it rises to 9.064% at a Spearman of 0.6 and falls to 8.219% by 0.7. It has to turn over, because at a rank correlation of one the two variables are a deterministic function of each other and there is no interaction left for a split to leak. So the margin of the table turns over before any cell in it does. The same table at seven correlations

A margin that turns over

A skewed covariate's leak grows without limit as the dependence strengthens. A copula's own leak does not — it peaks at a rank correlation of 0.6 and falls. The margin of the table turns over before any cell in it does.

 One factor moves and the other does not. The two factors of the same average, each drawn against its own largest value so that they share an axis. The rate at which the fifteen candidates disagree about the tuning parameter rises from 28.6% at 4 values on the list to 43.3% at 8, a factor of 1.52. What a disagreement costs, given that there was one, is 0.00975 ± 0.00224 and 0.00848 ± 0.00113 at the same two points — 0.5 standard errors apart, and the paired comparison on the draws that disagree under both lists puts it the other way. The guess this field was written to test was that a longer list makes disagreements commoner and each one smaller. The first half is right and there is no second half. The rate and the size of a disagreement

A rate times a size

A sweep reported what it costs to let every candidate choose its own tuning parameter and found it flat across the list. It was reporting a product, and the two things multiplied together do not behave the same way at all.

 What each variant loses before anything has been searched for. The mean loss differential of each of the eight variants against the benchmark, over 600 tables of 60 origins, with every fit given 71 rows. The series is an AR(1) and every variant adds a lag whose coefficient is zero, so in population the two forecasts are the same forecast and the difference drawn here is estimation noise and nothing else. The marked line is σ²(q₁ − q₀)/n = -0.01408, which is an expression in how many coefficients each model has and how many rows it was fitted on — it knows nothing about the series, the persistence or which lag the variant added, and every bar is within a fifth of it. This is the amount a reference distribution recentred at each column's own sample mean believes the candidates are already behind by. Searching among fitted models

A table of nested models

A benchmark and eight variants of it, each adding one thing. Every variant is behind before the search begins, by an amount that can be written down before the data exists — and the two most natural ways of reading the table are wrong in opposite directions.

 The construction survives a difference of two weighted means. Coverage of δ̂ ± t√(S_D²/H) on b − 1 degrees of freedom, over 900 runs at a requirement of 0.3, where δ̂ is the block differences weighted by h_b = (1/m_A + 1/m_B)⁻¹ and H is their total. The theorem the one-mean field rests on goes through with h_b in place of the block size, and the reason is that the weights a weighted least squares decomposition needs are the inverse variances — which is exactly what h_b is. The stopping rule reads only within-arm within-block contrasts, so it is a function of nothing the interval reports, whatever it does with the block sizes. Each bar is within 2.9% of the level it claims. A promise about two arms

A width promised for a difference

The exact fixed-width interval was built for one mean. Two arms make the target 42.7 units of effective size and each unit costs four observations, so the same promise about a difference costs 169.4 rather than 42.7 — and the theorem survives untouched with the harmonic size in place of the block size.

 Three intervals, one shortfall. What each of three intervals actually covers, at four rules and two block windows, over 300 samples of 120 rows. All three are built from the same resamples on the same draws, so a difference between them is a difference in what is done with the resampled series. Not one of the twenty-four cells reaches the ninety-five per cent it promises. The studentised interval runs from 75.7% to 92.3%, the percentile interval — the earlier field's — from 80.0% to 89.7%, and a normal interval on the same scale from 81.7% to 89.0%. The standard repair for a percentile interval's shortfall does not repair it. The interval, studentised

An interval that carries its scale

A percentile interval inherits the resampled distribution's skewness and its scale error together. The standard repair is one extra variance per resample. It was named and not run, so this runs it.

 The area under the window is what the band actually costs. The three windows' weight sequences at a width of 30 lags, drawn against the lag as a share of the window. A truncated window applies a weight of one to every lag inside it and zero outside, which is why its sum is the width and why every conventional charge is right for it — and it is a covariance matrix on almost no sample, so it cannot be used. The Bartlett window falls linearly to zero and its weights sum to exactly 15.000000000000004, which is half the width, at every width: Σ(1 − k/(L+1)) over k = 1 … L is L − L/2. The Parzen window sums to 11.13 here, three eighths of the width, and it gets there by holding a weight near one over the first few lags and then falling faster. A plug-in estimate multiplied by a weight below one is a shrunk estimate, and a shrunk estimate is worth less than a free one — which is the whole of why a charge levied per lag is a charge for parameters the window has already spent. A charge for a covariance's own dimension

The charge nobody derived

A band of lags is charged one log-likelihood unit apiece, because that is what a regression coefficient costs. A band's numbers are not regression coefficients, and measuring what they actually cost puts the convention out by a factor of nearly three.

 The eighth was not a constant. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations between the candidates, over 800 draws apiece. The earlier field reports this flat at about an eighth across list length, on a table and a world it never varies. Vary how much the omitted coefficients are worth — one multiplier, with the table, the list, the law and the sample size all held — and it runs from 17.6% to 1.5%, a factor of 11.75. The world in which every candidate is true is the world in which the tuning list decides most; the world in which one candidate dominates is the world in which it decides nothing. What decides whether a tuning list decides

The eighth that was not a constant

How often a per-candidate tuning list changes which candidate wins is reported flat at about an eighth across list length. Vary how far apart the candidates are instead and it runs from 17.6% to 1.5%.

 A quantile is the dearer reading, everywhere. The error each rule and window delivers on the two error readings, over 400 draws. The lower pair of lines is the implied long-run variance — the instrument the earlier field uses — and the upper pair is the 95% point of the standardised resampled mean, read against the finite-sample truth of 3.889 found by simulating the law directly. The quantile costs more at every one of the eight cells: at the plug-in rule it is 59.1% against 45.3% for the taper. That is not a defect in the bootstrap; a quantile is a statement about the shape of a distribution as well as its scale, and a fixed number of resamples estimates a tail worse than a variance. What matters for the comparison is that the two orderings between the windows are not the same, which the margins figure is about. The block length read on a quantile

The instrument and the reading

Every comparison between two block windows in this collection is an error in an implied long-run variance. Nobody reads a long-run variance. Read on the 95% point a test uses, the same bootstrap costs half as much again.

 The curvature is in the denominator. The optimism a Bartlett band of each width actually costs, divided by that width, on two ways of measuring the width, over 2000 draws at 120 rows. Measured in the weights the band spends — Σ w(k), which is what the earlier field levies its charges on — the reading falls from 0.9528 at two lags to 0.7486 at thirty, so a charge proportional to the summed weights is too dear at one end and too cheap at the other. Measured in the pairs the band uses — Σ w(k)(1 − k/n), because a lag of k is an average over n − k products — the same readings are flat from 4 lags up, at 0.0084 of χ² per width against 0.2359. The correction has no fitted parameter in it: it is a function of the window, the width and the sample size. A charge that is not a straight line

The width a band is measured in

A tapered covariance band spends 84% of its own weights at two lags and 74% at thirty. Every charge in the collection is a straight line through the origin in those weights, so it is too dear at one end and too cheap at the other.

 What each pair is made of. Each pair of searches, over 300 draws, split into the two effects its excess is the difference of. The overlap is what the second search loses by having the first already run at its own answer; the interaction is what the joint search finds by moving the first off it. They subtract to the excess exactly, on every draw, because the pinned supremum cancels. Two disjoint dictionaries of independent columns read an excess of 0.000011 and are made of 0.000514 and 0.000503. A break paired with a dictionary of step columns has an interaction of exactly 0 and is all overlap. And a break paired with an independent column has an overlap of -0.004395 against an interaction of 0.002364, which is what puts its excess below zero. Overlap and complementarity, separated

Two effects in one number

How much two searches over one sample share is measured as the net of two things — ground both of them find, and configurations only the joint search reaches. One extra supremum per draw separates them exactly.

 What a mean split leaves, with both halves varying. The share of a mean split's interaction that survives the rule balancing it, at every copula and every marginal, matched at a Spearman correlation of 0.40. The three radially symmetric copulas leave exactly nothing with a symmetric covariate and rise steeply with the skew. The two asymmetric ones start at 7.707% and go opposite ways: the lower-tail copula falls to 0.002% at a skewness of 0.95 — the two failures cancel almost exactly, and a guarantee both fields report as broken is restored — while the upper-tail one climbs to 40.288%. And the heavy-tailed symmetric covariate, which leaks exactly nothing on its own, doubles what the asymmetric copulas leak: 14.229% against 7.707%. Both halves of the dependence at once

Two failures that cancel

A mildly skewed covariate under a lower-tail copula leaks 0.002% of an interaction where each failure alone leaks eight and seven per cent. Turn the copula over and the same pair compounds.

 Two searches find some of the same luck. What each search reports on a sample with no break in it, and what the two report together, on four dependences. The dashed line is the sum of the two — what a rule charging each search separately would levy — and the two together always come in below it: 19.30, 15.44, 14.24, 26.64 short, on 100%, 99%, 99%, 100% of draws. The shortfall is not a rounding. Under AR(1) at 0.8 it is 19.30 of the 34.70 the break search manufactures on its own, which is more than half of it. Two searches over one sample are looking at the same noise, and the second one has less left to find. Two searches over one sample

Two searches, one sample

A searched break in a regression manufactures 34.7 of likelihood ratio where a count of coefficients says 11.1. A searched window manufactures 84.0. The two together manufacture 99.4, not 118.7.

 What the rule blocks is not where it splits. How much of the separating direction each probe carries, over 100 designs of 14 units whose admissible set is enumerated and split into two pieces. The deferral this field answers proposed the constraint's active set — which exchanges the tolerance box actually blocks — as a better probe than the design's own leverage, on the ground that leverage is a heuristic and the active set is the quantity. Modelled from the design and the tolerance, it reads 0.4272 against leverage's 0.5395, at 4.43 paired standard errors the wrong way. Counted exactly over the enumerated set — at a cost no trial can pay — it reads 0.3854, worse again. Both beat a random direction at 0.2622, so they are probes; neither beats the two the earlier field already had. A probe from what the rule blocks

What the rule blocks

A balancing rule breaks the admissible set into pieces by refusing exchanges. Which exchanges it refuses is computable from the design and the tolerance alone, before any assignment exists — and it makes a probe.

 What x-bar plus or minus 2 sample standard deviations holds, at n = 10. The content of the band is a random variable. Across 20,000 normal samples of 10 it averages 91.1%, its fifth percentile is 74.7%, and it falls short of 95% on 59.9% of samples. The band that is drawn to show where 95% of the data lies. The interval that holds observations, not a mean

Two standard deviations of what

The 95.45% inside two standard deviations is a fact about a curve whose centre and width are given. Drawn from ten observations, the same band holds 91.1% on average and less than 95% on 59.9% of samples — and the average is the reading that hides it.

 Which side a 95% t interval misses on, exponential source. Both tails should be 2.5%. At 8 observations the interval falls short of the mean on 9.75% of samples and overshoots on 0.31%. At 500 they are 3.31% and 2.05%, and the total is 5.36% — which a coverage table reports as very nearly right. Shape, and what it does to a two-sample test

Where the two tails disagree

A 95% t interval on an exponential source at 120 observations covers 94.81%, which reads as very nearly right. It misses below the mean on 4.08% of samples and above on 1.11% — one tail 63% too heavy and the other 56% too light, and the total is the statistic that hides it.

 What a second positive is worth, prevalence 0.10%. A 90% sensitive, 95% specific test. One positive gives 1.77%. Two independent positives give 24.49%, which is what multiplying the likelihood ratios says. At a correlation of 0.1 between the tests' errors it is 10.16%, and at 0.5 it is 3.16%. Two tests, a threshold, and the rate they are read against

The second test that is not a second opinion

Two positives from a 90/95 test on a one-in-a-thousand condition give a 24.49% chance of disease if the tests are independent. At a correlation of 0.1 between their errors it is 10.16%, and at 0.5 it is 3.16% — barely more than the 1.77% one positive was worth.

 Which groups partial pooling serves, standard error 1 population width. Pooling's expected squared error for a group, divided by its own mean's, against how far the group truly sits from the centre. It is ×0.25 at the centre and crosses ×1 at 1.732 population widths, beyond which 8.33% of a normal population lies; capping the shift at one standard error holds every group under ×2. What partial pooling does to one group, to the set, and to a ranking

A group from the population's own tail

Partial pooling halves the total squared error when a group's own standard error equals the spread between groups. Every group whose true effect sits more than 1.73 population widths from the centre — 8.33% of a perfectly normal population — does worse than it would have with its own mean, and its loss grows without bound. Among eight groups with the spread estimated, the most extreme is worse off in 61.6% of datasets. Capping the shift at one standard error keeps the total at 0.528 of the unpooled error and holds every group under twice it.

 The curvature is in the denominator. The optimism a Bartlett band of each width actually costs, divided by that width, on two ways of measuring the width, over 2000 draws at 120 rows. Measured in the weights the band spends — Σ w(k), which is what the earlier field levies its charges on — the reading falls from 0.9528 at two lags to 0.7486 at thirty, so a charge proportional to the summed weights is too dear at one end and too cheap at the other. Measured in the pairs the band uses — Σ w(k)(1 − k/n), because a lag of k is an average over n − k products — the same readings are flat from 4 lags up, at 0.0084 of χ² per width against 0.2359. The correction has no fitted parameter in it: it is a function of the window, the width and the sample size. A charge that is not a straight line

A lag the sample has less of

A sample autocovariance at lag k is an average over n − k products, not n. Count a band's width in the pairs it actually has and the curvature in its charge goes away, on a correction with nothing fitted in it.

 Two instruments, two block lengths. The block length that would actually have been best on each draw, for each of the two error readings, averaged over 400 samples of 120 rows. For the rectangular window the implied long-run variance wants 18.92 and the 95% point wants 16.05; for the tapered window, 21.82 against 17.74. The quantile wants a shorter block under both windows — a ratio of 0.848 and 0.813. That is the mechanism the whole field turns on: a rule for choosing a block length is a way of guessing a target, and the two instruments do not have the same target. A rule tuned to one is systematically long for the other, and the two windows do not pay the same price for being long. The block length read on a quantile

A length for each instrument

The block length that is best for an implied variance is 18.92; the one best for the 95% point of the same resamples is 16.05. A rule is a way of guessing a target, and there are two targets.

 A model of the active set, and the active set. Each of the 14 units of one design, at the share of its exchanges the tolerance box blocks — computed from the design's columns and the tolerance under a uniform position in the box, against counted over all 116 admissible assignments. The diagonal is where the two would agree. Over 192 designs they agree about the ordering of the units at a correlation of 0.8141 ± 0.0112, negative on 0.5% of them, and disagree about the level: 0.8442 counted against 0.8170 modelled, a gap of 0.0272 ± 0.0051. An admissible assignment does not sit uniformly in its box, and this is the size of that. A probe from what the rule blocks

A model and a count

The share of a unit's exchanges a tolerance box refuses can be modelled from the design or counted over the admissible set. They order the units the same way at a correlation of 0.81 and disagree about the level by 0.027.

 Three charges, and only one of them is a test. What each of three thresholds does to the same decision, under AR(1) at 0.8, against the size of a genuine break in the mean at row 60. A chi-square on the 5 coefficients a split adds — 11.07 — declares a break on 73.6% of samples that have none: it is not a test at all. The break search's own 95% point, 69.6, carried into a rule that also chooses its window, fires on 0.0% of null samples and on 0.0% of samples with the largest break measured — the natural way of combining two published corrections does not lose a little power, it switches the test off. The calibrated charge, 27.2, holds 5.6% at no break and reaches 29.2% at the largest. Two searches over one sample

The charge that is not a sum

Charging two searches what each costs on its own is conservative, and conservative here means the test never fires. At the largest break measured it declares nothing, on every draw, while a calibrated threshold reaches 29%.

 What a disagreement costs, split on whether it decided anything. The regret from choosing the tuning parameter per candidate, on the draws where the candidates disagreed, split on whether the disagreement changed which candidate the table selects. Over 1200 draws at each list length: when the winner changes the regret is 0.02215, 0.03029, 0.03145; when it does not it is -0.00243, -0.00069, -0.00065 — negative, and small enough that it is inside two standard errors of nothing at every length. The whole of the cost lives in the first column, and the second column is not merely small but slightly the wrong sign: when the table's answer is unaffected, letting each candidate use its own window is a very slightly better rule than making them share one. So a disagreement about the tuning parameter is not a cost. A disagreement that changes the winner is. The rate and the size of a disagreement

The quarrel that changes the winner

A disagreement about the tuning parameter costs 0.031 when it changes which candidate the table selects and −0.0007 when it does not. The distance between the values disagreed about has nothing to do with it.

 No block size is best at both things the procedure claims. Two claims and one dial. The honest interval's half-width falls as the blocks get smaller, because the interval's degrees of freedom are the number of blocks: 0.2602 at blocks of two against 0.2933 at blocks of sixteen. The fixed-width claim — that the mean is within 0.25 of the truth — gets more reliable as they get larger, because the sample size is less variable: 92.40% against 95.00%. Both are computed from the same runs, and the second is reproduced to within a tenth of a point by E[2Φ(d√N/σ) − 1], which needs the sample-size distribution and nothing else. The schedules sit at the bottom left: as narrow as the smallest fixed block and as few observations, with the rule's spread estimate on half as many degrees of freedom again. The block size as a schedule

Two degrees of freedom, one total

The block size is a dial, and the two things a fixed-width procedure claims move in opposite directions along it. Divide the width by the square root of the sample size and one of them turns out to depend on the number of blocks and on nothing else.

 Two factors, opposite directions. The two factors the cost of a per-candidate tuning parameter is a product of, as the candidates are pulled apart, over 800 draws at each of 5 separations. How often the candidates disagree about the tuning parameter rises from 31.8% to 88.8%; the share of those disagreements that change which candidate the table selects falls from 51.6% to 1.7%. So the setting where the candidates quarrel most about the tuning parameter is the setting where the quarrel matters least, and a sweep that reads the rate and stops has read the factor pointing the wrong way. What decides whether a tuning list decides

Two factors pointing opposite ways

As the candidates on a table are pulled apart, they quarrel about the tuning parameter three times as often and the quarrel decides the winner thirty times less often. A sweep that reads the first factor has read the one pointing the wrong way.

 Power at an effect of 0.5 standard deviations. The curve is the non-central t on 2n − 2 degrees of freedom with δ = d√(n/2); the dots are 4,000 experiments run at each size. Reaching 80% power needs 64 per arm. Tests, and the second number

What a p-value does not say

The same p of 0.04 corresponds to a large effect in ten observations and a negligible one in two thousand. A p-value alone cannot be interpreted, and the number that makes it interpretable is almost never printed beside it.

 What a positive test means, sensitivity 90%, specificity 95%. At a prevalence of one in a thousand, 98 of every hundred positives are false. At one in 10, 33 are. The test has not changed. Reversals that are not errors

What a positive test is worth

A test that is 90% sensitive and 95% specific sounds accurate. For a condition affecting one person in a thousand, 98% of its positive results are wrong, and a worse test on a commoner condition beats a better test on a rare one.

 What studentising costs. How much wider the studentised interval is than the percentile one, cell by cell, over 300 draws, with what each cell gains in coverage beside it. Averaged over the eight cells the interval is 2.09 times as wide and covers 0.46 points better. At the two rules that choose short blocks the two intervals are within a fifth of each other; at the oracle's length, where a resample holds two or three whole blocks, the studentised interval is 4.37 and 5.04 times as wide. A repair that doubles the width and buys half a point is not one a reader could not have had by widening the interval it replaced. The interval, studentised

What studentising costs

Averaged over eight cells the studentised interval is 2.09 times as wide as the percentile one and covers 0.46 points better. At the block lengths the rules choose, the scale it divides by rests on two or three numbers.

 The split decides the width. The width of the interval against the share of 200 observations spent on fitting rather than on calibrating, over 3000 draws. Spending more on the fit shrinks the residuals; spending more on calibration builds the interval at a less extreme order statistic. The two meet at 0.5, where the width is 4.0416 against 4.1603 at 0.1 and 4.3820 at 0.9. Full conformal, which spends the same 200 points on both jobs, is 3.9865 — so the whole cost of splitting is 1.38%. Coverage without a distribution

What the split costs

Splitting a sample between fitting and calibrating looks like a trade against the guarantee, and it is not: coverage moves 0.63 points across nine splits and every reading sits on its own promise. The whole cost is 1.38% of width — and at sixty observations the width falls, rises and falls again.

 Three bands called 95%, at n = 20. Half-widths in sample standard deviations: 0.468 for the mean, 2.14 for one future observation, 2.75 to hold 95% of the population. The first two differ by exactly the square root of n + 1, which is 4.58 here. The interval that holds observations, not a mean

A tenth as wide, and both of them right

The interval for a mean and the interval for one future observation are both labelled 95%, and at a hundred observations one is 10.05 times the other — exactly the square root of n + 1. Read the narrow one as the wide one and it covers a new value 15.7% of the time.

 Two studies with an R-squared of 0.85. The left study's points sit 0.50 from the line and the right study's 2.00 — a factor of 4.0. Both report an R-squared of 0.85 and, at the same sample size, the same standard error for the slope. The design was chosen to make it so, and it can always be chosen. What a summary of a scatter is a property of

The t statistic wearing different clothes

For a simple regression, t² = (n − 2)R²/(1 − R²), exactly, on every dataset — checked to sixteen significant figures over five hundred fits. So a paper reporting R² and a p-value has reported one number twice, and two studies with the same R² have points four times further from the line.

 What the plot says, and what the interval does, at n = 40. For each source: how often a quantile plot of the data leaves its pointwise band, and how often the 95% t interval for the mean misses. The two-lump source leaves the band on 100% of samples and its interval covers 94.80%; the t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of the five. What a diagnostic plot is showing

The plot is about the wrong quantity

A t interval needs the sampling distribution of the mean to be normal, not the data. A two-lump source leaves its quantile band on 100% of samples of forty and its interval covers 94.80%; a t on three degrees of freedom leaves it on 57% and covers 95.73%, the best of five sources.

 Worst and average coverage of six 95% intervals for a proportion, 30 trials. The worst coverage over every proportion beside the average over a uniform one, with the average expected width. Clopper–Pearson: worst 95.05%, average 97.34%, width 0.299. Blaker: worst 95.00%, average 96.31%, width 0.283. Wilson: worst 83.71%, average 95.24%, width 0.271. A proportion's interval near the boundary, and the coin

What a guaranteed minimum costs

Clopper–Pearson's interval never covers less than 95%, and at thirty trials it averages 97.34% and is 10.4% wider than Wilson's. Blaker's interval keeps the same guarantee, averages 96.31% and is 4.6% wider. The difference is not waste: Clopper–Pearson guarantees each side separately, holding both below 2.5%, and Blaker guarantees only their sum — so at ten trials and a proportion of 0.15 it misses on one side 5.00% of the time.

 Two 95% intervals 2.772 standard errors of the difference apart, standard errors in the ratio 1. The intervals are separated, and the test of the difference gives p = 0.0056. Two 95% intervals with equal standard errors just touch at p = 0.0056. An interval read beside something else

Two intervals that overlap

Two 95% intervals that just touch are read as a difference at the edge of significance. With equal standard errors their difference has p = 0.0056, not 0.05; two intervals can overlap by 58.6% of an arm and still differ at exactly 5%; standard-error bars that just touch mark p = 0.157; and when the two estimates are correlated at 0.8, touching intervals conceal a difference of 6.2 standard errors. Read as a test, non-overlap needs 1.66 times the sample for the same power.

 How the truth, the raw means, the posterior means and the constrained estimates spread, standard error 1. Beyond two population widths above the centre lie 2.28% of the true values, 7.86% of the raw means, 0.234% of the posterior means, and 2.28% of the constrained estimates. What partial pooling does to one group, to the set, and to a ranking

Estimates that are too alike

Posterior means give each group its least-error estimate, and as a set they are too alike: with each group's standard error equal to the population's spread, they spread 0.707 as widely as the truth. Beyond two population widths lie 2.28% of the true effects, 7.86% of the groups' own means, and 0.234% of the posterior means — a tenth of the truth. Rescaling the estimates to the right spread counts the tail exactly and costs 17% more squared error; summing each group's posterior chance of being beyond the line counts it without changing any estimate.

 The chance a trial succeeds against its size, when the expected effect of 0.5 is uncertain by four amounts. With the effect known, 80% is reached at 63 per arm. With the effect uncertain by 0.25 standard deviations it takes 113; by 0.5, 1268; by 0.75, no sample size at all, because the chance can never exceed the 74.8% prior probability that the effect is positive. What a sample-size calculation was given

The chance a trial succeeds

A trial of sixty-four per arm has 80% power at an effect of half a standard deviation. If the effect is only believed to be about half a standard deviation, give or take a quarter, the chance the trial reaches significance is 69.2%; give or take a half, 61.4%. Reaching 80% then takes 113 per arm, or 1,268 — and when the belief is uncertain by three quarters of a standard deviation no number of patients reaches 80%, because the chance can never exceed the 74.8% probability that the effect is positive at all.

 What a search costs is not a property of that search. The likelihood ratio a searched break in the regression reports, two ways, on every law. On its own — the whole rule being a split of the sample, at no whitening — it averages 34.70 under AR(1) at 0.8, against the 11.07 a chi-square on the five coefficients a split adds would use as a threshold. Inside a rule that also chooses a window from a list of eight, the same search adds only 15.40 — less than half. Most of what a break search finds under correlated errors is the correlation, and a whitening chosen from the same sample has taken it already. A charge measured for one search, carried into a rule that makes two, is not conservative in some harmless direction: it is measuring a different quantity. Two searches over one sample

A charge that depends on the rule

The break search's charge is 34.7 on its own and 15.4 once a window has been chosen from the same sample. Most of what a break search finds under correlated errors is the correlation, and a whitening has taken it already.

 Three copulas that break nothing, and a factor of two between them. The three radially symmetric copulas, at a matched Spearman correlation of 0.40, against the covariate's marginal. All three leave exactly nothing with a symmetric covariate — that is the guarantee, and it holds to twenty decimal places. What they do to a skewed covariate is not the same at all: at a skewness of 2.26 a Frank copula leaves 12.118% where a Gaussian leaves 21.539% and a t on four degrees of freedom leaves 23.640%. A factor of 2.0 between two copulas that are both symmetric, both matched on rank correlation, and both harmless on their own. So the copula matters to the marginal's leak without breaking any symmetry of its own, which is a milder version of the same finding and applies to every trial rather than to the asymmetric ones. Both halves of the dependence at once

A copula that halves a marginal

Three copulas break nothing on their own and put a factor of two between the same skewed covariate's leaks — 12.118% under a Frank against 23.640% under a t, at the same rank correlation.

 A line in the right width beats two curves. How far each candidate charge sits from the measured optimism across the plateau, in units of each width's own standard error, over 2000 draws. The straight line through the origin in the band's summed weights — which is what the earlier field levies — misses by 0.2382 per width. The same straight line in the pairs the band actually uses, Σ w(k)(1 − k/n), misses by 0.0095. A fitted power law misses by 0.0293 and a fitted decaying rate by 0.0172, both on one fitted constant more. The deferral this field answers asked for a curve; the answer is a line, in a variable with nothing fitted in it. A charge that is not a straight line

A line that beats two curves

A deferral asked for a curve. Fitted against the same measurements, a straight line in a variable nobody had to fit describes the plateau better than either curve does with a constant more — and for three windows out of four it does not.

 What each probe can see. How far apart the two components of the admissible set are on each probe, over the spread inside a component, on 100 designs whose set is enumerated and split. It is the population quantity a chain is trying to report. The separating direction itself reads 10.5646; the projected fourth power 5.0800, the design's own leverage 3.8362, the modelled active set 1.9529, the counted active set 2.0170 and a random direction in the same subspace 0.9422. The two active-set probes beat the random direction and lose to both of the earlier field's, which is the field's answer to the question that opened it. A probe from what the rule blocks

A quantity that loses to a heuristic

Leverage is a heuristic about which units a balancing rule has most to say about. The constraint's active set is the thing the rule actually does. As a probe, the heuristic wins by 4.4 paired standard errors.

 The split depends on the order. How much two searches share, measured both ways round, over 300 draws. Pinning the first search at its own answer and searching the second gives one overlap; pinning the second and searching the first gives another. A break paired with an independent column reads -0.004395 one way and 0.002163 the other, at 8.70 paired standard errors and on opposite sides of zero. The excess the two components subtract to is the same in both orders by construction, so what changes is only how it is attributed. There is no order-free way to say which of two searches found ground both can reach, and the two orders bracket it. Overlap and complementarity, separated

A split that depends on the order

Run the second search first and pin that instead, and the same draw gives a different overlap and a different interaction — with the same difference. And one pair has no second order at all.

 Flat along a row, apart between them. The probability that a per-candidate tuning list changes the winner, at three list lengths on three candidate tables, over 800 draws in each of the nine cells. Along a row — the reading the earlier field takes — it moves by a factor of at most 1.21, so that field's invariant survives on every table. Down a column it moves by up to 1.98. The list length is the dial that does not move this number and the table is one that does, and the earlier field varied only the first. What decides whether a tuning list decides

A table and a list

A nested ladder of candidates differing by one coefficient was predicted to turn over more often at every list length. It turns over less at every one, and its list changes the winner half as often.

 Cut the charge and the width follows it. The band width each charge picks, averaged over 400 draws of 120 rows under AR(1) at 0.8, with the standard deviation across draws beside it. Schwarz's charge — half a log n a lag, which is 2.39 here — picks 3.67. Akaike's picks 6.02. Charging the numbers the window actually leaves free, which is half the width, picks 10.12; charging what the optimism measures, 0.767 of that, picks 14.15. A charge and the width it buys are very nearly reciprocal, which is what a likelihood rising at a fixed rate a lag implies and is why the four answers span a factor of 3.86. The width that was actually best on the draw averages 13.90 and moves by 10.30 from draw to draw — three times as much as any rule's answer does. A charge for a covariance's own dimension

A width that moves and an error that does not

Four charges give four widths a factor of four apart and four errors half a per cent apart. The derived charge wins, significantly, by a quarter of what was on offer — and none of the four is an estimate of anything.

 Four cells change their answer. The four cells of the twenty whose excess changes sign as the dependence strengthens, over 7 recalibrations. Above the line the two failures compound — the cell leaks more than adding the copula's own leak and the marginal's — and below it they cancel. All four start above and end below, and all four are at the two most skewed covariates: skew 0.90 under heavy-tailed, skew 0.95 under heavy-tailed, skew 0.90 under upper tail, skew 0.95 under upper tail. Whether two failures of a dependence compound or cancel is therefore not a property of the pair. It is a property of the pair at a strength of dependence, and a fifth of the table changes its answer inside the range measured here. The same table at seven correlations

An answer that changes

Eleven of twenty cells cancel and nine compound, at one rank correlation. Sweep the correlation and four of the twenty change sides — all four from compounding to cancelling, all four at the most skewed covariates.

 The quantity that does not depend on the list. The probability that letting each candidate choose its own tuning parameter changes which candidate the table selects — the product of the two moving shares — against the length of the list, over 1200 draws apiece. It is 14.2%, 11.9%, 12.3%: a spread of 2.2% across a list length that moves the disagreement rate by a factor of 1.52. This is the invariant the whole field turns on. Everything downstream of the winner — the coefficients, the regret, whatever a reader is going to quote — is a function of whether the winner changed, and how often that happens is not something the list controls. A longer list changes how often the candidates quarrel and not how often the quarrel matters. The rate and the size of a disagreement

How often it matters

The disagreement rate rises by half across the list and the share of disagreements that decide anything falls by nearly the same factor. Their product — how often the tuning list changes which candidate wins — sits at an eighth and does not move.

 Two arms leave one degree of freedom per block unaccounted for. Each point is one run. The one-mean field's identity is (b − 1) + (N − b) = N − 1, and every schedule moves along that line rather than off it. Two arms give the rule N − 2b and the interval b − 1, which come to N − b − 1 — short of the N − 2 two arms leave by exactly one per block, since a block's arm counts absorb one degree of freedom each and only one of the two directions carries the difference. The hollow points add what the block sums are worth, b − 1 more, and land on the total. The missing degrees of freedom are not lost; they are in a place the interval has to be shown it may read. A promise about two arms

The degrees of freedom in the sums

One arm partitions N − 1 exactly. Two arms give the rule N − 2b and the interval b − 1, which is short by one per block — and the missing ones are in the block sums, which are correlated with the differences at −0.79 and are usable anyway.

 Four rules of four change sign. The margin between the two block windows in points of coverage, under each of four rules, on each of three intervals built from the same resamples, over 300 draws. Positive is the tapered window covering better. On the percentile interval the taper wins at all four rules, by 5.33, 1.67, 5.00 and 4.00 points, which is the earlier field's own reading. On the studentised interval the rectangle wins at all four, by 4.33, 7.00, 4.33 and 2.67. And a normal interval, which uses no resampling at all, puts the two within a third of a point at every rule — so the disagreement is manufactured entirely by what is done with the resamples. The interval, studentised

The ordering reverses again

One field found two of four rules changing sign between two readings of one resampling. Turn the same resamples into a studentised interval instead of a percentile one and all four change sign.

 Power to find a real effect of 3 standard errors, 10 of 20 real. no correction finds 85.1%, Bonferroni finds 49.1%, Holm finds 52.5%, Benjamini–Hochberg finds 74.9%. The uncorrected procedure finds the most and controls nothing. Corrections, and what each controls

The price of control

Every correction is paid for in power, and the exchange rate can be measured. Holm buys familywise control for 33 percentage points of power; Benjamini–Hochberg buys a weaker guarantee for 10. Neither is free and neither is a matter of taste.

 The reversal is a property of the instrument. The margin between the two block windows under each of four rules, on three readings of the same resampled means, signed so that a positive bar is the tapered window winning. On the implied long-run variance the taper wins at the best available block length and at one estimated from the sample and loses at a length written into a protocol and at the rule of thumb — which is the reversal the earlier field's whole argument turns on, at 1.48 and 4.52 points. On the 95% point a test actually reads, the taper wins at all four, by 6.13 to 7.08 points. On the coverage the interval actually delivers, the taper wins at all four again, by 2.50 to 5.75 percentage points. Two of the four rules change sign between the first reading and the other two, and the two that change are exactly the two the earlier field's recommendation is about. The block length read on a quantile

The reversal that was the instrument's

On an implied variance the rectangle wins at a protocol length and at the rule of thumb. On the 95% point a test reads, and on the coverage an interval delivers, the taper wins at all four rules.

 12,000 studies of a real effect of 0.3, n = 16. Power is 21%. The studies that reached significance report a mean effect of 0.621 — 2.07 times the truth. Every one of them is honest; the selection did the inflating. Tests, and the second number

The winner's curse

Filter honest studies down to the ones that reached significance and the effects they report are systematically too large. At low power the inflation is a factor of two, nobody has done anything wrong, and the selection did all of it.

 The ordering is the same under every law. Each pair's overlap under each of the four laws, over 400 draws apiece. The scale is fixed at both ends by construction: two searches over disjoint sets of independent columns read -0.001, 0.013, -0.002, 0.004, and a break search paired with a step column it already contains reads exactly one under every law. Between them the pair that reads one residual series twice runs 0.763, 0.805, 0.577, 0.752 — lowest under long memory, where a whitening has most to do and the break search has least left to find that the whitening has not taken. The pair that moves most is the pair of step dictionaries, from -0.148 under a moving average to 0.419 under a break; and the pair that is negative is negative under all four. Two searches over different features

Three quarters of the way to one search

The pair that started this reads 0.762 on a scale whose one is containment. And the pair that shares nothing but its response reads −0.306, so the sign the earlier field found does not transport at all.

 The overshoot is the last block size and nothing else. A run stops at a multiple of its own block sizes and cannot land between them, so it ends past its own target by about half a block. Fixed sizes overshoot by 1.5, 2.2, 3.1, 4.7, 8.5 observations as the size goes 2, 3, 5, 8, 16. Every schedule here ends in blocks of two and every one of them lands where blocks of two land — 1.62, 1.32, 1.37 against 1.48 — while having spent most of the run inside blocks four and eight times larger. That is the one thing on this page a schedule genuinely takes from both ends. The block size as a schedule

What a schedule actually buys

Big blocks early and small blocks late is the right instinct and it does not take both ends of the trade, because there are not two ends to take. What it does take is the overshoot — about four per cent of the observations — and a steadier stopping point.

 A filled value is not an observation. What a 95% interval for the slope actually covers after each way of handling 35.0% missing outcomes, counted over 4000 studies of 200 rows. Dropping the incomplete rows covers 95.93%. Filling with the observed mean covers 13.85%, because the estimate itself has moved. Filling with a fitted value covers 80.85% against a closed prediction of 79.73%: the estimate is right and the reported standard error is short by a factor of 0.6567 against a predicted 0.6500, because the residual sum of squares is divided by the whole sample's degrees of freedom. Adding residual noise recovers the spread and covers 85.78% against a predicted 84.62%, since the interval still ignores the variance of having imputed at all. The value that is not there

One imputation is not an observation

Three ways of filling a missing outcome, under a mechanism that makes dropping the rows beyond reproach. Filling with the observed mean covers 13.85%, filling with a fitted value covers 80.85%, adding noise covers 85.78%, and the thing all three were meant to improve on covers 95.93%.

 What the family-wise correction does to the effect it lets through. At two standard errors the estimate that clears an uncorrected 5% threshold averages 1.35 times the truth, and the one that clears the family-wise threshold averages 1.69 times it. The correction fixes the error rate by demanding a larger estimate, and a larger estimate is a more selected one. The analyses that were available and not run

The correction that makes the estimate worse

Correcting for twenty analyses repairs the p-value by demanding a larger statistic, and a larger statistic is a more selected one. At two standard errors the surviving estimate averages 1.35 times the truth before the correction and 1.69 times it after — so the honest error rate is bought with a more inflated effect.

 Estimating a prevalence of 0.10% from 1,000 tests. The positive rate reads 5.09%, which is 50.9 times the truth. The Rogan-Gladen correction averages 0.100% — unbiased — with a standard deviation of 0.820 points against the positive rate's 0.695, and it comes out negative on 48.6% of samples. Two tests, a threshold, and the rate they are read against

The prevalence the test has to estimate

Every predictive value takes a prevalence as given, and the prevalence is usually estimated from the same test's positive rate. At a true prevalence of one in a thousand that rate reads 5.09% — fifty times the truth — and the correction that inverts it is unbiased, 18% more variable, and negative on 48.6% of samples of a thousand.

 Welch's test on skewed groups: the low and high rejection rates in every cell, with equal means throughout. Each cell should read 2.5 / 2.5. Two identical exponentials at 20 and 20 read 2.04 / 2.22; the worst cell, a wide exponential against a normal at 8 and 32, reads 9.79 / 0.47. Shape, and what it does to a two-sample test

The skewness of a difference

Welch's test holds its size to within half a point when both groups are normal. Give both groups the same skewed population and it still balances at twenty and twenty — and at eight and thirty-two it rejects low on 7.16% of samples and high on 0.66%. One number decides which: the skewness of the difference of the two means, which ranks twenty-five cells by their imbalance with a correlation of 0.997.

 Two studies of the same effect, z statistics with mean 1.96: where one is significant and the other is not. Five hundred pairs. With the true effect identical in both, exactly one of the two is significant in 50.0% of pairs; among those, the difference between the two is significant in 9.7%. The dashed lines mark 1.96 on each axis; the diagonal lines mark a significant difference. An interval read beside something else

Significant in one, not in the other

Two studies of exactly the same effect, each with 50% power, disagree about significance half the time — and when they do, the test of the difference between them is significant in 9.75% of cases. A p of 0.01 beside a p of 0.20 is a difference with p = 0.36. Among four subgroups sharing one effect, at least one significant and one not happens 87.5% of the time, and the test that would tell a real difference apart needs four times the sample the effect itself needed.

 How many of a league table's top ten are small groups, ranked four ways. A hundred groups with sizes from 4 to 400, of which 36% have twenty units or fewer. Small groups make up 36.3% of the true top ten, 62.1% of the top ten by raw means, 13.4% by posterior means and 22.9% by the posterior chance of being in the top ten. The three rankings recover 4.43, 5.38 and 5.47 of the true top ten. What partial pooling does to one group, to the set, and to a ranking

A league table of a hundred

A hundred groups with sizes from 4 to 400, and a top ten to publish. Ranked by their own means, small groups fill 62.1% of the top ten against their 36.3% share of the true top ten. Ranked by posterior means they fill 13.4%. The ranking built from each group's chance of being in the top ten recovers 5.47 of the true ten, the best of three and barely half; and the group ranked first could hold any rank from 1 to 31.

 A wrong weight costs width; a random weight costs level. Five weightings on a trial whose variance ratio drifts by a factor of 20.1 between the first block and the last, over 4000 runs. The rule that knows every λ_b covers at 95.1% and sets the width. One ratio for the whole trial is wrong for every block and costs nothing in level — 94.8% — while being 20% wider; equal weights are calibrated by an identity and 22% wider. The ratio estimated inside each block is the only rule aimed at the quantity that actually varies, and it is the only one that misses the level, at 92.0%: a weight computed from a handful of degrees of freedom is mostly noise, and noise in a weight is not a wrong weight. Modelling the drift across blocks recovers the oracle's width at 94.8%. What a block may vary

A ratio that changes between blocks

A wrong weight costs width and a random weight costs level. The rule aimed at the quantity that actually varies is the only one that misses its own coverage, and the rule that models it across blocks recovers the whole of what knowing it is worth.

 Power at an effect of 0.5 standard deviations. The curve is the non-central t on 2n − 2 degrees of freedom with δ = d√(n/2); the dots are 4,000 experiments run at each size. Reaching 80% power needs 64 per arm. Decided before the data

How many subjects

Sixty-four per arm for 80% power at half a standard deviation — a power figure that could only be simulated, with nothing to disagree with, until the non-central t was written. Two routes now, agreeing to within the simulation's own error.

 The false-positive rate against the number of analyses, on pure noise. The data has no effect in it. Each analysis is correct and each p-value is honest. With 20 correlated outcomes available, something reaches p below 0.05 58% of the time. Tests, and the second number

Twenty analyses of nothing

Twenty honest, correct analyses of data with no effect in it find something significant 57% of the time. Nobody p-hacked, every individual p-value is right, and the reported one is the smallest of twenty.

 The scale moves the width; the curve does not. The band width each charge picks, averaged over 150 draws of 120 rows. The two conventions — a unit a lag and half a log n a lag — pick 6.08 and 3.65 lags. The four charges derived from the measured optimism pick 15.05, 15.60, 15.47 and 15.44, against a best width on the draw of 14.13. So the scale a charge is levied on moves the width by a factor of 4.27 and the shape of the charge moves it by 3.6%. None of the six is an estimate of the draw's own best width: the correlations are -0.006, -0.003, 0.017, -0.001, 0.016, -0.004. A charge that is not a straight line

What a better charge buys

Four charges derived from the same measurements pick band widths within six per cent of each other and deliver errors within two per cent of the gap any of them leaves. The scale a charge is levied on decides the width; the shape of the charge decides nothing.

 What the interval actually covers. The coverage of the two-sided interval each rule and window builds, over 400 samples of 120 rows, against the 95% it promises. Not one of the eight reaches it: the best is 91.0% and the worst is 80.8%, on a promise of 95%. So the first thing this instrument says is that the choice between the two windows is a choice inside a range that is already four to fourteen points short, which neither of the other two readings can express at all. The second is the ordering: the tapered window covers better under every one of the four rules, by 5.00, 2.50, 5.75 and 4.00 points — including at a length written into a protocol and at the rule of thumb, where the implied variance says the rectangle wins. The block length read on a quantile

What the interval covers

Eight rules and windows, and not one of them reaches its promised 95%. The range is 80.8% to 91.0%, and the choice between two block windows is a choice inside a shortfall that is four times larger.

 Eight encompassing nulls, all of them true at once. The forecast under test is the variance-minimising combination of the eight candidates, and the first-order condition that defines its weights is cov(e_c, e_j) = var(e_c) for every j — so every one of the eight nulls is exactly true simultaneously and every rejection counted here is false. 800 draws of 60 origins. The largest of the eight statistics rejects 25.0% of the time at a nominal 5% and one comparison stated in advance rejects 3.1%, which makes the set worth about 9.1 independent comparisons — nearly the eight it has. Bonferroni, which is far inside its level on a search over accuracy comparisons of the same eight forecasters, is at 4.8% here. Searching among fitted models

When every null is true

A reality check assumes that every candidate in the set is exactly as good as the benchmark, which is a configuration nobody's data is ever in. Test a combination against its own parts and that configuration is not assumed — it is what the arithmetic makes true.

 One penalty, read along two dials. How much wider the studentised interval is than the percentile one, at every sample size and every block length on the grid, with the number of whole blocks each cell leaves written beneath. Read across a row and the block length changes; read down a column and the sample size does. The penalty is nearly a function of the block count alone: the cells at 15 blocks read 1.16, 1.20, 1.17, 1.15, 1.13, 1.10 across three sample sizes and three block lengths, while the cells at one block length read anything from 1.10 to 3.95. The largest penalty on the grid is 3.95, at the cell with 3 whole blocks in it. The interval, studentised

The count or the length

A block length and a block count are one number read two ways at one sample size. Read at three, the studentised interval's width penalty tracks the count — with an R² of 0.9911 against a closed form that has no length in it — and its coverage tracks the length.

 The same dial, on a list that steps by one. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations, for both tuning parameters at a matched list length of 8, with errors from a first-order autoregression. The sieve order runs from 13.1% to 2.6%, a factor of 5.00; the whitening window, from 16.4% to 1.5%, a factor of 11.75. What is held is the number of options, the table, the law, the sample size and the seeds; what cannot be held is the size of a step, since an integer step and a geometric step are different amounts of change. The dial moves both, and it moves them by 2.35 times as much on one as on the other. What decides whether a tuning list decides

A step that is not a ratio

Run the separation sweep on a tuning list of integers rather than a geometric ladder and the two factors still point opposite ways. The invariant does not survive: along a row of integers the probability moves by 2.163 where along the geometric ladder it moves by 1.208.

 Six scores, one coverage. What each nonconformity score's interval covers, over 2000 draws with 200 calibration points, against the 95.0249% the rank argument promises. The column runs from 94.30% to 94.90%, a spread of 0.60% against a standard error of a difference of 0.69% — one number, six times. That includes a score aimed five units off the fit and a score that never reads the response at all, because the rank argument does not read the score either: it needs the scores exchangeable and nothing else. Every decision a modeller makes has to show up somewhere else, and the next two readings are where. Coverage without a distribution

The score is the modelling

Six nonconformity scores on the same draws cover within 0.60 points of each other, against a standard error of a difference of 0.69 — one number six times. Their widths run over a factor of 2.361 and their adaptivity over a factor of 8.377.

 Which samples Wilson and Clopper–Pearson each cover, n = 50, p = 0.2. Each bar is the probability of one count, shaded by which interval built on that count contains 0.2. Both cover 95.1% of samples, only Wilson 0.0%, only Clopper–Pearson 1.6%, neither 3.3%. The correlation between their hits is 0.810, so on shared draws the variance of their difference is 4.891 times smaller than on independent ones. What makes it checkable

The same draws for both methods

Two intervals computed on the same simulated datasets give a difference in coverage whose variance can be 4.891 times smaller than on separate datasets — or, for a pair that covers different samples, 1.164 times larger. Which one a comparison gets is an exact sum over the counts each interval covers, and a standard error that ignores the sharing covers 100.00% for one pair and 93.07% for the other.

 Four 95% intervals for the mean of 30 exponential observations, split by the side they miss on. Both bars should read 2.5%. The t interval misses below the mean on 6.38% of samples. Widened until its total is exactly 5%, it misses below on 4.69% and above on 0.30%. Hall's transformation misses below on 3.31% and above on 1.96%. Shape, and what it does to a two-sample test

The side a bound is read from

On thirty exponential observations the upper limit of a 95% t interval is exceeded by the true mean 6.38% of the time, against the 2.5% a safety margin set from it assumes. Widen the interval until its total coverage is exactly 95% and the upper limit is still exceeded 4.69% of the time. A symmetric repair fixes the number that is reported and not the one that is used; Hall's transformation, which bends the interval, takes the same rate to 3.31%.

 The factor that holds all of the next m observations with 95% probability, from a sample of 10. From 10 observations, the band for one future value has factor 2.371, for ten 3.716, for a hundred 4.942 and for a thousand 6.008. The 95%-content tolerance factor is 3.382 and is passed by m = 10; the Bonferroni stretch of the prediction factor reaches 7.567 at a thousand. The interval that holds observations, not a mean

All of the next ten

A warranty, a batch release or a monitoring rule promises something about every one of the next ten observations, not about one. From a sample of ten, the band that holds all ten with 95% probability reaches 3.716 sample standard deviations either side of the mean — already wider than the 3.382 of a tolerance interval for 95% of the population — and it keeps widening: 4.942 for a hundred, 6.008 for a thousand, with no ceiling. A 95% prediction interval, read as the answer, holds all ten 67.9% of the time: more than 0.95 to the tenth power, because the ten succeed and fail together.

 Where two verdicts disagree, across the true effects of two studies, beside where the test of their difference finds it. The shading is the chance that exactly one of two independent studies is significant, over the plane of their true z means. It is highest on a cross centred where either mean is 1.96 and lowest in the corner where both are large, whatever the difference between them. The difference test's power is constant along lines parallel to the diagonal: 50% on the inner pair and 80% on the outer. Two studies at means 8 and 4 — one effect half the other — split 2.1% of the time, and the difference test finds the halving 80.7% of the time. An interval read beside something else

The verdicts watch the threshold, not the gap

Two studies whose effects really differ are read through their verdicts as often as through their difference. Halves of an 80%-power study whose effects differ by the whole effect split 72.87% of the time against 49.99% when they do not — a split is never more than 1.86 times as likely under any difference up to twice the effect. And a halving of the effect in two large studies leaves both significant 97.93% of the time, while the test of the difference finds it 80.74%.

 The sample a trial needs for the same power, as a share of the unadjusted analysis's, by how strongly a baseline covariate predicts the outcome. Adjusting for the covariate as measured needs one minus the squared correlation of the unadjusted sample; cut at the median, one minus 2/π of the squared correlation. At a correlation of 0.7 those are 0.510 and 0.688; cut in three, 0.611; the change from baseline, 0.600. The change score needs fewer patients than the median-cut adjustment once the correlation passes 0.624. What a sample-size calculation was given

A baseline cut in two

Adjusting a trial's result for a baseline measurement that predicts the outcome shrinks the sample it needs to 1 − ρ² of the unadjusted one. Adjusting for whether that measurement was above its median shrinks it only to 1 − (2/π)ρ² — the cut keeps 63.7% of what the covariate could remove, exactly the fraction a cut outcome keeps. In patients the loss grows with the covariate's strength: a cut costs 12% more patients at a correlation of 0.5, 35% at 0.7 and 2.55 times as many at 0.9, where a simple change from baseline would have done better than the cut adjustment.

 Thirty surveys of a thousand people, true prevalence 0.1%, test 90% sensitive and 95% specific. Each row is one survey. The thin bar is the corrected estimate ± 1.96 standard errors; the heavy bar is the exact interval for the positive rate carried through the correction; the tick is the one-sided 95% upper limit. 30 of the thirty printed intervals reach below zero and 0 lie entirely below it. Two tests, a threshold, and the rate they are read against

An upper limit is the finding

A survey of a thousand people with a 90%-sensitive, 95%-specific test, estimating a prevalence of one in a thousand, prints a corrected 95% interval that covers the truth 95.17% of the time — and reaches below zero in 97.95% of surveys and lies entirely below zero in 3.35%. Coverage is not the problem. The problem is what the interval can say, and the honest summary of a typical survey is one number: the prevalence is below 1.64%. Bringing that limit down to twice the truth takes 184,521 people at this specificity and 5,083 with a test that never gives a false positive.

 Three summaries of a fitted line's geometry on Anscombe's four datasets, with each rule's threshold. Largest leverage: 0.318, 0.318, 0.318, 1.000 against a threshold of 0.364. Largest Cook's distance: 0.489, 0.808, 1.393, undefined, the point fits itself against 1 and against 0.364. Largest studentised deleted residual: 2.1, 2.2, 1192.2, undefined against a Bonferroni threshold of 3.90. What a summary of a scatter is a property of

Two numbers for the fit's geometry

No summary of the dependence between x and y can flag Anscombe's third and fourth datasets. A summary of the fit's own geometry can, and it takes two numbers of different kinds: the largest leverage, which reads the design before any y is seen and flags the fourth with certainty, and the largest deleted residual against a Bonferroni threshold, which flags the third at 1,192 and raises a false alarm on 4.78% of honest datasets. The rule of thumb most often quoted, Cook's distance over 4/n, flags the honest line too — and 66.45% of honest datasets like it.

 What a companion with a known expectation can remove from the variance of a simulated 5% rate. For a rate of 0.05, a companion used linearly removes at most 22.4% of the variance however well it correlates. At a correlation of 0.9 it removes 18.1%, the companion's own rate at the same cut 38.2%, and the best function of the companion 48.1%; at 0.99, 21.9%, 77.0% and 82.8%. The companion's rate overtakes the linear companion at a correlation of 0.64. A cut point, at a correlation

A rate needs a rate beside it

A simulated rejection rate is an average of indicators, and a companion with an exactly known expectation can remove at most φ(c)²/p(1 − p) of its variance when it is used as a straight line — 22.4% for a 5% rate, 7.2% for a 1% rate — however closely it tracks the statistic. The ceiling is the share of a normal's information a cut keeps. Used as its own rate at the same cut, the same companion removes 38.2% at a correlation of 0.9 and 77.0% at 0.99. On the t test's size for lognormal data it was worth 1.43 times the draws used linearly and 2.32 times used as a rate at a matched level.

 How often two units contain a break that is there. 300 draws at each of 5 break sizes, a shift from row 60 of 120 on first-order autoregressive errors. The share of draws on which the profile's two-unit set contains the true row: at the fixed, unwhitened window 12.0% at 0.3, 19.0% at 0.6, 31.7% at 1, 51.7% at 1.5, 69.3% at 2; with the window searched 22.7% at 0.3, 30.0% at 0.6, 44.3% at 1, 63.3% at 1.5, 75.3% at 2. A 95% interval would sit at the top of the frame. Two searches over one sample

A break that is there

Put a real break in the regression at row 60 and the profile's two-unit set still misses it on most draws — it contains the true row on 12% of draws at a small break and 69% at a large one, without whitening; with the window searched as well, 23% to 75%. A set that would contain it on 95% of draws needs a threshold of about 19 units without whitening and about 8.5 with it, and either honest set holds most of the sample: fifty-seven of the seventy-three candidate rows at a break of one standard deviation. Searching the window does not move where the break is found.

 Which factor the sample settles. The field's decomposition at the full tuning list, by sample size, over 1,200 draws at each: how often fifteen candidates choose different values, what share of those disagreements change which candidate the table selects, and the product — how often the list changes the winner. The window: 120 rows 43.3%, 28.5%, 12.33%; 240 rows 17.5%, 14.8%, 2.58%; 480 rows 38.3%, 2.6%, 1.00%; 960 rows 11.6%, 2.2%, 0.25%. The order: 120 rows 48.9%, 21.6%, 10.58%; 240 rows 47.5%, 13.2%, 6.25%; 480 rows 46.1%, 5.1%, 2.33%. The rate and the size of a disagreement

What a longer sample settles

At a hundred and twenty rows, letting each candidate choose its own whitening changes which of fifteen candidates wins on about an eighth of draws, whatever the list, the table or the law. At four hundred and eighty rows it changes the winner on 1.0% of draws for a window and 2.3% for an order, and at nine hundred and sixty on a quarter of one per cent. The sample settles it through one factor only: how often the candidates disagree about the tuning value hardly moves, and the share of their disagreements that change the winner collapses.

 What a lag of band costs, as the sample grows. The optimism one more lag of a Bartlett band adds, measured by fitting on one sample and scoring on an independent one, 2,400 pairs at each size, with two standard errors: 0.3745 ± 0.0176 at 120 rows, 0.3842 ± 0.0266 at 240 rows, 0.4523 ± 0.0392 at 480 rows, 0.4022 ± 0.0563 at 960 rows. The window's summed weights give one half a lag and the convention charges one. A charge for a covariance's own dimension

The charge at four times the sample

A band of lags is charged a full unit apiece and is worth about three eighths of one at a hundred and twenty rows. The field's last essay said the error in that convention was harmless only because the error surface is flat, and that a steeper surface would give it a cost. A longer sample was the obvious place for one. It is not there: from 120 rows to 480 the error's spread across band widths falls by three quarters, and the derived charge's advantage over the convention falls from 5.5 standard errors to 1.7. What the sample does move is the charge itself, from 0.37 of a unit a lag towards the half a unit the window's weights predict.

 The table swept along the covariate instead. What a rule holding a mean of each covariate fails to remove of their interaction, at each of 4 copulas, as the covariate is skewed further and the rank correlation is held at 0.4. The sweep runs from a symmetric covariate at g = 0 to a skewness of 11.16, and the three settings the earlier table names — g = 0.3, 0.6 and 0.9 — are on it, where this sweep reproduces that table to the last digit. Every row rises and then falls: the lower-tail copula from 7.707% through 0.0015% and back to 5.587%, the upper-tail one to a maximum of 36.213%. So the quantity a trial is exposed to is not monotone in how skewed its covariate is. The same table at seven correlations

The other dial

The table is swept along the strength of the dependence and never along the shape of the covariate. Swept along the shape at a fixed correlation, the same two copulas cross, the same way — and the near-zero cell turns out to be a minimum in both directions at once.

 The pairing recovers most of it and passes nothing. How much of the separating direction each probe carries, over 100 designs of 14 units whose admissible set is enumerated and split in two. The four rows the pairing adds are the dominant direction of what each blocking matrix keeps past its degrees, and the cut that direction's signs induce. Counted, they read 0.5266 and 0.5258 against the counted per-unit share's 0.3854 — most of the gap between that share and the design's own leverage at 0.5395, closed. Modelled, they read 0.4274 and 0.4954 against 0.4272. Nothing built from the active set passes leverage, and the projected fourth power is still ahead of all of them at 0.6583. A probe from what the rule blocks

A set of pairs, not a vector

The active set is a graph on the units, and every probe built from it so far has been its degree. Read as a graph it recovers 0.1326 of the alignment the summary lost — and draws level with leverage rather than passing it.

 Largest where least is needed. What the pairs correction supplies against what each window's measured profile needs, across this field's plateau, over 2000 draws at 120 rows. Both are stated as the multiplicative rise the charge per unit of width has to take between four lags and thirty. What the correction supplies is arithmetic — (1 − μ(4)/n)/(1 − μ(30)/n), where μ is the mean lag of the weight the band adds — and it runs 1.0795, 1.0580, 1.0456, 1.0539 for the four windows. What the measurement needs runs 1.1076, 1.2928, 1.2296, 1.6550. The two orderings are opposite: the plain Bartlett window has the longest mean lag, so it gets the biggest correction, and the flattest profile, so it needs the smallest. They coincide to 0.9746 of each other, and nowhere else does the correction account for more than 85.0% of the fall. A charge that is not a straight line

What the correction assumes

A correction with nothing fitted in it repairs one window of four. The reason is that its size is set by where a window puts its weight and the curvature it must repair is set by something else — and for one window at one sample size the two happen to agree.

 How long an honest forecaster looks broken for. The calibration error shown by a forecaster with no miscalibration in it at all, at five record lengths, drawn against one over the square root of the length so that the closed form is a straight line through the origin. It is 0.1252 at fifty forecasts and 0.0090 at ten thousand, against a closed form of √(2K/πn) times the mean root bin variance which gives 0.1257 and 0.0089. The threshold drawn across it is 0.02, a figure routinely read as evidence that something is wrong; the mean falls under it at 1976 forecasts and the 95th percentile at about 4111. Below that, an honest forecaster and a miscalibrated one are being told apart by a statistic that is mostly the sample size. A forecast that is a probability

The miscalibration a perfect forecaster shows

A forecaster whose true reliability is exactly zero shows a calibration error of 0.1252 on fifty forecasts and 0.0090 on ten thousand. Every one of 1,200 blameless hundred-forecast records exceeds the 0.02 routinely read as evidence of a problem, and the mean does not fall under it until 1,976 forecasts.

 The p-value of a study with 80% power, twenty thousand times. Twenty thousand two-sided z-tests, each on 25 observations whose true mean is 0.5603 standard deviations from the null, a noncentrality of 2.802. The bars are the counted share of p-values in bins a quarter of a power of ten wide, with the leftmost bin holding everything smaller; the line is the closed form. The middle eighty per cent of the p-values runs from 4.4×10⁻⁵ to 0.13, 3.46 orders of magnitude, the median is 0.0051, and 80.0% fall below 0.05, which is what the power means. Tests, and the second number

The p-value a replication gets

Under a true null a p-value is flat. Under a real effect its distribution is closed form and wide — a study with 80% power returns anything from 4.4×10⁻⁵ to 0.13 in eight runs of ten — and the chance that an exact replication of a p = 0.05 result is significant again is exactly one half, under both of the models people use without naming them.

 The risk of one cause, estimated two ways. Two causes of an ending event with constant hazards 0.2 (the one of interest) and 0.3 (the competitor), random dropout at 0.1 and follow-up to 6. The lower line is the cumulative incidence, (0.2/0.5)(1 − e^(−0.5t)), the chance of actually having had this event by t; the dots on it are the Aalen–Johansen estimate over 2000 studies of 300, 0.3670 at t = 5 against 0.3672. The upper line is 1 − e^(−0.2t), and the dots on it are one minus Kaplan–Meier with the competing event treated as censoring: 0.6318 at t = 5 against 0.6321. The second is larger by a factor of 1.722 at t = 5, and it is not an error of estimation. It estimates, correctly, the risk in a population where the competing cause does not exist. When the data stops early

One minus Kaplan–Meier is not a risk

With two ways for observation to end, one minus Kaplan–Meier for one cause reads 0.6318 at t = 5 where the chance of actually having had that event is 0.3670. Added across the two causes, the complements reach 1.4088 — more than the whole cohort. Nothing is estimated badly: the complement estimates, correctly, the risk in a world where the other cause does not exist.

 Forty trials at a true effect of 0.16, under the rule "power at the trend < 10%". The upper line is the benefit boundary (4.56, 3.23, 2.63, 2.28, 2.04); the lower line is where the rule stops a trial for futility (0.40 at 80, 0.66 at 160, 0.95 at 240, 1.31 at 320). Of forty trials with a real effect, 29 cross for benefit and 11 are stopped for futility. Stopping rules

A boundary for giving up

Adding "stop if z is below zero" to an O'Brien–Fleming trial costs 5.20 points of power at the effect it was designed for and halves the observations a trial with no effect uses. Stopping when conditional power at the observed trend falls under 10% costs 13.23 points and stops 21.28% of trials with a real effect. Making that rule binding lowers the benefit boundary from 2.040 to 1.901, and a binding rule that is then ignored rejects a true null 3.523% of the time instead of 2.5%.

 Storey's estimate of the share of true nulls over twenty thousand families, independent and correlated at 0.6. The true share is 0.5. Independent tests: mean 0.610, spread 0.160, below half the truth in 0.92% of families. Correlated at 0.6: mean 0.609, spread 0.240, below half the truth in 9.33%. Corrections, and what each controls

Estimating how many nulls are true

Benjamini–Hochberg at 5% delivers 2.55% when half of twenty nulls are false, because it cannot tell how many are. Storey's estimate of that share, read off the p-values above one half, spends the rest and finds 81.93% of the real effects instead of 74.70% on independent tests. Correlated at 0.9, the same procedure reports a finding in 19.29% of families in which every null is true.

 A whole trial's z against a subgroup's, the subgroup 20% of it with the same effect as the rest. Five hundred pairs, correlated at 0.447 — the square root of its share — because the subgroup is part of the whole. The whole trial is significant and the subgroup is not in 57.5% of trials; the subgroup differs significantly from the rest in 5.0%. The dashed lines mark 1.96 on each axis and the slanted lines mark a significant difference between the subgroup and the rest. An interval read beside something else

A subgroup inside its own trial

A trial with 80% power, significant overall and not in the fifth of it that is women, has told the reader almost nothing about women: that pattern turns up 57.45% of the time when women have the full effect and 58.26% when they have none. The subgroup is part of the whole, so the two verdicts are correlated, and the one test that answers the question — the subgroup against everyone else — can be recovered from the two printed intervals and nothing more.

 The band that allows at most r of the next m observations outside it, 95% of the time, from a sample of ten. At a hundred future observations the band that holds all of them has factor 4.942; allowing one outside, 4.289; two, 3.948; five, 3.378. The tolerance factors for 99% and 95% content are 4.445 and 3.382. The interval that holds observations, not a mean

A band allowed a few misses

From ten observations, a band holding every one of the next hundred with 95% probability reaches 4.942 sample standard deviations either side of the mean. Allowing one of the hundred outside brings it to 4.289; allowing five, to 3.378 — the 95%-content tolerance factor, 3.382, to within half a hundredth. The tolerance interval turns out to be the limit of a promise about a share of failures. And the misses arrive together: when this band misses once, it misses again 45.4% of the time, where independent misses at the same rate would do so 12.5%.

 Five upper limits for the mean of 15 observations from a exponential source, by margin and by failure rate. t limit: exceeded 8.51% at a mean margin of 0.523; Hall's transformation: exceeded 4.61% at a mean margin of 1.038; gamma family, shape known: exceeded 2.59% at a mean margin of 0.787; gamma family, shape by likelihood: exceeded 4.04% at a mean margin of 0.746; gamma family, shape by skewness: exceeded 17.51% at a mean margin of 0.421. The maximum-likelihood shape has a median of 1.10 and the skewness-based shape 3.20, against a true shape of 1. Shape, and what it does to a two-sample test

A limit from the family it came from

On fifteen exponential observations an upper limit for the mean built from the gamma family's own exact pivot, with the shape estimated by maximum likelihood, is exceeded 4.04% of the time at a mean margin of 0.746 standard deviations. Hall's transformation is exceeded 4.61% at 1.038, and the same family limit with the shape read from the sample skewness is exceeded 17.51%. Knowing the family and estimating its one parameter well recovers most of what knowing the population buys — and the parameter is estimated well only from the logarithms.

 What four adjusted analyses report about their own precision and what they have, 50 patients an arm, baseline correlated 0.7. Reported against actual variance of the treatment estimate, as shares of the unadjusted analysis's, over 20,000 trials of 50 patients an arm. Adjusted for the baseline as measured: 0.529 reported, 0.518 actual. Median cut fixed in advance: 0.716 and 0.700. The decile cut with the best fit: 0.671 and 0.736. The decile cut with the smallest p-value: 0.762 and 1.615. What a sample-size calculation was given

The cut that fitted best

A trial that adjusts for its baseline cut at whichever of the sample's nine deciles fits the outcomes best — without ever looking at the treatment difference — reports a variance 0.671 of the unadjusted analysis's where its estimates actually have 0.736, which is worse than the 0.700 a median fixed in advance delivers. The sample says the chosen cut kept 72.4% of what the baseline could remove; the population says it kept less than a median. A true null is rejected 5.87% of the time at fifty patients an arm and 9.31% at ten, and a cut chosen for its p-value rejects 14.22%.

 Post-stratifying a simulated 5% rate on a companion correlated 0.9: the share of its variance removed, by the number of strata. The best any function of the companion can do is 48.1%; its own rate at the same cut, 38.2%. Strata spent across the band where the conditional probability changes reach 95% of the best at 8 strata; equal-probability strata need 32. At eight strata the two remove 46.8% and 29.9%. A cut point, at a correlation

A companion cut into many

A simulated rate can be estimated within bands of a companion whose distribution is exact, and the bands weighted back by their exact probabilities. One band boundary at the target's threshold is the companion's own rate; infinitely many are the best any function of the companion can do, which for a 5% rate at a correlation of 0.9 removes 48.1% of the variance. Eight bands placed where the target's conditional probability is changing remove 46.8%; eight at equal probabilities remove 29.9%, and need thirty-two to catch up. On the lognormal t test's size, thirty-two bands of the logarithms' t statistic were worth 2.99 times the draws and two hundred and fifty-six, 2.14.

 How slow a return each test can see at 200 observations: the relation estimated against the relation supplied. The estimated relation's test reaches 80% power up to a half-life of 5.05 steps and the supplied relation's up to 6.88. At a half-life of 8 the two find the pair 36.3% and 71.9% of the time; at 12, 16.9% and 39.3%. Their 5% points are −3.38 and −2.89. Series that move together

A relation supplied in advance

When theory supplies the long-run coefficient, the gap between two series can be tested directly and the search the Engle–Granger test pays for disappears: its 5% point moves from −3.38 to −2.89, and at two hundred observations the slowest return found four times in five goes from a half-life of 5.05 steps to 6.88. The price is in what a rejection means. With a gap that halves in three steps, a coefficient a fifth too large is confirmed 99.1% of the time and one half again too large 81.9%, though every such gap contains a random walk — and at a thousand observations the second is confirmed more often, not less.

 How many of 100 points a fitted slope rests on, by the distribution the covariate was drawn from. The effective number of points behind a slope is n divided by the design's kurtosis. Over designs of 100, its median share is two points, half at each: 1.000; uniform: 0.551; normal: 0.349; exponential: 0.160; Student's t, 3 df: 0.148; lognormal, σ = 1: 0.076 — 34.9 of 100 points for a normal covariate and 7.6 for a lognormal one. What a summary of a scatter is a property of

The points a slope rests on

Before any outcome is measured, a regression's design already says how many of its points the slope will rest on: n divided by the covariate's kurtosis, exactly. A hundred points of a normal covariate carry a slope on about 35 of them; a hundred of a lognormal one, on 7.6. The largest leverage follows the same tail — above one half in 15.7% of lognormal designs of a hundred and in none of the normal ones — and on a correct straight line, deleting the single highest-leverage point moves the slope by more than its standard error in 16.3% of lognormal datasets of a hundred and 0.1% of normal ones.

 How far three estimates of the chosen arm sit from its true effect, with all eight alike. Eight arms of sixty, the best carried forward with thirty more. Average error: both stages, as published +0.123; stage two alone −0.001; stage one shrunk, then both +0.020. The arms' spread estimates to zero on 57.1% of trials. Designs that change while they run

Shrinking the arm that was chosen

A trial that carries the best of eight arms forward reports it as better than it is — by +0.123 when the eight arms are alike — and its second stage alone is unbiased but the least accurate of the estimates. Pulling the chosen arm's first stage back towards the eight arms' mean before combining removes most of the bias when the arms are alike (+0.020) and gives the smallest error of any estimate, 0.133 against the published 0.182. When one arm really is ahead by half a standard deviation, the same shrinkage pulls the true winner back towards the losers and understates it by 0.103, with more error than the published estimate. The weight is estimated from eight numbers, and it is right when the selection was noise and wrong when the selection was right.

 A threshold read along the skew. What a rule balancing a threshold at the value 1 on each covariate's own scale removes of the interaction between the thresholds, against the covariate's skewness, on five copulas at a Spearman rank correlation of 0.4. Gaussian: 16.41% with no skew, 10.07% at a skewness of 11.16; t on four: 22.18% with no skew, 12.11% at a skewness of 11.16; Frank: 13.19% with no skew, 10.14% at a skewness of 11.16; Clayton: 5.33% with no skew, 4.82% at a skewness of 11.16; Clayton turned over: 33.34% with no skew, 17.61% at a skewness of 11.16. Every leak falls, because the value moves towards the median. The same table at seven correlations

Where the skew puts a threshold

The field read the mean-balancing rule along a sequence of skewness and left the threshold rule — a cut at a fixed value on the covariate's own scale — unread. Read, it turns out to be no new curve. Skewing the scale moves the value 1 from the 84th percentile of the latent covariate to the 74th, and the rule's leak is the copula field's cut sweep read at that quantile: it falls on every copula, from 16.41% to 10.07% under a Gaussian. Whether skew and dependence compound or cancel is then a fact about the slope of each copula's curve, and unlike the mean rule's, no cell changes its answer along the sequence.

 What balancing a recorded number does to the true one. The imbalance three rules leave in the true covariate X when what they read is a recording W of it, at six reliabilities and n = 200, 600 trials each, as a share of a coin's exact 2/√n. The lower curve is √(1 − λ), the closed-form floor for any rule that reads W: X = λW plus a residual of variance 1 − λ that W says nothing about, and a coin is all any rule can be on that residual. The rule that reads the number sits on it — 0.125 at 1, 0.345 at 0.9, 0.452 at 0.8, 0.612 at 0.64, 0.718 at 0.5, 0.834 at 0.3 — while balancing W itself to 0.125–0.133 of a coin's throughout. The upper curve is √(1 − 2λ/π), the same floor for minimisation on a median split of W, and minimisation sits on that one. At λ = 2/π the rule that reads the number is exactly as good as a median split of an error-free covariate: a cut is a measurement with reliability 0.637. Balancing what has no levels

A number read with error

A rule that balances the recorded covariate balances the true one only as far as the recording reaches. At a reliability of 0.8 it leaves 45% of a coin's imbalance in the covariate that drives the outcome, back on the coin's 1/√n rate — and a median split turns out to be a recording with reliability 2/π.

 A basis drawn rather than chosen. The share of the worst of seven shapes a balancing rule removes, over two covariates and a dictionary of twenty-four functions, for the best fixed basis of each size (lower line) and for a basis drawn at random from the least favourable mixture (upper line), at one to four functions. The fixed basis guarantees 0.0000, 0.0000, 0.4244, 0.6366: nothing at one function or two, because two of the shapes are pure interactions and two of them are main effects of different covariates, and no one or two functions touch all of them. The drawn basis guarantees 0.2042, 0.4084, 0.6133, 0.7920 in expectation over the draw — about a fifth of a shape for every function up to three. The marks above the upper line are the best reply to nature's mixture, 0.2051, 0.4098, 0.6142, 0.7931: the game's value lies between the two. When the set is too large to walk

A basis drawn rather than chosen

On two covariates no fixed basis of one or two functions protects every shape an outcome might have — its worst case is exactly zero. Draw the basis at random from the right mixture and the guarantee is a fifth of a shape for every function. The price is per trial — at three functions, two draws in five leave one shape wholly unprotected.

 The sums, pooled as they are and made independent. Coverage on four designs at a requirement of 0.9, 20,000 runs each, about 3.0, 3.6, 3.4, 3.0 blocks. Refusing the sums: 94.91%, 95.06%, 94.96%, 94.86%. Pooling them as they are: 94.98%, 93.67%, 96.17%, 94.93% — exact at balance, short when the allocation is unbalanced and the same in every block, long when it alternates. Pooling the residualised sums, each block's sum less the part of it its own difference predicts: 94.98%, 95.16%, 95.06%, 94.70%, on every design. Each bar is ±0.31%. A promise about two arms

The sums made independent

The block sums are a free second copy of a two-arm trial's variance only at balance. With ten and two in every block, pooling them covers 93.67% on four blocks; swap the arms each block and it covers 96.17%. Take from each sum the part its own difference predicts and the copy is independent at every allocation.

 How often each reading rejects when every null is true. At the optimal combination of eight forecasts, where all eight encompassing nulls are exactly true, the share of 1,200 draws on which each reading rejects at 5%, at 60, 120, 240, 480 origins. The joint Wald test against a chi-square on seven degrees of freedom: 26.6%, 16.4%, 9.8%, 8.6%; with Hotelling's finite-sample F: 15.1%, 12.4%, 8.3%, 8.0%; the largest of the eight read as a single comparison: 25.0%, 25.9%, 24.7%, 23.4%; the largest with Bonferroni over eight: 5.2%, 4.9%, 4.7%, 4.4%. Searching among fitted models

One test for eight nulls

A combination of eight forecasts can be tested against its parts one comparison at a time or all at once. All at once is a Wald test on the vector of eight encompassing means — which has seven degrees of freedom, not eight, because the means sum to zero on every origin when weighted by the combination. On sixty origins it rejects a true null 26.6% of the time, Hotelling's correction brings that to 15.1%, and handed its own true critical value it is less powerful than Bonferroni over the largest of the eight against both wrong combinations measured: 12% against 18%, and 47% against 96%.

 The displacement per parameter, under five predictor laws. The out-of-sample differential one extra parameter costs a correct candidate, at a window of 80 rows, by two routes that share no arithmetic: counted from rolling forecasts over 1,500 draws of the fifteen-candidate table, and implied by the expected leverage of a fresh row over 80,000 draws of the predictors alone. The Gaussian closed form is 0.01386. normal: 0.01382 counted, 0.01386 from the leverage; Student's t, 8: 0.01379 counted, 0.01407 from the leverage; Student's t, 5: 0.01419 counted, 0.01448 from the leverage; Student's t, 3: 0.01751 counted, 0.01780 from the leverage; exponential: 0.01465 counted, 0.01487 from the leverage. A search with no fixed point

The displacement on a heavy tail

A correct candidate with one more parameter is behind out of sample by a closed-form amount, and the closed form rests on one Gaussian fact: the expected leverage of a fresh row is q/(R − q − 1). On predictors that are not Gaussian it becomes whatever the leverage is, and two routes that share no arithmetic agree on what that is. With a finite fourth moment it barely moves — 1.5% more per parameter at eight degrees of freedom, 4.5% at five, 7.3% on an exponential. At three degrees of freedom it is 28% more, and the estimate of it will not settle: four batches of twenty thousand draws disagree by more than a quarter.

 Which side the misses fall on, skewed shocks, 95.0% intervals. With skewed shocks and the model's parameters known, the share of 10,000 forecasts that fall below and above a nominal 95.0% normal interval, at each horizon: 1 ahead, 0.0% below and 5.2% above; 2 ahead, 0.0% below and 4.9% above; 4 ahead, 0.0% below and 4.6% above; 8 ahead, 0.0% below and 4.2% above; 12 ahead, 0.0% below and 4.9% above. The observation that has not happened

Shocks that are not normal

Every forecast interval in this field reads a normal quantile, and every series it was scored on had normal shocks. Give a first-order autoregression shocks with heavy tails and the 95% interval still covers about 95%; its 99% interval covers 98.2%. Give it skewed shocks and the 95% interval covers 94.8% one step ahead with every miss on one side — 0.0% below, 5.2% above — and the horizon does not repair it: twelve steps ahead the misses are still 0.0% below and 4.9% above, because a persistence of 0.7 keeps the error's skewness at 1.11 however far out the forecast goes. A two-sided coverage count cannot see any of this.

 Six intervals for the white predictor, a five-period moving average. Under a five-period moving average, over 200 regressions of 120 rows with an eight-lag band, how often each 95% interval for the white predictor's coefficient covers it: two-step band, as reported 95.5%; joint fit, as reported 72.0%; joint fit, White's sandwich 71.0%; joint fit, four-lag sandwich 71.0%; joint fit, band maximised again 85.5%; the law's own covariance 92.0%. The bootstrap redraws each regression 24 times from its fitted coefficients and fitted band, maximising the band afresh on every redraw. A covariance with no parameter

An interval that knows the band was maximised

A regression whose coefficients and eight-lag error band are maximised together prints an interval that covers 72% to 83% where 95% was promised, because it reports nearly the precision of a known covariance and delivers less. Two repairs were named. A sandwich standard error read off the whitened residuals repairs nothing — 71% to 83% — because the maximisation has already made those residuals look independent. A parametric bootstrap that redraws from the fitted band and maximises the band again on every redraw repairs about half the gap, to 85.5% to 89%, by pricing the maximisation through repeating it. What it cannot price is that the band it redraws from is the maximised one.

 The same dial, on a list that steps by one. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations, for both tuning parameters at a matched list length of 8, with errors from a five-period moving average. The sieve order runs from 29.5% to 5.8%, a factor of 5.13; the whitening window, from 26.3% to 4.8%, a factor of 5.53. What is held is the number of options, the table, the law, the sample size and the seeds; what cannot be held is the size of a step, since an integer step and a geometric step are different amounts of change. The dial moves both, and it moves them by 1.08 times as much on one as on the other. What decides whether a tuning list decides

The same dial under a second law

On a first-order autoregression, separating the candidates moved the chance that a tuning list changes the winner 2.35 times as far on a whitening window as on a sieve order, and the difference was put down to the step: an integer step is worth a third of a geometric one. Rerun the whole sweep with errors from a five-period moving average and the difference is gone — the probability spans a factor of 5.53 on the window and 5.13 on the order, a ratio of 1.08 — and near the null both lists leave about five and a half candidates within reach, where the autoregression left 3.6 and 1.9. The two factors still point opposite ways under both laws. The step's size was a property of the law the lists were tuned on.

 Power as the recentring's threshold moves. The recentred set test's power at a persistence of 0.65, where one forecaster genuinely beats the benchmark, over 300 draws of 100 origins, as its threshold runs from zero to three times Hansen's. With the eight windows alone: 30.3%, 28.3%, 26.0%, 26.0%, 26.0%, 26.0%. With sixteen stale candidates added: 30.3%, 28.3%, 24.3%, 21.0%, 19.3%, 17.7%. The studentised reality check, which sets nothing aside, has 26.0% and 17.7%. The best of a set, and what the search costs

The threshold the recentring reads

A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.

 Where two profile likelihoods put the break. Under a break in the persistence at row 60 of 120, and the same break turned over, over 300 draws: how often each search's estimate lands within twelve rows of the truth, with its average and spread beside it. persistent first · one scale for the whole series: 46.7%, at 74.8 with a spread of 11.8; persistent first · a scale per regime: 94.3%, at 61.8 with a spread of 5.8; persistent last · one scale for the whole series: 49.3%, at 45.3 with a spread of 11.7; persistent last · a scale per regime: 93.0%, at 56.7 with a spread of 5.9. Paying for a search

The break a shared scale hid

The break searches in this field all maximised a likelihood with one innovation variance for the whole series, and on a break from 0.95 to 0.65 they found it badly: an average of 74.8 where the truth is 60, within twelve rows on 47% of draws, from a profile everyone called flat. Give each regime its own innovation variance — one more parameter, and the one the law actually has — and the same search lands at 61.8 with a spread of 5.8, within twelve rows on 94% of draws, reporting four times the likelihood. The break was never barely there; the likelihood was blind to the half of its evidence that is in the scale. Under no break the extra parameter costs about one and a half units of manufactured likelihood.

 Past the cap, long memory's surface is flat. The prediction error at every band width from four lags to eighty, above the lowest on the extended list, under long memory at d = 4/9 at three sample sizes, 600 draws each. At 120 rows the floor is at 12 lags and eighty is 0.0039 above it. At 480 the floor is at 30 lags and every width from twenty-four to eighty is within 0.0003 of it. At 960 it is at 48 and thirty to eighty lags are within 0.0000. The dashed line is the list's cap: the surface does not rise behind it, it levels off. A charge for a covariance's own dimension

A cap that decides nothing

Every band-width list this field priced stopped at thirty lags, and long memory at a large sample is the one dependence whose best band wants to be wider. At 960 rows it does: the draw's own best width lies past the cap on 48.7% of draws. Lift the cap to eighty and the measured charge follows, from 28 lags to 46 — and the error moves by two ten-thousandths, in the wrong direction and inside its standard error, because the surface behind the cap is flat. The field's last conditional fails where it had its best chance.

 When a longer sample settles the tuning list. How often the tuning list changes which candidate the table selects, against the sample size, for the whitening window (solid) and the sieve order (dashed), on three tables of candidates, twelve hundred draws at each size. Every subset of four predictors: window 12.33% at 120, 2.58% at 240, 1.00% at 480, 0.25% at 960; order 10.58% at 120, 6.25% at 240, 2.33% at 480. The nested sequence of four: window 5.83% at 120, 1.25% at 240, 0.67% at 480, 0.25% at 960; order 5.83% at 120, 4.00% at 240, 2.25% at 480. Six subsets of two predictors each: window 11.75% at 120, 3.75% at 240, 5.75% at 480, 2.33% at 960; order 9.42% at 120, 6.67% at 240, 4.42% at 480. The nested table starts at half the full table's rate and reaches a small one a sample size sooner; the table whose candidates all have the same dimension is still changing its winner on one draw in forty-three at 960 rows. The rate and the size of a disagreement

The table decides when the sample settles it

Letting each candidate regression choose its own whitening changes which candidate wins on about an eighth of draws at 120 rows, and a longer sample settles it — for fifteen nested-and-crossed subsets, at 0.25% by 960 rows. A nested sequence of four candidates gets there a sample size sooner, from half the rate. Six candidates that all hold two predictors never get there: at 960 rows their tuning list still changes the winner on 2.3% of draws, nine times the full table's, because a disagreement among candidates of one size goes on deciding the winner as often at 960 rows as at 240.

 How often each set holds the break. For a shift added to the response from row 60 of 120 on first-order errors at 0.8, 300 samples at each of five sizes and 120 simulations per sample, the coverage of sets for the break row built at a threshold each method supplies. simulated at the truth: 95.0%, 93.7%, 93.7%, 95.0%, 94.0%; estimated break, true persistence: 90.0%, 88.3%, 88.3%, 89.3%, 90.3%; estimated break, persistence matched: 81.0%, 83.3%, 81.3%, 81.3%, 82.3%; estimated break and persistence: 64.0%, 66.0%, 67.7%, 70.7%, 75.7%; a fixed threshold of 21 units: 96.0%, 94.0%, 96.0%, 97.3%, 98.3%; the two-unit set: 12.3%, 19.7%, 35.0%, 49.0%, 66.7%, at shifts of 0.3, 0.6, 1, 1.5, 2. Simulated at the truth the construction is exact; simulated at what one sample can estimate it misses the true row a quarter to a third of the time. Two searches over one sample

A set calibrated at the estimate

A confidence set for where a regression breaks needs a threshold on the profile likelihood, and no fixed one works: the honest threshold falls as the break grows. Simulating the profile's deficit at the true break, shift and persistence gives a set that covers 93.7% to 95.0% at every break size and narrows from 67 rows to 23. Simulating it at what one sample can estimate gives a set that covers 64% to 76%, and most of the shortfall is one number: a split regression's residuals report a persistence of 0.62 where the errors have 0.8.

 The floor a repair has to choose. Regret of the truncated covariance repaired by flooring its eigenvalues, at a persistence of 0.85 on 120 rows, 120 draws, by floor, on a log scale. Window 4: 1.5649 at 0.001, 0.0945 at 0.01, 0.0385 at 0.05, 0.0285 at 0.1, 0.0226 at 0.2, 0.0235 at 0.4, 0.0292 at 0.7; the taper at the same window 0.0275. Window 8: 1.1919 at 0.001, 0.0980 at 0.01, 0.0334 at 0.05, 0.0258 at 0.1, 0.0197 at 0.2, 0.0209 at 0.4, 0.0249 at 0.7; the taper at the same window 0.0178. Window 12: 0.8692 at 0.001, 0.0621 at 0.01, 0.0239 at 0.05, 0.0209 at 0.1, 0.0212 at 0.2, 0.0239 at 0.4, 0.0282 at 0.7; the taper at the same window 0.0150. Window 20: 1.7110 at 0.001, 0.2495 at 0.01, 0.0399 at 0.05, 0.0238 at 0.1, 0.0202 at 0.2, 0.0219 at 0.4, 0.0259 at 0.7; the taper at the same window 0.0135. Least squares gives up 0.0890 and the whitening told the truth 0.0075. Estimating the dependence, not naming it

A covariance repaired

A truncated autocovariance sequence fails to be a covariance matrix on most draws of a persistent series, and this line of essays has always scored it on the draws where it survives. The textbook repair is to floor its eigenvalues. At a floor of 0.001 that moves the matrix by four per cent and gives up 1.56 of regret, seventeen times what least squares gives up, because a whitening reads an inverse and an inverse reads the smallest eigenvalues. At a floor of 0.2 it beats the taper at the shortest windows. So the repair is not a repair: it is a second dial.

 Which window each reading prefers. The difference between the tapered and the rectangular window, in points, on five readings at four rules: for the two errors a negative number favours the taper, and for the two coverages a positive one does. implied, variance: 1.5, 4.5, -1.1, -2.1. 95% point, uncentred: -6.1, -3.2, -7.1, -6.1. 95% point, centred: 0.9, 3.5, -0.8, -1.0. coverage, uncentred: 5.0, 2.5, 5.8, 4.0. coverage, centred: -1.2, -3.2, 1.0, 0.0. The two uncentred readings favour the taper throughout, and they are the only ones that disagree with the variance. The block length read on a quantile

A shrink towards the answer

This line of essays reported that the 95% point is a dearer reading than a variance, that a further point wants a shorter block, and that a taper beats a rectangle on what a test reads where a variance says the opposite. All three came from one uncentred quantile. Read about the resamples' own mean, the point costs 28.5% against the variance's 45.8%, every level wants about the same block, and the windows order alike on all readings. The taper's coverage had a separate cause: its resampled mean is 0.875 of the sample's, and every series here had a true mean of exactly zero.

 Reading the draw changes what is charged, not what is tracked. The correlation between the band width each rule picks and the best band width on the same draw, over 400 draws. The three fixed charges read -0.069, -0.066, 0.012. The three that read the sample read -0.012, -0.019, -0.041. None of the six is distinguishable from nothing. The statistic the first plug-in reads does vary — the draw's own summed squared autocorrelation runs from 2.06 to 10.19 with a mean of 4.36 — so the failure is not that the charge stopped moving. It is that what it moves with carries no information about which width this draw wanted. A charge that is not a straight line

A charge that reads the draw

Three charges built to read the sample track the best band width on their own draw at −0.012, −0.019 and −0.041, deliver more error than the fixed rule they are calibrated to, and pick a width half again as variable. The statistic moves; the answer does not.

 One treatment, a different hazard ratio at every follow-up. The hazard ratio a Cox model converges to, found as the root of its expected score by numerical integration, as the trial runs longer; dropout at 0.1 throughout. The proportional treatment reads 0.5 at every τ. The waning treatment reads 0.5000 at τ = 1, 0.6362 at τ = 3 and 0.7890 at τ = 8 — the same two arms, the same effect in the same first year, and a number that drifts towards one as later, effect-free events are added to the average. The dots are the mean of 400 Cox fits with 400 subjects an arm: 0.5014 at τ = 1, 0.7020 at τ = 2, 0.7635 at τ = 3, 0.8034 at τ = 5, 0.8208 at τ = 8. The crossing treatment reads 0.3429 at τ = 1, exactly 1 at τ = 3 by construction, and 1.0611 at τ = 8: beneficial, null or harmful according to when the trial stopped. When the data stops early

The hazard ratio the follow-up chose

A treatment that halves the hazard for one year and then does nothing has a Cox hazard ratio of 0.5000 if the trial stops at one year, 0.7617 at three and 0.8194 at eight. Nothing about the treatment differs between those numbers. When hazards are not proportional the hazard ratio is an average, and the length of follow-up and the dropout rate choose its weights.

 Three intervals as one strength is spread thinner, at a concentration of 8. Coverage of three nominal 95.0% intervals on the same 1000 draws of 200 rows at each count, when a total concentration parameter of 8 is spread over 1 to 32 instruments. Two-stage least squares covers 97.2%, 96.4%, 94.0%, 86.7%, 73.2%, 51.5%. Building each row's fitted treatment from a first stage that never saw that row covers 97.1%, 97.2%, 98.3%, 97.8%, 97.9%, 98.7%. Limited-information maximum likelihood with its conventional standard error covers 97.2%, 96.7%, 95.7%, 90.9%, 85.2%, 79.0%. At one instrument the likelihood estimator is two-stage least squares exactly, which is why the first readings of those two agree to the last draw. A variable that moves one thing only

Leaving each row out of its own first stage

Spread a fixed first-stage strength over thirty-two instruments and two-stage least squares covers 51.5%. Build each row's fitted treatment from a first stage that never saw that row and the same draws cover 98.7% — through an interval 5.99 times as wide, around an estimate that misses by more than the whole effect on 34.7% of draws. At eight times the strength the same repair covers 95.3% and costs a width factor of 1.66.

 One curve, and two forecasters that are each a single point. The ROC curve of an honest forecaster whose signal has correlation 1 with the latent state — every threshold on its probability, from the bivariate normal — and two forecasters that only ever say 0 or 1. The one an absolute-error score pays for says 1 wherever the honest probability exceeds a half: it has a true-positive rate of 0.6742 and a false-positive rate of 0.1375, and its "curve" is the two straight segments through that point, with area (TPR + TNR)/2 = 0.7684. The honest curve's area is 0.8683. Thresholding at the base rate of 0.3744 instead of at a half gives the largest area any two-valued forecaster can have here, 0.7818, and it is still 0.0866 short. A forecast that is a probability

The liar with two answers

The forecaster an absolute-error score pays for says only 0 or 1, and on the ROC square it is a single point: its area is (TPR + TNR)/2 = 0.7684, against the honest forecaster's 0.8683, and it falls below the honest one on 200 of 200 counted records. No relabelling of its two answers returns what it threw away — the best recovers a Brier score short of the honest one by exactly the 0.022154 of resolution lost — and below a signal correlation of 0.7332 the same score prefers saying no every time to an honest forecast.

 What each group gains from being pooled, τ = 1. Eight groups whose sizes span a factor of 13.3. The smallest gains 2.076 of squared error, which is 36.8% of the total reduction; the largest gains 0.030, which is 0.5%. 2 of the eight account for half of everything pooling buys. Groups that borrow

Where the borrowing goes

Pooling cuts the total squared error across eight groups by 56%. Two of the eight take 61% of that reduction, the four best-measured groups share 11% between them, and the largest group gets 1.5% of what the smallest does. The headline is a fact about the groups nobody was asking about.

 What each wrong count costs, 4 steps ahead. Squared forecast error 4 steps ahead at each imposed rank, relative to the correctly specified fit, at 200 observations. With 1 genuine relations, imposing 0 costs 13.3% and imposing 2 costs 4.8%. With 2 genuine relations, imposing 1 costs 15.6% and imposing 3 costs 2.5%. Under-counting is the more expensive mistake in both systems, and it is the one the procedure's level does not bound. Three series, and a count

Which mistake about the rank costs

On a system with two relations, imposing none costs 29.2% of squared forecast error and imposing three costs 2.5%. The expensive mistake is under-counting, which is the error the procedure's 5% does not bound — so the guarantee protects the cheap side.

 Three contrasts on one dataset, three different splits. The variance-minimising allocation for each of three ways of reporting the same two-arm comparison, against the first arm's proportion, with the second at 0.1. A risk difference wants the arm with the larger p(1 − p) to get more units; a log odds ratio wants it to get fewer, and the two curves are exact reflections of each other in the half line. A log risk ratio wants something else again. At a first-arm proportion of 0.6 they ask for 62.0%, 21.4% and 38.0% of the units. A trial reporting more than one of them cannot be optimal for either. Splitting the units

Two contrasts, one split

A risk difference wants 62.0% of the units in the first arm, a log risk ratio wants 21.4% and a log odds ratio wants 38.0% — on one dataset, with one pair of proportions. The difference's rule and the odds ratio's are exact reflections of each other, so no split can be near-optimal for both.

 Expected loss of three upper limits against the level each is set at, fifteen exponential observations, penalty 20. A failure costs 20 standard deviations of margin. At the conventional 97.5% the t limit's loss is 2.143, Hall's 1.922 and the fitted family's 1.530. Each at its own best level: the t limit 1.263 at 99.95%, Hall's 1.862 at 98.50%, the family 1.308 at 99.25%. The best fixed multiple of the standard error costs 1.255. Shape, and what it does to a two-sample test

The level a limit should be set at

When a failure costs twenty times a standard deviation of margin, a t limit on fifteen exponential observations set at the conventional 97.5% has an expected loss of 2.143. Set at 99.95%, the same formula's loss is 1.263 — within 1% of the best any fixed multiple of s/√n can do, and ahead of both Hall's transformation and the fitted gamma family at their own best levels. Hall's is the worst at every level on every source. Choosing the level does more than choosing the construction, and the correction that reads the sample is the one no level rescues.

 The variance of the baseline-adjusted estimate for four arrangements of visits, against the correlation between visits. At a correlation of 0.5 between visits, the adjusted analysis has 0.750 of the unadjusted variance with one visit before treatment and one after, 0.667 with two before, 0.500 with two after and 0.417 with two of each. What a sample-size calculation was given

A visit before or a visit after

A trial that can afford one more measurement per patient can take it before treatment, to sharpen the baseline adjustment, or after it, to average the outcome. With every pair of visits correlated 0.5, the second visit after treatment removes a quarter of the variance and the second visit before it removes a twelfth, and it is never the other way round: the baseline's contribution is ρ²(1 − ρ)/(1 + ρ), which peaks at ρ⁵ = 0.090 when ρ is the golden ratio's reciprocal. Below a correlation of one half, a single extra follow-up beats any number of baselines. When the correlation fades with the time between visits, averaging an earlier baseline into the adjustment makes the trial less precise, not more.

 How slow a shared return a panel of pairs can see at 200 observations each. The average of the pairs' Engle–Granger statistics, against its own 5% point. One pair reaches 80% power up to a half-life of 5.1 steps, four pairs 9.2, sixteen 15.0. At a half-life of 12 the three find the relation 20.0%, 54.2% and 99.8% of the time. Series that move together

A slow return across many pairs

One pair of two hundred observations is found tied four times in five only if its gap halves within 5.1 steps. Sixteen independent pairs sharing the same half-life, tested by the average of their Engle–Granger statistics, are found four times in five up to a half-life of 15.0 — and at a half-life of twelve, 99.8% of the time against one pair's 20.0%. What the widened test rejects is a statement about the panel: with half the pairs tied it rejects 55.4% of the time, with a quarter 21.2%, and a pair-by-pair reading of a panel in which all sixteen are tied finds 2.65 of them, against 0.78 when none are.

 Where a thousand validation studies of 100 diseased and 300 healthy put the threshold, at a prevalence of 26%. The best threshold flags 100.0% of healthy people. Reading the study's own operating curve, 7.7% of studies recommend flagging nearly everyone, and the recommended thresholds cost 15.4% more than the best on average; fitting a normal curve to each group's scores first, 47.4% do, at an average excess of 2.5%. Two tests, a threshold, and the rate they are read against

The threshold a validation study can see

When diseased scores are twice as spread as healthy ones, the harm-minimising threshold jumps near a prevalence of 26% from an interior cut to flagging everyone, because the true operating curve keeps climbing in a tail where few diseased people score. A validation study of a hundred diseased and three hundred healthy cannot see that tail: its own curve reaches 100% sensitivity at its lowest diseased score and goes flat. Just past the jump, only 7.7% of such studies recommend flagging everyone, and their recommended thresholds cause 15.4% more harm than the best on average. Fitting a normal curve to each group first sees the tail: 47.4% of studies then recommend flagging everyone and the excess harm falls to 2.5%. More healthy people in the study change nothing; more diseased people do.

 How often a trial of 80 shows a Simpson reversal on its strongest baseline variable, against how many cut points are tried. The treatment helps every patient equally. Cut at its median, the variable's table reverses in 4.4% of trials; trying 2, 3, 5, 9 cuts, in 6.4%, 8.7%, 10.8%, 13.2%. Independent chances would give 33.3% at nine. When the stratified answer and the pooled one disagree

Five places to cut one variable

A continuous baseline variable has to be cut before it makes a subgroup table, and the cut is chosen. In a randomised trial of eighty where the treatment helps every patient equally, the strongest baseline variable cut at its median shows a Simpson reversal in 4.4% of trials; tried at five candidate cuts, some cut reverses in 10.8%, and at nine, 13.2%. Nine correlated cuts are worth about five independent chances, not nine. Across twenty continuous variables cut at five points each, some table reverses in 32.2% of trials — twice the rate with each variable cut once, and about thirty-one independent chances where the table holds a hundred.

 How often a trial whose effect is the same for everyone shows a significant subgroup interaction, by the size of its subgroup table. Four hundred patients, 80% power overall, characteristics whose latents correlate 0 (their split indicators 0.000). Some interaction test reaches 5% in 5.10% at 1, 9.13% at 2, 17.43% at 4, 34.63% at 8, 47.13% at 12, 56.23% at 16; the equicorrelated-normal integral gives 5.00%, 9.75%, 18.55%, 33.66%, 45.96%, 55.99%. A Bonferroni reading of the eight holds 5.57%. An interval read beside something else

Sixteen subgroups and one effect

A trial of four hundred with 80% power tabulates its result by eight binary characteristics — sixteen subgroups — and the treatment's effect is the same for every patient. Among trials significant overall, 93.5% report at least one subgroup where it is not, and the median one reports seven. Some subgroup-against-the-rest interaction reaches 5% in 34.6% of trials — eight independent chances — and when the characteristics overlap as two age thresholds do, 18.6%, about four. The effective count is set by the correlation between the characteristics' split indicators, (2/π) arcsin of their latent correlation, and a Bonferroni reading of the eight holds the rate at 5% whatever the overlap.

 How often a quantile plot's shape test sees a skewed error distribution, on the errors and on residuals of fits of more columns. On the forty errors themselves the test flags 82.9%. On the residuals of fits with 2, 5, 10, 20 columns: 79.1%, 71.4%, 51.2%, 23.0%. On 40 − p errors, which is what losing the fitted columns' worth of observations alone would give: 80.6%, 76.1%, 69.1%, 49.5%. What a diagnostic plot is showing

Residuals that look too normal

The quantile plot most often read is of a regression's residuals, and residuals look more normal than the errors behind them. Each residual is a mix of all the errors, and the mix moves towards a bell as the fit takes more columns. On forty rows with skewed errors, the straightness test flags 82.9% of samples of the errors themselves, 79.1% of the residuals of a line, 71.4% with five columns, 51.2% with ten and 23.0% with twenty — far below the 49.5% that twenty errors would give, so the fit costs more than the observations it uses. Light tails vanish faster: 73.0% on the errors, 4.7% on twenty-column residuals. The band's level holds at every design, and one point with leverage 0.73 costs almost nothing; what costs is the number of columns.

 How many observations a band from a sample's extremes needs to let at most r of the next 100 outside, 95% of the time, for any population. From the sample's smallest and largest: none of the next 100 outside needs 3,850 observations, at most one 637, at most five 107 and at most ten 50. From the second smallest and second largest, 7,750, 1,204, 187 and 85. The interval that holds observations, not a mean

A range allowed a few misses

Without assuming a shape, the band a sample can offer is its own smallest and largest values, and the number of future draws falling outside it is beta-binomial for every continuous population. A promise that at most one of the next hundred lands outside, with 95% probability, then needs 637 observations; none of the hundred, 3,850; at most five, 107. The normal-theory band keeps the one-in-a-hundred promise from ten. On normal data, at the sizes the range needs, it is only 6% wider than the normal band at the same size — the price of dropping the assumption is paid in observations, not in width — and its misses cluster far less than a band from ten: a second follows a first 19.7% of the time, against 45.4%.

 A point that decides a coefficient, on two covariates correlated at 0.9. A hundred points on two normal covariates with correlation 0.9. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the first coefficient's information, 14.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 26.1 effective points, with a variance inflation of 6.27. What a summary of a scatter is a property of

A point ordinary on every axis

In a multiple regression each coefficient rests on its own number of points, n divided by the kurtosis of that covariate's residual after the others, exactly. Correlation between normal covariates inflates a coefficient's variance fivefold and leaves it resting on the same third of the points. What correlation does instead is hide the point that decides the coefficient: at a correlation of 0.9 it stands out on no single covariate's plot in 77.3% of designs of a hundred, and the largest leverage a normal design reaches depends on the number of points and covariates and on nothing else.

 How far four estimates of the chosen arm sit from its true effect, with one arm ahead by 0.5. Eight arms of sixty, the best carried forward with thirty more, 20,000 trials. Average error: both stages, as published +0.006; shrunk under a normal prior −0.103; shrunk under a t prior, 3 df −0.101; normal prior with a fifth vague −0.073. Root-mean-square error: 0.151, 0.189, 0.192, 0.176. Designs that change while they run

A prior that lets one arm stand out

Shrinking the chosen arm of eight under a normal prior understates a genuine winner by 0.103. A t prior on three degrees of freedom with the same spread understates it by 0.101 — the tail is too far out to reach a lead of three and a half standard errors. A prior with a fifth of its weight on a vague component understates it by 0.073, costs 0.010 of bias when the arms are alike, and still has more error than the unshrunk estimate. With sixteen arms the same mixture recovers three fifths of the loss, because a bell of fifteen shows its outlier; with eight, the arms put more than half their posterior weight on 'one stands out' in 0.4% of trials where one does.

 What two exact companions are worth to a simulated rate: the lognormal t test's size post-stratified three ways, by the number of cells. To first order, bands of the logarithms' t statistic are worth at most 2.77 times the draws; a 32 × 32 grid in the pair 4.68; bands of a score fitted to the pair on a thousand samples 4.64 at sixteen bands and 4.69 at sixty-four. Counted over runs of 4,000 samples, the grid of 1,024 cells was worth 1.97, with 94 cells a run left empty. A cut point, at a correlation

A companion with two coordinates

The logarithms' t statistic, cut into bands, is worth at most 2.77 times the draws to a simulation of the lognormal t test's size. It is one function of two exactly known statistics, the logarithms' mean and spread, and the pair is worth 4.64 when a score fitted to it on a pilot of a thousand samples is cut into sixteen bands whose probabilities are integrated exactly. A grid of cells in the pair needs a thousand cells to reach the same and, counted on runs of four thousand samples, is worth 1.97. Band weights estimated from a pilot instead of computed are worth less than the pilot's draws would have been had they simply been counted — exactly nothing at a pilot the size of the run.

 Separation, how far against whether. The median separation each probe carries between the pieces of a split admissible set, over the 100 designs whose set is split. The share refused per unit carries 1.953; the refusal pairing's direction 3.103, and with the saturated pairs dropped 2.917. The depth pairing's direction carries 2.345, and each unit's mean depth 3.836 — exactly the leverage's. A random direction carries 0.942 and the projected fourth power 5.080. A probe from what the rule blocks

How far, not whether

The blocking matrix records whether a tolerance box refuses an exchange. Dropping the pairs it refuses for certain changes nothing. Recording instead how far each exchange moves the balance vector turns the pairing into its degree plus a part that lies inside the rule's own span — and the degree is the design's leverage, exactly. The heuristic that beat every reading of what the rule blocks was the unclamped pairing's degree all along.

 Three charges and two counts. The optimism of an in-sample fit of the widest candidate on 120 rows, in units of its own error variance, over 1500 draws, by persistence. Counted on the 40 rows straight after the sample: 11.1, 21.5, 35.6, 51.5, 65.5, 87.8, 119.1. Counted on rows 600 further on: 9.3, 18.3, 33.0, 50.6, 67.0, 96.1, 159.0. Charged by the long-series inflation: 10.0, 19.3, 34.7, 54.4, 74.3, 114.2, 234.1. Charged by the same inflation summed over the sample's rows: 10.0, 19.2, 34.2, 53.1, 71.8, 108.2, 208.8. Charged by twice the trace of the hat matrix against the errors' correlation on the sample's own design: 10.0, 18.7, 32.0, 47.5, 61.5, 85.2, 132.2. Scoring a search without spending data

A charge the sample can afford

The inflation charge for dependent rows runs to twice the counted optimism at persistence 0.95, and the reason offered was that a series of n rows cannot be optimistic about movements slower than n rows. Summing the inflation over n rows does almost nothing: 234 becomes 209, against 119 counted. Evaluating the same variance on the sample's own design gives 132, and moving the fresh rows away from the sample raises the count to 159. So the charge can be made nearly right, and when it is, the regret doubles from 0.032 to 0.061. A selection wants a much larger charge than the optimism, and past about fifty per predictor it is no longer selecting.

 Which side the misses fall on. Misses of a nominal 95% forecast interval 1 step ahead on 60 observations of a first-order autoregression at 0.7, split by side. normal shocks: the interval as printed 3.1% below and 3.1% above, shocks from the residuals 3.8% below and 4.2% above, the same, coefficient re-estimated 3.3% below and 4.4% above. Student's t on five: the interval as printed 3.3% below and 3.0% above, shocks from the residuals 3.9% below and 3.0% above, the same, coefficient re-estimated 3.6% below and 3.1% above. skewed shocks: the interval as printed 0.0% below and 6.5% above, shocks from the residuals 4.3% below and 4.1% above, the same, coefficient re-estimated 1.8% below and 4.3% above. The observation that has not happened

A tail sixty residuals have barely seen

A forecast interval that reads a normal quantile has every one of its misses on one side when the shocks are skewed, and the natural repair is to draw future shocks from the model's own residuals. It works on the side: 0.0% below and 6.5% above becomes 4.3% and 4.1%. It pays for that with coverage, 93.5% to 91.6%, and on sixty observations it does this under normal shocks too. And a nominal 99% interval has a ceiling nothing in the resampling can lift, because one new shock beats all of sixty old ones in either direction with probability 3.3%.

 Coverage as the effect moves with the level. Coverage on ten and two in every block, with the level between blocks moving at a spread of 0.5 and the effect moving with it at each slope, 4000 runs each. At -3: refusing the sums 94.4%, pooling them 88.8%, residualised 97.0%; At -2.5: refusing the sums 94.3%, pooling them 88.9%, residualised 95.6%; At -2: refusing the sums 94.9%, pooling them 90.0%, residualised 93.8%; At -1.5: refusing the sums 95.1%, pooling them 91.7%, residualised 92.7%; At -1: refusing the sums 95.1%, pooling them 94.8%, residualised 94.0%; At -0.5: refusing the sums 95.1%, pooling them 97.2%, residualised 97.7%; At 0: refusing the sums 95.0%, pooling them 98.8%, residualised 99.6%; At 0.5: refusing the sums 95.0%, pooling them 99.4%, residualised 99.9%; At 1: refusing the sums 95.0%, pooling them 99.4%, residualised 99.9%. A promise about two arms

The sums under a moving level

A trial's block sums are a free second estimate of its variance only while they see what the differences see, and a level that moves between blocks with an effect moving against it breaks that: pooled raw at a ceiling they cover 87.6% to 90.0%. Residualising the sums on an unbalanced design does not make them safe. It moves the failure. On ten and two, 90.0% at the ceiling becomes 93.8%, the trough leaves for a slope of about −1.5, and over the rest of the range the same estimate over-covers, at 99.9% on an interval half as wide again as it needs to be.

 Coverage lost against the honest rule. Coverage lost in points against the rule that reads only within-block spreads, on the same observations, blocks of 2, 6000 runs, at half-widths 0.5, 0.35, 0.25, 0.15. the reported spread in full: 0.32 ± 0.27, 2.53 ± 0.37, 5.33 ± 0.41, 3.75 ± 0.35. a fixed half: 0.72 ± 0.26, 2.43 ± 0.29, 0.98 ± 0.26, -0.25 ± 0.20. κ = (b − 1)/(N − 1): 0.37 ± 0.21, 1.18 ± 0.25, 0.58 ± 0.23, -0.18 ± 0.19. the same, phased in as b/(b + 20): 0.22 ± 0.10, 0.35 ± 0.16, 0.23 ± 0.18, -0.10 ± 0.19. the same, read in only after twenty blocks: 0.00 ± 0.00, -0.02 ± 0.04, 0.17 ± 0.19, -0.17 ± 0.19. The block size as a schedule

A rule that sets its own κ

A stopping rule may read the spread inside its blocks and may not read the spread between them, and the dial between the two costs nothing for a long stretch and then a lot. The share an efficiency argument recommends is each estimate's share of the degrees of freedom, and on blocks of two at a half-width of 0.15 that loses nothing and steadies the experiment by 30%. At 0.35 it loses 1.18 points. Deferring the forbidden reading until twenty blocks costs nothing measurable at any requirement, and it keeps most of the steadiness where steadiness can be had.

 Four rules and what each leaves. The share of a coin's imbalance each rule leaves in the true value of each covariate, at 200 units and a reliability of 0.8. one reading of X: 21.4% in X and 100.1% in Z; two readings of X: 12.3% in X and 105.0% in Z; one reading each of X and Z: 21.7% in X and 21.5% in Z; two of X and one of Z: 13.4% in X and 22.0% in Z. The floors in closed form are 20.0% for one reading and 11.1% for two. Balancing what has no levels

A second reading or a second covariate

A rule that balances a recording balances the truth only as far as the recording reaches, and a second reading of the covariate raises that reach. So does a first reading of something else. At a reliability of 0.8 the first reading of a second covariate does more once that covariate matters a third as much as the first, and the threshold is exact: the square root of (1 − λ)/(1 + λ), with no trial size in it. A protocol's list of measurements turns out to be a small knapsack, and it is rarely filled by reading anything twice.

 What the interval covers as the effect stops being one number. Coverage at 120 units under minimisation, 600 trials and 199 re-run allocations each, as the effect's slope on the prognosis rises from 0 to 1.5. the inverted test, unadjusted, for the population's mean: 96.7%, 96.2%, 95.7%, 95.7%, 94.8%. the same, for the sample's mean: 96.7%, 96.0%, 96.2%, 97.3%, 98.3%. the inverted test, adjusted, for the population's mean: 96.5%, 95.5%, 95.0%, 95.3%, 96.0%. the same, for the sample's mean: 96.5%, 95.7%, 96.5%, 98.0%, 99.5%. the regression with an interaction, for the population's mean: 96.2%, 95.5%, 94.0%, 91.0%, 84.0%. the same, for the sample's mean: 96.2%, 95.8%, 96.2%, 95.7%, 95.2%. Balancing on what was recorded first

An effect that is not one number

A randomisation test becomes an interval for the effect by testing every constant effect in turn, so when the effect grows with the prognosis every hypothesis it tests is false. It does not fall apart. It covers the population's mean effect within a point of its level at every slope measured, and its width grows by exactly what the sample's own composition adds. A regression that models the interaction stays narrow at 0.72, covers the sample's mean at 95%, and covers the population's at 84%. The extra width was never a defect. It was the difference between two questions.

 The guess rate through a trial. Guess rates in each tenth of a trial of 150 patients on three arms with the rule taking its preferred arm with probability 0.8, over 1500 trials. knowing everything: 65.9%, 72.2%, 72.0%, 72.8%, 72.4%, 72.2%, 72.5%, 72.7%, 72.7%, 72.4%. a joint table learned in earlier trials: 53.4%, 55.4%, 54.8%, 55.1%, 54.3%, 53.8%, 54.9%, 54.8%, 54.1%, 54.8%. the centre's totals: 45.4%, 50.3%, 50.2%, 50.8%, 49.8%, 50.2%, 50.9%, 50.6%, 50.4%, 50.7%. following whichever has been right: 46.5%, 50.2%, 49.3%, 49.6%, 48.9%, 48.7%, 49.3%, 49.0%, 49.0%, 48.7%. a joint table learned in this trial: 36.4%, 43.8%, 45.0%, 46.2%, 46.7%, 46.8%, 48.1%, 48.5%, 47.9%, 48.9%. the centre's own factors, scored: 46.8%, 51.4%, 49.7%, 49.9%, 48.4%, 48.6%, 48.5%, 47.7%, 47.6%, 46.7%. Chance is 33.3%. More arms than two

The guesser who learns

An investigator who watches a three-arm minimisation can tally which of several partial readings has been right so far and follow the leader. It does not help. Following the leader names 48.9% of assignments where the best single reading names 49.9%, and a table of what the three readings say together, learned inside the trial, starts at 36% and has not caught up by a hundred and fifty patients. Learned over earlier trials, that table reaches 54.5%. The full guesser's 71.8% stays out of reach of all of them, because the gap to it is information rather than practice.

 What a floor costs a drawn basis. The expected guarantee of a basis drawn from a mixture restricted to bases that each remove at least a stated share of every shape, solved at each distinct floor. 2 functions, of 276 bases: 40.8% at a floor of 0.0%; the best fixed basis floors at 0.0%. 3 functions, of 2024 bases: 61.3% at a floor of 0.0%, 50.0% at a floor of 31.8%, 50.0% at a floor of 34.7%, 49.7% at a floor of 40.5%, 42.4% at a floor of 42.4%; the best fixed basis floors at 42.4%. 4 functions, of 10626 bases: 79.2% at a floor of 0.0%, 77.1% at a floor of 31.8%, 77.1% at a floor of 34.7%, 77.1% at a floor of 35.1%, 77.1% at a floor of 35.6%, 77.1% at a floor of 37.0%, 77.1% at a floor of 38.3%, 77.1% at a floor of 40.5%, 76.5% at a floor of 41.0%, 76.5% at a floor of 42.4%, 76.5% at a floor of 42.8%, 76.5% at a floor of 42.9%, 76.5% at a floor of 43.5%, 76.5% at a floor of 44.7%, 76.5% at a floor of 45.3%, 76.5% at a floor of 46.0%, 76.5% at a floor of 46.8%, 76.5% at a floor of 47.2%, 76.5% at a floor of 50.0%, 76.5% at a floor of 56.0%, 76.5% at a floor of 63.7%; the best fixed basis floors at 63.7%. When the set is too large to walk

A floor for the draw

A basis drawn at random guarantees more on average than any fixed one, at the price of leaving some shape wholly untouched in a large share of trials. Requiring every basis in the draw to keep a floor looks like a smooth trade between the two. It is a staircase, and the first step does nearly all the work. No basis has a floor between zero and 31.83%, so there is nothing to give up until a floor of that size is asked for. And at four functions, seven bases tie at the fixed basis's own 63.66%: a mixture of them keeps that floor exactly and adds thirteen points to the guarantee.

 What a stated power leaves of the main effects. The main-effect efficiency of the best design reaching a power of 0.8 at level 0.05, as N|δ|² rises through 50, 60, 70, 80, 100, 130, 170, 220, 300, 400, 600, 1000. a saddle, two-degree test, the compound path: out of reach, out of reach, 0.806, 0.841, 0.881, 0.914, 0.937, 0.952, 0.966, 0.975, 0.984, 0.990. a bowl, two-degree test, the compound path: 0.782, 0.823, 0.851, 0.871, 0.898, 0.923, 0.941, 0.955, 0.967, 0.976, 0.984, 0.990. a bowl, its own one-degree contrast, the compound path: 0.827, 0.859, 0.880, 0.896, 0.918, 0.937, 0.952, 0.964, 0.973, 0.980, 0.987, 0.992. a bowl, a factorial with centre points: 0.914, 0.930, 0.940, 0.948, 0.959, 0.969, 0.976, 0.982, 0.987, 0.990, 0.993, 0.996. The criterion, and what it assumes

A design chosen for a test

A screening experiment that wants its main effects and a check on curvature has been told to trade them with a weight nobody can state. It can state a power. Read against a power of 80%, the frontier collapses to one curve in runs times the squared size of the curvature, and a large enough trial pays almost nothing. But the curve belongs to a test. A factorial with a few centre points beats the entire frontier on a bowl, spending 4% of its main-effect information where the frontier spends 10%, and against a saddle of the same size it has exactly its 5% level.

 What Hansen's threshold costs, by the number of origins. The power a threshold of zero keeps over Hansen's, on the same draws. with sixteen stale candidates: 7.5%, 7.3%, 6.8%, 5.3%, 3.7%, 6.0%. the eight windows alone: 2.7%, 3.8%, 4.7%, 5.3%, 3.7%, 6.0%, at 25, 50, 100, 200, 400, 800 origins. The price does not fall as the sample grows. The best of a set, and what the search costs

A threshold against its sample

A recentred test's threshold exists so that the test is right in the limit, and at a hundred origins it costs a fifth of the power against a stale set. The natural defence is that the price falls as the sample grows and the guarantee starts to earn it. Across twenty-five to eight hundred origins it does not: a threshold of zero keeps four to seven points more power at every size, and its size at the corner stays at the level from fifty origins on. What does change with the sample is the stale set, which stops costing anything by two hundred origins under Hansen's rule and by eight hundred under the most cautious.

 The cut-off lag, before and after the correction. Coverage at 120 rows when the block length is chosen from a share read off the sample's autocovariances to a stated lag, as read and corrected for the centring. first-order autoregression, as read: 93.9% at lag 6, 93.3% at lag 12, 88.7% at lag 24, 84.5% at lag 40. the same, corrected: 94.0% at lag 6, 94.5% at lag 12, 91.9% at lag 24, 89.2% at lag 40. moving average, as read: 94.5% at lag 6, 92.3% at lag 12, 88.7% at lag 24, 85.0% at lag 40. the same, corrected: 94.7% at lag 6, 93.7% at lag 12, 91.7% at lag 24, 89.8% at lag 40. a break, as read: 92.1% at lag 6, 93.1% at lag 12, 90.1% at lag 24, 83.5% at lag 40. the same, corrected: 92.2% at lag 6, 93.8% at lag 12, 92.2% at lag 24, 89.7% at lag 40. A block, weighted inside itself

A sequence that knows it was centred

A block length read off a sample's own autocovariances gets worse the further out it reads, because a centred series' autocovariances sum to exactly zero. To first order every lag loses the same amount, about the long-run variance over the sample size, and that can be added back. Corrected, a reading to lag forty goes from 84.5% to 89.2%, and to lag twelve from 93.3% to 94.5%. But the correction overshoots on the long lags, and on the rule that picks its own cut-off, which is short, it moves coverage by three tenths of a point. The identity was the reason the lag mattered. It was not the reason the rule falls short.

 Every finding Benjamini–Hochberg made in thirty families, with its interval, at real effects of 2. 85 findings, sorted by their estimate. 12 of their ordinary 95% intervals miss the true effect, every one of them on the far side; 6 of the wider false-coverage-rate intervals miss. Corrections, and what each controls

Intervals for the findings

Benjamini–Hochberg's findings usually go out each with its ordinary 95% interval. With ten real effects of two standard errors among twenty tests, 11.59% of those intervals miss their effect, every miss on the far side, and the interval around the most prominent finding covers 72.36% of the time — 2.38% when the effects are one standard error. Intervals widened for the number of findings hold the share that miss under 5%.

 The honest curve, and the same forecaster in three coarse vocabularies. The ROC curve of the honest probability at full signal, area 0.8683, beside the same forecaster rounded to the nearest whole number, area 0.7684 — the two-valued liar — to the nearest half, area 0.8205, and to the nearest tenth, area 0.8650. A vocabulary of v values is v points on the square joined by straight segments, and tied reports count half. A forecast that is a probability

A forecaster that rounds

An honest probability issued in tenths loses 0.0033 of ROC area and 0.000708 of resolution — the variance its bands average away, and 89.5% of the 0.000792 it adds to the Brier score. Two hundred records of two thousand forecasts show that loss on 189; it takes about 3,300 forecasts to put it two standard errors from zero. And 3.207 in every thousand forecasts in tenths are a 0% on an event that happened, which a logarithmic score charges without limit.

 Three intervals as one strength is spread thinner, at a concentration of 8. Coverage of four nominal 95.0% intervals on the same 1000 draws of 200 rows at each count, when a total concentration parameter of 8 is spread over 1 to 32 instruments. Two-stage least squares covers 97.2%, 96.4%, 94.0%, 86.7%, 73.2%, 51.5%. Building each row's fitted treatment from a first stage that never saw that row covers 97.1%, 97.2%, 98.3%, 97.8%, 97.9%, 98.7%. Limited-information maximum likelihood with its conventional standard error covers 97.2%, 96.7%, 95.7%, 90.9%, 85.2%, 79.0%. The same estimate with Bekker's many-instrument standard error covers 97.2%, 97.2%, 97.2%, 95.0%, 94.3%, 93.8%. At one instrument the likelihood estimator is two-stage least squares exactly, which is why the first readings of those two agree to the last draw. A variable that moves one thing only

A standard error that knows about the instruments

Limited-information maximum likelihood came out least biased when a concentration parameter of 8 was spread over thirty-two instruments, and its conventional interval covered 79.0%. Bekker's many-instrument standard error covers 93.8% on the same draws, at 63% of the jackknife's width — and it gets there with a median standard error of 0.561 against a true spread of 0.797, because it is large on the draws that need it. At eight times the strength it covers 94.9% at 91% of the jackknife's width, and nothing measured here beats it.

 What the forecast interval is short by, φ = 0.85, 6 steps ahead. The plug-in interval covers 88.42% against a claimed 95%. Correcting the variance recovers 0.56 points, propagating the persistence's own standard error recovers 0.40, correcting the persistence recovers 2.66, and all three together recover 4.20 — leaving 2.38 points unaccounted for. Comparing two forecasters

What the interval is short by

The forecast interval covers 88.42% where it claims 95%. Correcting the persistence recovers 2.66 points, correcting the innovation variance 0.56, propagating the persistence's own standard error 0.40 — and all three together recover 4.20 of the 6.58, leaving a residual none of the standard repairs reaches.

 How fast a gap has to close before a sample can see it close. The power of the test against the half-life of a disagreement, at 100, 200, 400 observations, each read against its own simulated critical value. Every pair in every reading is genuinely tied together, so a non-rejection is a miss. At 200 observations a gap that halves in 3 steps is found 99.9% of the time and one that halves in 12 steps is found 15.3% of the time — and by 35 steps the reading is 6.1%, which is the test's own size. Beyond that the curves are flat because there is nothing left to detect with. When the observations repeat each other

How slow a return a sample can see

At two hundred observations the test finds a gap that halves in five steps four times in five, one that halves in eight 37.3% of the time, and one that halves in fifty 4.95% of the time — which is the rate at which it finds pairs with no mechanism at all. The boundary moves with the sample, not with its square root.

 One estimator, three answers, and only the reference changes. Coverage of the cluster-robust 95% interval for the slope against the number of clusters, at 30 rows in each. The estimator is identical in all three curves; what differs is the number it is compared against. At 5 clusters it covers 75.05% against a normal, 85.30% against a t on 4 degrees of freedom and 87.95% against a t on 3. At 80 clusters the three agree to within a point. The correction costs nothing: the same standard error, a different table. A standard error for a model that is wrong

The reference the sandwich is read against

The cluster-robust interval covers 75.05% at five clusters and 93.58% at eighty. The same estimate read against a t on G − 2 covers 87.95% at five, and the estimator is unchanged — three hundred rows grouped into five clusters cover 74.28% where the same three hundred grouped into seventy-five cover 94.63%.

 A proxy removes less than its reliability, always. The share of the confounding bias removed by adjusting for a proxy, against how well the proxy measures the confounder. The diagonal is the answer a reader would guess — a covariate that is 80% signal removes 80% of the problem. The curve is what the arithmetic gives: the reliability, times one minus the squared correlation between the treatment and the confounder, divided by one minus the product of those two. That squared correlation is 0.4475. A reliability of 0.8 removes 68.85% and one of 0.6 removes 45.32%. The two agree only at the ends, and the gap is widest where most applied covariates sit. What conditioning on a variable does

Adjusting for a shadow

A covariate that is 80% signal removes 68.85% of the confounding, not 80% — the share is λ(1 − ρ²)/(1 − λρ²) and it is below the reliability everywhere. The residual bias is 0.1084 against an effect of 0.5, and at 25,600 rows it is 17.6 standard errors wide.

 Three promises, and no procedure keeps all three. Average coverage and worst-case coverage for four 95% intervals for a proportion at n = 40, computed exactly. Their expected widths are 0.2418, 0.2417, 0.2472, 0.2641 in the same order. The textbook interval and the score interval have the same expected width to four digits — 0.2418 and 0.2417 — and worst-case coverages of 55.31% and 92.21%. The exact interval never breaks its promise and is 9.3% wider than the score interval to do it. Each of the three columns orders the four procedures differently. Intervals, counted

An interval that covers and says nothing

A procedure returning the whole line 95% of the time and the empty set otherwise has coverage exactly 95% at every parameter value. Two real intervals at forty observations have expected widths of 0.2418 and 0.2417 and worst-case coverages of 55.31% and 92.21%.

 Two companions on one simulation, two hundredfold apart. How many times as many draws each companion is worth, on the same 4,000 simulated samples of 40 observations. The coverage of the interval is estimated with the observed count as its companion, whose expectation is 12 exactly; they correlate at 0.2665 and the companion is worth 1.08 times the draws. The expected width is estimated with p̂(1 − p̂) as its companion, whose expectation is 0.20475 exactly; they correlate at 0.9977 because the width is a monotone function of it, and the companion is worth 214 times the draws — 856 thousand simulated samples' worth of precision from four thousand. What makes it checkable

The check worth more than the check

The same exactly known companion that verifies a simulation can sharpen it. On one set of four thousand draws, one companion is worth 1.08 times the draws and another is worth 214 times them, and the factor is 1 − ρ² with nothing else in it.

 Each interval covers one question and not the other. Coverage of each interval for the overall mean, scored against both estimands, over 20,000 two-site studies of 10 observations apiece. The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and over-covers the two sites in hand at 98.25%. Both are correct; they are answers to different questions printed in the same place. Hierarchy past one number

What a two-unit study should report

The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and is 11.6 times wider.

 How many starting observations a range needs to flag at most r of the next 100, fixed at the start or learning as the lot is used. 95% of the time, for any continuous population. At most 0: fixed 3,850, learning 3,850; At most 1: fixed 637, learning 512; At most 2: fixed 300, learning 197; At most 3: fixed 190, learning 101; At most 5: fixed 107, learning 36; At most 7: fixed 74, learning 15; At most 10: fixed 50, learning 4. With no flag allowed the two are the same, because a lot with no flag never moves the learning range. The interval that holds observations, not a mean

A range that grows with the lot

A band set from the range of everything seen so far flags a unit only when it sets a new record, and for any continuous population those flags are independent, with the k-th unit flagged with probability 2/(n + k). That makes a few-misses promise far cheaper than the fixed range's — at most five of the next hundred from 36 starting observations rather than 107 — and exactly as expensive when no miss is allowed, because a lot with no flag never moves the band. The saving is bought by believing every flag. After a two-standard-deviation step in the mean, the fixed range flags 25.03 of the eighty units that follow and the learning range 3.62; by the last twenty units it sees a unit outside in 25.5% of lots, against 18.4% with no step at all.

 How often the pooled test finds a relation in a panel with none, when the pairs share a shock. Against the 5% point computed for independent pairs, over 1,200 panels of pairs of 200 observations with no pair tied. Sixteen pairs, raw: 4.6% at a common share of 0, 7.2% at a common share of 0.25, 15.5% at a common share of 0.5, 27.3% at a common share of 0.75, 31.6% at a common share of 0.9; after removing the cross-sectional averages: 5.3%, 5.1%, 5.8%, 5.3%, 4.8%. Four pairs, raw: 5.4%, 6.5%, 6.8%, 12.6%, 16.9%. Series that move together

A shock every pair shares

Sixteen pairs of series whose shocks are half common are worth 5.37 independent pairs to the panel's average statistic, and sixteen that share nine tenths are worth 1.44. Read against the 5% point computed for independent pairs, the pooled test then calls a panel with no relation tied 15.5% and 31.6% of the time; read against its own point, it finds a relation shared by every pair 22.3% of the time at nine tenths common, against one pair's 20.0%. Removing each date's cross-sectional average gives back the level and all sixteen pairs when every pair feels the shock equally, and leaves 13.3% false alarms when the loadings run from a half to one and a half.

 How often one genuine interaction in a table of eight is found, by its size, at latent correlation 0. A trial of 400 with 80% power for its main effect, eight characteristics, the first modifying the effect. At h = 0, 0.25, 0.5, 0.75, 1 the real interaction crosses 5% in 4.2%, 10.3%, 29.2%, 56.6%, 81.0% of trials; Bonferroni's 2.734 in 0.4%, 1.8%, 8.7%, 26.6%, 53.6%; the exact threshold for this overlap, 2.727, in 0.4%, 1.9%, 8.8%, 27.0%, 53.8%. An interval read beside something else

One characteristic that really matters

A trial of four hundred tabulates eight characteristics and one of them genuinely changes the treatment's effect — the whole effect on one side, none on the other. Tested alone at 5% the real interaction is found 81.0% of the time; at the Bonferroni threshold that protects the table, 53.6%. It is the table's largest interaction in 82.4% of trials when the characteristics are independent and in 54.4% when they overlap like nested age bands, and in that case another characteristic crosses the Bonferroni line in half of all trials and is the top finding in a quarter. A family-wise threshold counts false rows; it cannot say which of two overlapping rows is the cause.

 How often a Simpson reversal appears on the strongest baseline variable when its cut is slid along it, by the trial's size. The treatment helps every patient by the same six points. At 40, 80, 160, 320, 640 patients: some cut in the middle 80% reverses in 18.8%, 23.4%, 24.3%, 19.2%, 10.6% of trials; some decile in 12.2%, 13.2%, 14.1%, 11.0%, 6.2%; the median in 3.9%, 4.4%, 4.2%, 3.1%, 1.7%. When the stratified answer and the pooled one disagree

A cut that slides

Slide the cut on a trial's strongest baseline variable along every gap between observed values in its middle eighty per cent, and a trial of eighty whose treatment helps everyone equally shows a Simpson reversal somewhere in 23.4% of trials — against 13.2% for the nine deciles and 4.4% for the median. Sixty-five cut positions are worth about 9.5 independent cuts, and at 640 patients, with 513 positions, about 7.2: a table that moves one patient at a time is a continuous scan, and its count is set by the range scanned rather than by the number of cuts. The reversing cuts spread along the whole range, in 2.62 separate runs at eighty patients and 8.36 at 640.

 Whether residuals built to be independent see a skewed error better than ordinary ones. The straightness test, each version read at its own 5% point. On all forty errors: 82.9%. At 2, 5, 10, 20 columns: ordinary residuals 79.1%, 71.4%, 51.2%, 23.0%; BLUS residuals 76.4%, 64.9%, 38.3%, 16.2%; recursive residuals 64.4%, 49.5%, 25.9%, 9.1%; 40 − p errors 80.6%, 76.1%, 69.1%, 49.5%. What a diagnostic plot is showing

Residuals made independent

Recursive residuals — each row predicted from the fit to the rows before it — are exactly independent normals under the null for any design, so a quantile plot of them has an exact band. They see a skewed error worse than the ordinary residuals do, not better: 25.9% against 51.2% at ten columns on forty rows, 9.1% against 23.0% at twenty. BLUS residuals, the independent set closest to the errors, reach 38.3% and 16.2%. Every such set is a rotation of the ordinary residuals, so none holds information they lack, and independence is bought by mixing some residuals almost entirely out of other rows' errors — the least pure recursive residual at twenty columns is 0.4% its own.

 The excess harm of the thresholds validation studies recommend, when the diseased scores are no lower tail. A thousand studies of a hundred diseased and three hundred healthy people, prevalence 0.26, a miss costing a hundred false alarms. The best threshold flags 7.6% of healthy people. Excess harm per hundred people, and the share recommending that everyone be flagged: the study's own curve 16.26 (0.0%); a binormal fit 67.08 (89.7%); a two-normal mixture fit 22.71 (2.3%); binormal if normality passes 16.26 (0.0%). Two tests, a threshold, and the rate they are read against

A test whose tail is not normal

A validation study that fits a normal curve to each group saw the tail where a threshold jumps — when the diseased scores really were normal. Give the same published test a diseased group with no lower tail at all, and the binormal fit invents one: 89.7% of studies recommend flagging everyone, adding 67.08 false alarms' worth of harm per hundred people to a best threshold that costs 5.61. Give it a lump of mild cases instead, and the binormal fit misses the tail that is there and does worse than the study's own curve. A normality test on the hundred diseased scores sees the missing tail every time and the lump 60.4% of the time, and a study that fits the binormal model only when normality is not rejected is never the worst of the four.

 The extra cost of each follow-up schedule over the cheapest, across every correlation curve with adjacent visits at 0.7, when a patient costs 5 visits to recruit. With one baseline, 1 follow-up costs at most 28.0% more than the cheapest schedule, 2 follow-ups cost at most 11.1% more than the cheapest schedule, 3 follow-ups cost at most 24.1% more than the cheapest schedule, 4 follow-ups cost at most 34.8% more than the cheapest schedule, over every stable share from 0 to 0.7. The schedule whose worst case is smallest is 2 follow-ups; its worst case is at a stable share of 0.00. Under compound symmetry the cheapest is 3; under a stable share of 0.4 it is 1. What a sample-size calculation was given

The correlation a pilot can see

Two models of repeated visits agree that adjacent visits correlate 0.7 and disagree about how many follow-ups a trial should buy: three under equal correlation, one when the correlation fades. Planning on the wrong one costs 28.0% or 9.9%. Two follow-ups cost at most 3.2% more than the best under either, and at most 11.1% under any curve with that adjacent correlation. A pilot of twenty patients at four visits chooses worse on average than that rule when the correlation fades; it takes about forty to draw level with it, and at an effect of half a standard deviation those forty patients' extra visits cost fifty times what the better schedule saves.

 A point that decides an interaction, on two covariates correlated at 0. A hundred points on two normal covariates with correlation 0. Dashed lines mark where the most extreme five per cent of each covariate begin. The point carrying the largest share of the interaction's information, 10.2%, is inside the lines on both axes, so neither covariate's plot on its own would show it. The coefficient rests on 19.0 effective points. What a summary of a scatter is a property of

A term built from the others

A regression's squared terms and interactions are columns built from its covariates, and each coefficient on one rests on the number of points divided by the kurtosis of what the built column adds — fifteen for the square of a normal covariate, nine for the product of two uncorrelated ones, three for a treatment's interaction with a normal covariate, which is the same as the covariate's own coefficient. A design of a hundred rests on more than the population says, 12.2 points behind a square against 6.7, because a hundred draws rarely hold the tail that sets the kurtosis. Centring changes nothing about the square's coefficient, to the last digit. And correlation between two covariates hides the point that decides each one's own coefficient while it reveals the point that decides their interaction.

 The error of the chosen arm's estimate against a fixed second stage's size, and a second stage sized by the interim, 8 arms, one ahead by half a standard deviation. Fixed second stages of 10, 25, 40, 55, 70 patients: both stages combined err by 0.172, 0.155, 0.143, 0.133, 0.126; the mixture-shrunk estimate by 0.208, 0.183, 0.164, 0.151, 0.140. The rule that sizes the second stage by the interim spends 41.4 on average and errs by 0.142 and 0.159; its second stage alone errs by 0.225. Designs that change while they run

A second stage sized by the first

A drop-the-losers trial can size its second stage by what the first stage says about the arms' shape — more patients when the chosen arm's lead looks like luck, fewer when it looks real. The rule does what it was built to do: at sixteen arms with a genuine winner it spends 51.0 patients a group when the interim chose the wrong arm and 24.2 when it chose the right one. And it buys nothing. Against a fixed second stage of the same average size, every estimate of the chosen arm errs by the same to within a few thousandths; the published combination's bias rises from 0.014 to 0.022, and the second stage alone, still unbiased, errs by 0.319 against 0.281.

 Expected loss of three upper limits against the level each is set at, when a failure is charged by its size, fifteen exponential observations, penalty 20. A failure costs 20 times the distance, in standard deviations, by which the mean exceeds the limit. At the conventional 97.5% the t limit's loss is 0.728, Hall's 1.136 and the fitted family's 0.835. Each at its own best level: the t limit 0.726 at 97.00%, Hall's 0.805 at 90.00%, the family 0.735 at 92.50%. Shape, and what it does to a two-sample test

A failure charged by its size

Charge an upper limit's failure by how far the true mean exceeds it rather than by a fixed amount, and the level a limit should be set at falls further than any other change in this comparison moves it: on fifteen exponential observations, a penalty of twenty per standard deviation of shortfall asks for a t limit at 97% where a penalty of twenty per failure asked for 99.95%. The failures are small whichever limit produces them — about an eighth of a standard deviation — so a charge by size is a small charge per failure. The ranking hardly moves: the t limit stays within about 1% of the fitted family, whose failures are no smaller than its own, and Hall's transformation stays last by 11%.

 Four laws, one flat floor. The error a band of each fixed width delivers, averaged over 400 draws of 120 rows under each law, drawn above that law's own best single width. From one lag to the best width the error falls by 0.02763 under AR(1) at 0.8, 0.02517 under a five-period moving average, 0.01122 under long memory at d = 4/9, 0.03507 under a break in the persistence; from six lags to thirty the whole surface spans only 0.00269, 0.00445, 0.00255, 0.00521. The best single width is 12, 24, 24, 16 lags. The flattest surface end to end is the one under long memory at d = 4/9, which is the law a steeper surface was expected from. A charge that is not a straight line

The surface under every law

A draw-reading charge for a covariance band failed on one law, and the failure was deferred to a law with a steeper error surface. Run under all four laws in the collection, the surface is flat from six lags to thirty under every one, no draw-reading charge tracks anything, and long memory — the law expected to be steepest — is the flattest end to end. What the law moves is the level of the charge, not whether a draw can be read.

 Approximate weights against the size of the run. What sixteen bands of the logarithms' t statistic are worth to a simulation of the lognormal t test's lower-tail size, in multiples of the draws, against the size of the run, with the band weights from Student's t exactly and from three approximations to it. Exact weights are worth 2.709 at every size. Fisher's one-term expansion is worth 2.708 at four thousand draws and 2.523 at a million, and breaks even with counting at 23.7 million draws. The variance-matched normal is worth 1.849 at four thousand and breaks even at 14,698. The standard normal is worth 0.762 at a thousand and breaks even at 669: a bias that does not shrink is multiplied by every draw added. A cut point, at a correlation

Weights from an approximate law

A companion whose law is only approximately known can still be cut into bands, with each band's weight computed from the approximation. The weight's error is a bias that does not shrink, and the run multiplies it: the stratified rate is worth p(1 − p)/(v + Nb²) times the draws. Reading Student's t on nineteen degrees of freedom as normal breaks even at 669 draws; the normal with the t's variance at about fifteen thousand; Fisher's one-term expansion at twenty-four million. What decides the bias is how the weight errors line up with the band rates, not how large they are — Fisher's is wrong by sixty per cent in one band and biases the rate by six hundredths of one.

 The weight each of a hundred outcomes carries in the fitted value at the 97.5th percentile of a lognormal covariate. One design of a hundred lognormal covariate values. The fitted value at a covariate value of 7.10 is a weighted sum of the outcomes, with weights fixed by the design. Under a line and its square it rests on 10.7 effective points, the largest carrying 19.1% of its variance. Under a natural spline with knots at 0.17, 0.57, 1.18, 3.95 it rests on 3.8, the largest carrying 41.1%. What a summary of a scatter is a property of

A value read off a curve

No coefficient of a spline is read; what is read is the fitted curve's value at a covariate value of interest, and that value is a weighted sum of the outcomes with weights the design fixes before any outcome is seen. In the tail of a lognormal covariate a natural spline's value rests on 4.8 of a hundred points, fewer than the squared term's coefficient's 6.9, and the squared term's own value there rests on 10.2 — yet it is three times further from the truth, because its extra points are borrowed from where the curve has a different shape and buy no precision at all. At four hundred points, moving the last knot past the value raises its count threefold and halves its error; doubling the data does neither.

 What it costs to set the t limit at one level without knowing which skewed source the 15 observations came from. For the t limit on 15 observations, charged twenty per standard deviation of shortfall plus its margin, the loss at each nominal level as an excess over the best level for each source. The best levels are 97% for the exponential, 96.25% for the gamma, shape 2, 97.75% for the gamma, shape ½, 98.25% for the lognormal. The minimax level, 97.25%, costs at most 0.8% on any of them: 0.1% on the exponential, 0.6% on the gamma, shape 2, 0.2% on the gamma, shape ½, 0.8% on the lognormal. The conventional 97.5% costs up to 1.0%. Shape, and what it does to a two-sample test

A level set before the source is known

Every best level for an upper limit on skewed data was found knowing the source, and the fifteen observations that would tell the sources apart are the ones being used to set the limit. Chosen instead to protect the worst of four skewed sources, the t limit's level is 97.25% when a failure is charged by the size of its shortfall, and it costs at most 0.8% more than each source's own best — the sources disagree about the best level and agree that missing it a little is cheap. The fitted gamma family, the construction meant to adapt to the source, is the one that needs it: its best levels run from 90% to 96.5% and its minimax level costs 3.4%. Charged a fixed amount per failure, the t limit's minimax level is 99.97% and costs 6.5%; the conventional 97.5% costs up to 119.9%.

 How far the published estimate of the chosen arm sits from the best arm's effect, by how six hundred patients are split between choosing and estimating. Eight arms and a control, six hundred patients: a stated number an arm in the first stage and the rest split equally between the chosen arm and the control in the second. Root mean squared error of the published estimate about the best arm's true effect, over 5,000 trials a split. All eight alike: best at 16 an arm (0.092), 0.183 at the familiar sixty; One modestly ahead: best at 56 an arm (0.151), 0.153 at the familiar sixty; One clearly ahead: best at 48 an arm (0.146), 0.154 at the familiar sixty; Two clearly ahead: best at 40 an arm (0.128), 0.151 at the familiar sixty; Spread evenly: best at 40 an arm (0.131), 0.146 at the familiar sixty. The split that guards every world puts 32 an arm before the choice and costs at most 21.3% over each world's best. Designs that change while they run

Patients spent before the choice

A trial that picks the best of eight arms and then confirms it splits its patients between choosing and estimating, and the familiar shape — sixty an arm before the choice, thirty after — spends ninety per cent of a six-hundred-patient budget on the choice. Measured against the effect a reader is told about, the best arm's, that split is nearly right when one arm is modestly ahead and twice as wrong as it needs to be when the arms are alike. A split of thirty-two an arm before the choice is never more than 21.3% worse than the best split for whichever world holds; sixty an arm can be 98.8% worse. The two targets an estimate could serve — the chosen arm's effect and the best arm's — pull the split in opposite directions, and only one of them is what gets published.

 What a trial's own first patients buy when they choose the follow-up schedule for the rest, powered for an effect of 0.2. Internal pilots measured on two follow-ups, the schedule that needs no pilot, whose correlation curve chooses the schedule for every later patient. Extra cost over the cheapest schedule at stable shares 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7: no pilot, two follow-ups throughout, 11.1%, 9.8%, 8.1%, 5.9%, 2.9%, 0.0%, 0.0%, 3.2%; an internal pilot of 20, 10.5%, 9.8%, 9.3%, 7.9%, 5.9%, 3.7%, 4.2%, 5.2%; an internal pilot of 40, 5.9%, 5.7%, 5.6%, 4.7%, 3.4%, 2.0%, 2.4%, 2.5%; an internal pilot of 80, 3.6%, 3.4%, 3.2%, 2.8%, 1.9%, 1.0%, 1.6%, 1.6%. Worst cases: no pilot 11.1%, 20 patients 10.5%, 40 patients 5.9%, 80 patients 3.6%. What a sample-size calculation was given

A schedule the trial chooses for itself

A separate pilot cannot pay for itself in choosing a trial's follow-up schedule, because its patients are recruited, measured and discarded. A trial can instead measure its own first patients on the schedule that needs no pilot — two follow-ups — fit the correlation curve to them, and give everyone after them the schedule that curve makes cheapest. Those first patients are counted in the analysis and cost no extra visit. Forty of them cut the trial's worst-case excess cost across every curve with adjacent correlation 0.7 from 11.1% to 5.9% in a trial of two hundred an arm, and eighty cut it to 3.6%; twenty buy almost nothing. And because the curve is read from within-arm deviations, the choice cannot see the effect: the pilot's effect estimate is the same whichever schedule it chose.

 The textbook (Wald) interval's coverage among studies whose count of 40 lands in the tails. Coverage of the nominal 95% textbook (Wald) interval at 40 trials, conditional on the count being at most k or at least 40 − k, computed exactly over the counts for every true proportion. With k = 2 its highest value anywhere is 88.96%; With k = 5 its highest value anywhere is 94.90%; With k = 8 its highest value anywhere is 95.57%. A set whose highest value stays below 95% is a set of counts on which the label is wrong whatever the proportion is. Intervals, counted

A count that bets against its interval

A reader who sees only the count can bet against an interval's 95%, and win whatever the proportion is, exactly when some set of counts carries less than 95% at every proportion. Out of forty trials the textbook interval has one — a count of five or fewer, or thirty-five or more, covers at most 94.90% — and the score interval has none: a prior exists under which every one of its forty-one intervals is 96.46% credible, and that prior is the proof that no bet can be found.

 The E-optimality certificate along the axis x₂ = 0. The certificate — a fifth of the squared distance of the point's squared radius from one, plus a fifth of the squared difference of its two squared coordinates — along the axis x₂ = 0, with its two pieces drawn alone at full weight: the radial piece, a third of the squared distance from the unit circle in squared radius, which reaches 0.333 here, and the cross piece, half the squared difference of the squared coordinates, which reaches 0.500. The certificate's largest value on the cut is 0.2000, reached at x = -1.0, 0.0, 1.0; the smallest eigenvalue of the nine-point design is 0.2. A design chosen rather than looked up

A certificate for the worst direction

E-optimality was the one design criterion here whose answer was claimed rather than proved. It has a certificate after all, and on the square it fits on one line — (1⁄5)(1 − x₁² − x₂²)² + (1⁄5)(x₁² − x₂²)², which never exceeds 1⁄5 and touches it at exactly the nine points of the 3² grid. That proves the design with 2⁄5 of its weight at the centre, 1⁄10 at each edge midpoint and 1⁄20 at each corner is E-optimal on the whole square, and it is built from two directions mixed three to two, either of which alone fails.

 Two halves of one dataset with 8 far rows of twenty: the trimmed fit's winner and the best half with the other sign. Twenty rows, 8 of them near x = 9 on a line with a reversed slope. The exact trimmed fit keeps the eleven rows whose own line has the smallest residual sum: 5 far rows and 6 clean ones, slope -0.629, residual sum 0.587. The best eleven rows whose line slopes the other way are all clean, slope 0.398, residual sum 0.657 — 1.12 times the winner's. Regression, and what the summary hides

The other half a trimmed fit rejected

The exact trimmed fit visits every half of the data, so it can report more than its winner: the best half whose line slopes the other way. Whenever the winner is reversed by a far cluster, that other half is the clean line. Printed beside the winner when its residual sum is within 1.70 times — a rule that raises five per cent of clean datasets — it flags 65% of the reversed fits at seven far rows of twenty, 55% at eight and 30% at nine, and no rule built on fit can do better at nine, where the far cluster has become the tighter half by every measure the data hold.

 The bias each model puts into the treatment effect when sixteen earlier controls stepped once. Sixteen earlier trials of two hundred, the first eight a step below the last eight, which sit at the current control's level; the current trial has fifty patients an arm and no treatment effect; 1,500 trials at each step. Exchangeable trials: -0.003, 0.020, 0.041, 0.056, 0.063, 0.065, 0.054. A line in time: -0.003, -0.016, -0.029, -0.041, -0.052, -0.067, -0.075. A change point: -0.003, 0.003, 0.003, -0.001, -0.002, -0.003, -0.003 — at steps of 0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5. The spread, and its own uncertainty

A history that stepped

When sixteen earlier control arms stepped once — the first eight a fifth of a standard deviation below the last — borrowing them as exchangeable flatters a treatment with no effect by 0.063 and stops borrowing, and fitting a line through them penalises it by 0.052. A model with a change point is unbiased at every step size, borrows more the larger the step, 512 patients at a fifth, and puts two thirds of its belief on the right place for the change. Against a steady drift it is the one that fails, so each of the three is safe only against the shape of history it assumes.

 Seven ways to report the height at the setting a response-surface analysis recommends, at noise 2. Bias and root mean squared error of the reported height about the true height at the chosen setting, over 2,000 studies with a central composite design of thirteen runs, or the same design run twice. 13 runs, the fit's own height: bias 0.884, error 1.537. 13 runs, optimism bootstrapped away: bias 0.183, error 1.396. One confirmation run alone: bias -0.005, error 2.005. Corrected fit and one confirmation run: bias 0.056, error 1.121. 26 runs, the fit's own height: bias 0.475, error 0.998. 26 runs, optimism bootstrapped away: bias 0.050, error 0.924. 26 runs split: choose on 13, estimate on 13: bias -0.014, error 1.213. The surface between the corners

The height at the chosen setting

The height a response-surface fit predicts at the setting it recommends reads high, because the setting was chosen where the fit was highest. The obvious repair — choose on some runs, estimate on the rest — is impossible in the usual thirteen-run design, which has nine distinct settings where two separate fits need twelve. A parametric bootstrap of the optimism removes four fifths of it at no cost in runs, 0.884 down to 0.183 at twice the noise. With the design run twice, correcting the full fit beats splitting it: an error of 0.924 against 1.213, at a better setting.

 Where three vocabularies of 5 values put their cuts, under the honest forecasts they divide. The density of the honest probability at full signal, piled toward small values because the event happens 37.4% of the time, and below it three ways of cutting it into 5 groups, each printing its group's event rate. 5 equal-width bins: cuts at 0.200, 0.400, 0.600, 0.800, printed values 0.075, 0.295, 0.497, 0.698, 0.904, resolution lost 0.003454. 5 equal-mass groups: cuts at 0.066, 0.211, 0.422, 0.695, printed values 0.025, 0.132, 0.311, 0.554, 0.850, resolution lost 0.004046. the best 5: cuts at 0.145, 0.339, 0.552, 0.777, printed values 0.054, 0.236, 0.441, 0.662, 0.892, resolution lost 0.003193. The density is cut off at 3.2 near zero, where it grows without bound. The best vocabulary's cuts sit halfway between neighbouring printed values, and its printed values sit at their groups' event rates; no other partition into 5 groups loses less. A forecast that is a probability

Five values chosen by the forecasts

The vocabulary of five calibrated values that loses least on the Brier score puts its cuts at 0.145, 0.339, 0.552 and 0.777, and gives up 0.003193 of resolution against 0.003454 for equal-width bins and 0.004046 for equal-mass groups. Eleven such values beat tenths by 18.5% — but only once someone knows where to cut. Fitted on a record of two thousand forecasts the same eleven add 0.001547, about twice what tenths add with nothing fitted, and they draw level only near twelve thousand. For users acting at chances near one in ten, a vocabulary cut for them lowers their regret by a factor of 7.5 and costs the Brier score nearly nine times as much.

 Three adjustment rules on four thousand structures, as every covariate gets noisier. Four thousand random structures of six covariates, the treatment's effect fixed at 0.5, every covariate observed with the same reliability. Mean absolute bias of adjusting for nothing stays 0.190; adjusting for everything measured falls from 0.377 with exact covariates to 0.279 at reliability 0.8 and 0.203 at 0.5; adjusting for the common causes rises from 0 to 0.056 and 0.118. Adjusting for everything beats adjusting for nothing on 33.8% at 1, 38.3% at 0.9, 42.0% at 0.8, 45.3% at 0.7, 48.3% at 0.6, 50.8% at 0.5. The common causes never do worse than nothing at any common reliability. What conditioning on a variable does

A set of shadows

Measure every covariate of four thousand random structures with reliability 0.8 and adjusting for the common causes still never does worse than adjusting for nothing — the bias left is each confounder's part times one minus its proxy's reliability, scaled alike. Adjusting for everything measured gets better, beating nothing on 42.0% of structures against 33.8% with exact covariates, because noise blunts the bad adjustments too. Spread the reliabilities to 0.75 ± 0.25 and the right set does harm on 7.15% of structures, every one of them with confounders pushing opposite ways.

 Coverage under a drifting scale, with the quantile taken over fewer and more recent scores. The noise scale grows linearly across the sample to 1 + v times its starting value; 4,000 draws at each v. Worst coverage over the sweep: all 199 scores 85.63%, the last 99 90.88%, the last 39 93.42%, the last 19 93.53%. At v = 2.12, where the best of three detectors first fires 80% of the time, they cover 88.11%, 91.86%, 94.02%, 94.45%. Every window covers 95% when nothing drifts. Coverage without a distribution

A quantile over the recent scores

A noise scale that grows across the sample takes a split conformal interval from 95% to 85.63%, and the best detector cannot see most of that loss. Take the quantile over the last thirty-nine calibration scores instead of all 199 and the worst coverage across the whole sweep is 93.42%, for 3.7% more width when nothing has drifted. The same window covers 91.30% where all 199 cover 93.53% under strongly correlated errors — and switching to it only when the detector fires is worse than either, because the detector fires on correlation too.

 Coverage of the cluster-robust slope interval at six, ten and twenty clusters of unequal size, by reference. 5,000 studies of 300 rows at each setting, within-cluster correlation 0.2. 6 clusters, equal sizes: 89.1%, 92.6%, 96.0%; 6 clusters, 5:1, big at the edges: 80.1%, 88.1%, 95.2%; 6 clusters, 5:1, big at the centre: 93.8%, 95.4%, 96.8%; 6 clusters, 15:1, big at the edges: 76.2%, 86.8%, 94.5%; 10 clusters, equal sizes: 91.1%, 93.0%, 95.8%; 10 clusters, 5:1, big at the edges: 86.3%, 90.3%, 94.7%; 10 clusters, 5:1, big at the centre: 94.5%, 95.7%, 96.4%; 10 clusters, 15:1, big at the edges: 84.9%, 89.6%, 94.8%; 20 clusters, equal sizes: 92.9%, 93.8%, 95.1%; 20 clusters, 5:1, big at the edges: 90.9%, 92.7%, 94.8%; 20 clusters, 5:1, big at the centre: 94.5%, 95.0%, 95.6%; 20 clusters, 15:1, big at the edges: 90.6%, 92.4%, 94.7% — for the usual variance on G − 2, the bias-reduced variance on G − 2, and the bias-reduced variance on its effective count. A standard error for a model that is wrong

An effective number of clusters

Ten clusters of thirty rows, a covariate spread evenly across them, and a cluster-robust interval for the slope read against a t on G − 2 = 8 covers 91.1%. The slope needed about five degrees of freedom, not eight, because the two outermost clusters carry 49% of its variance and the middle two 0.6%. A count computed from the sizes and positions before any outcome is seen — 4.45 here, 3.32 when the big clusters sit at the edges — with the bias-reduced variance covers between 94.5% and 96.8% in all twelve layouts measured, where the usual interval on G − 2 covers between 76.2% and 94.5%.

 Five 95% intervals for an AR(1) forecast 6 steps ahead at φ = 0.85, fifty observations. Coverage on 4,000 series, the same series for every interval, each simulated interval built from 499 simulated futures. The plug-in formula: 88.48%. The formula, all three repairs: 92.80%. Simulated: parameters drawn: 92.55%. Simulated: residual bootstrap: 88.25%. Simulated: bootstrap, bias removed: 93.47%. Comparing two forecasters

An interval built from simulated futures

A forecast interval read off simulated futures needs no delta method and no normal multiplier, and was expected to close the residual the three repairs to the formula leave. It does not close it on its own: the plain residual bootstrap covers 88.25% six steps ahead at φ = 0.85, the plug-in formula's 88.48% to within the count, because it simulates around the same biased persistence. With that bias removed inside the bootstrap it covers 93.47%, the best of five intervals, and its shortfall stops growing with the horizon — 93.80% at twelve steps, where the repaired formula covers 91.85%.

 A prior centred on the truth's population and too confident about it: coverage across the population, and how often the check flags it. Truths drawn from a population of proportions centred at 0.2 and worth ten observations; twenty trials per study; the analyst's prior centred at 0.2 with the weight on the axis. Coverage of the 95% credible interval, exact over the population: 94.7% at 2, 96.0% at 5, 95.0% at 10, 88.1% at 20, 75.9% at 35, 61.3% at 60, 48.2% at 100, 34.0% at 200, 23.9% at 400. Share of studies whose count the prior-predictive check puts in its 5% tail: 0.1%, 1.8%, 3.2%, 14.2%, 17.3%, 17.3%, 21.9%, 21.9%, 21.9%. The prior, doing visible work

A prior wrong about its own spread

A prior can be centred exactly where the truths lie and still be wrong about how far they spread. Ten times too confident — worth a hundred observations about proportions whose population is worth ten — its 95% credible intervals cover 48.2% of the population's studies at twenty trials, and nothing in the interval shows it: it is 0.456 of a flat prior's width, against 0.436 for a prior whose hundred observations' confidence is earned. A prior with tails too thin misses the one study in ten that comes from somewhere unusual a third of the time, and a flat component of two per cent brings that study's coverage from 65.6% to 84.0%.

 Coverage of the population mean from two units, by the true spread between units. 4,000 two-unit studies at each true spread, ten observations a unit, within-unit standard deviation 1.2. exact, on one degree of freedom: 94.97%, 95.50%, 94.95%, 94.83%, 95.15%, 95.20%, 94.67%. spread supplied as 1: 100.00%, 100.00%, 99.92%, 95.53%, 81.85%, 70.03%, 51.55%. half-normal, scale 1: 100.00%, 100.00%, 99.78%, 94.63%, 86.25%, 79.70%, 68.38%. half-Cauchy, scale 1: 100.00%, 100.00%, 100.00%, 98.98%, 95.50%, 92.38%, 88.00%. inverse-gamma(0.001, 0.001) on the variance: 100.00%, 100.00%, 100.00%, 99.15%, 97.05%, 95.28%, 92.65% — at spreads 0, 0.25, 0.5, 1, 1.5, 2, 3. The exact interval holds 95% at every spread; every integrated one over-covers when the spread is small and under-covers when it is large, and where it crosses 95% is set by the prior's tail. Hierarchy past one number

A prior on the spread between two sites

Integrate the between-site spread of a two-site study under a half-normal prior of scale 1 and the interval for the overall mean is 3.01 wide and covers 94.63% when the true spread is 1 — the supplied interval again, under another name. At a true spread of 3 it covers 68.38%. Every prior tried covers 95% averaged over its own draws and none covers it at every spread; each holds up to about its own scale, because two sites move the posterior median of the spread by less than a third of what the truth moves.

 The shape one real modifier leaves on a table of eight characteristics, and one trial's table. Expected interaction z when characteristic 1 modifies the effect at h = 1: 2.80, 2.00, 2.00, 2.00, 0.00, 0.00, 0.00, 0.00 — the real row 2.80, its three nested neighbours 0.713 of that, the four unrelated characteristics nothing. One seeded trial of 400 reads 1.91, 1.90, 2.06, 1.95, 0.14, 0.33, 0.39, 2.08. An interval read beside something else

The pattern a cause leaves on its neighbours

A characteristic that modifies a treatment's effect raises every characteristic that overlaps it, in proportion to the overlap, and fitting that pattern was supposed to say which one is the cause. Fitted with the noise's own correlation it names the largest row on every trial, because the noise draws the same pattern — so the 54.4% at which the largest row is right among nested characteristics is the most any reading of the table can manage, and the honest answer is a set of about five.

 Four treatments of a shock sixteen pairs share: false alarms at the independence point, and power at each one's own point. Sixteen pairs, two hundred dates, nine tenths of each series' variation common. every pair loads the shock equally: nothing removed, false alarms 31.6% and power 22.3%; the cross-sectional average removed, false alarms 4.8% and power 99.8%; an estimated factor removed, false alarms 9.2% and power 76.3%; the observed shocks removed, false alarms 4.5% and power 6.5%. loadings run from a half to one and a half: nothing removed, false alarms 30.8% and power 25.8%; the cross-sectional average removed, false alarms 13.3% and power 82.5%; an estimated factor removed, false alarms 8.9% and power 73.8%; the observed shocks removed, false alarms 4.5% and power 7.5%. Series that move together

A factor estimated from the panel

When sixteen pairs feel a common shock with loadings from a half to one and a half, removing the cross-sectional average leaves the pooled test rejecting an unrelated panel 13.3% of the time. Estimating the factor and each pair's loading instead brings that to 8.9% — and no further, because each loading comes from the same two hundred dates however many pairs there are — and it costs power against the average at the right critical value, 73.8% against 82.5%. Removing the shocks as observed restores 4.5% exactly and finds a shared relation 7.5% of the time: a series tied to its partner carries the partner's factor in its level, not its own shock.

 How much of a return level's uncertainty is the extremal index's, by how far past the record the level is. Delta-method variance of the level from a declustered peaks-over-threshold fit, averaged over 300 records of 4,000 observations. The cluster rate's share at 1.5, 2, 5, 20, 100, 1000 periods: independent, 0.374, 0.253, 0.084, 0.027, 0.012, 0.005; runs of exceedances, θ = 0.6, 0.558, 0.396, 0.141, 0.039, 0.015, 0.006; spaced clusters, θ = 0.4, 0.845, 0.562, 0.222, 0.057, 0.019, 0.007; runs of exceedances, θ = 0.25, 0.787, 0.788, 0.327, 0.083, 0.024, 0.008. The tail past the last observation

Where the extremal index matters to a return level

A return level from a declustered fit depends on how often clusters arrive, which is the extremal index estimated from the same record. For a level reached once in two periods of a twenty-period record the index's error is a quarter to four fifths of the level's variance, and an interval that leaves it out covers 59.0% when clusters are long; put in, 90.0%. For a level reached once in a hundred periods its share is under two and a half per cent and adding it changes no interval at all — what fails there is the shape, fitted from however many clusters the dependence left: 82.3% coverage from forty-six clusters, 65.7% from nineteen.

 Six intervals, one of them honest by right. How often each rule's reported 95% interval covers the true coefficient under a five-period moving average, over 1000 draws of 120 rows, for the most persistent predictor (its own lag-one correlation 0.9) and the white one. The law's own covariance covers 94.9% and 94.1%. Least squares covers 65.0% on the persistent predictor and 94.4% on the white one; a fitted ρ̂ 93.0% and 96.3%; the two-step band 86.5% and 97.3%. The joint fit covers 73.2% and 71.4% — short on both, and short on the white predictor where every other rule is honest — and counting its eight band coefficients gives 76.1% and 73.5%. When the observations repeat each other

The interval the joint fit reports

Maximising a regression's coefficients and an eight-lag covariance band together gives a better estimate than plugging the band in. It also prints an interval that covers 71% to 85% of the time where 95% was promised — on the white predictor too, where every other rule is honest — because it reports almost the precision of the law's own covariance and delivers less than half of that advantage. Counting the eight lags it estimated buys one to two points.

All themes