Concept

Effective sample size — where it appears

The number of independent observations a dependent set is worth, which is what a standard error dividing by √n is implicitly claiming n to be. An optimism theorem counts rows and a bootstrap counts draws, and both are wrong by the same factor when the rows repeat each other.

Named by 48 essays across 26 fields — each of them below, with the objects they name alongside it.

The correction is not a property of the sample. tr(HΩ)/q for each of fifteen candidates, at ρ = 0.7. Two candidates that fit the same number of coefficients need corrections that differ by as much as 1.49, because one of them is fitting the persistent predictors and the other is not — so no single number can be right for both, and the scalar n/n_eff = 5.537 is above every one of them. The four predictors carry persistences 0.9, 0.6, 0.3, 0; at one persistence for every column the whole spread collapses and a scalar looks exactly as good as the trace.

A penalty is a trace

Akaike's 2q is not a count of coefficients. It is the answer a trace collapses to when the rows are independent — and once they are not, the trace is still the right object and is no longer the count.

effective · Order selection

A proposal that moves more, refused more often. The two halves of the trade, both exact, on the 410 admissible assignments of twelve units. The integrated autocorrelation time of an imbalance the rule was never handed falls from 7.30 at one swap to 3.97 at three, and the acceptance rate falls with it, from 58.8% to 40.8%. A rejected proposal costs one evaluation and leaves the chain where it was, so acceptance is not the price of anything and the ranking by acceptance is the reverse of the ranking by cost. Past three the family folds: exchanging k of six from each arm is the complement of exchanging six − k, so k = 5 has the same 36 proposals as k = 1 and k = 6 has 1.

A proposal that moves more than two units

The walk's autocorrelation is a fact about its step size and not about its acceptance rate. Exchanging three units from each arm mixes nearly twice as fast as exchanging one, and is refused a third more often.

blocks · Randomisation

The test, checked where the answer is known. A fourteen-unit trial at eight tolerances. At each one the admissible set is enumerated — 1534, 886, 304, 158, 126, 116, 102, 84 assignments — and its components counted, which is only possible because 3432 equal splits of fourteen units can be walked. The dots are the test, which walks none of them: two chains, one started at an assignment and one at its complement, compared on a statistic the rule was not handed. Filled marks are tolerances the enumeration says leave the set in more than one piece. The test fires on every one of them and on none of the others, 0 misses and 0 false alarms.

A test rather than a survey

A thin admissible set falls into an arrangement and its mirror image, and the walk that samples it is uniform on half the reference distribution for ever. That was found by enumerating fourteen units, and enumeration stops at twenty-four.

reach · Randomisation

The construction survives a difference of two weighted means. Coverage of δ̂ ± t√(S_D²/H) on b − 1 degrees of freedom, over 900 runs at a requirement of 0.3, where δ̂ is the block differences weighted by h_b = (1/m_A + 1/m_B)⁻¹ and H is their total. The theorem the one-mean field rests on goes through with h_b in place of the block size, and the reason is that the weights a weighted least squares decomposition needs are the inverse variances — which is exactly what h_b is. The stopping rule reads only within-arm within-block contrasts, so it is a function of nothing the interval reports, whatever it does with the block sizes. Each bar is within 2.9% of the level it claims.

A width promised for a difference

The exact fixed-width interval was built for one mean. Two arms make the target 42.7 units of effective size and each unit costs four observations, so the same promise about a difference costs 169.4 rather than 42.7 — and the theorem survives untouched with the harmonic size in place of the block size.

contrast · Width

Twenty series with a lag-one correlation of 0.8. Every series has a true mean of zero and 60 observations. The marks on the right are the twenty sample means. The variance of that mean is 8.3 times what 60 independent observations would give, so the series is worth about 7 of them.

The observations that repeat each other

Almost every standard error divides by √n, which claims the observations carry independent information. At a lag-one correlation of 0.8 a fifty-point series is worth about six independent observations, and its 95% interval covers 47%.

timeseries · Dependence

Two diagnostics, one answer, two different moments. The two-chain statistic on a covariate probe and on the trial's own difference in arm means, at seven tolerances of a fourteen-unit rule, against the enumerated truth. Both are quiet wherever the set is one set and both fire wherever it is not, at every tolerance — which is what says the outcome probe is the same test rather than a resemblance of it. The difference between them is not accuracy and it is not power. It is when: the covariate probe can be run before a single outcome exists, when a practitioner can still loosen the rule or change the sampler, and it can be run again on a different function if it comes back quiet. The outcome probe runs after the trial, on the one column the trial produced, and what it can do with a positive verdict is repair the p-value rather than the design.

The statistic the p-value is about

The test for whether a balanced-assignment walk reaches its whole set is run on a covariate function chosen before the trial. Run on the difference in arm means it is the same test, and it is about the number the trial publishes.

after · Randomisation

Exact in the corner, where nothing was. Coverage of a nominal 95% interval on five designs, at a required half-width of 0.3. The first four are the two-arm field's own and the fifth is its corner — two variances, block sizes that swing by eight, and an allocation that alternates between five to one and one to five — where neither of that field's two conditions holds. The effective-size weights over-cover there at 98.40%; the weights h_b(λ) = (1/m_A + λ/m_B)⁻¹ cover at 94.84%, and at 94.84% when λ is estimated from the within-arm contrasts rather than known. Nothing here is supposed to move.

Weights that need only a ratio

A fixed-width interval about a difference is exact under either of two conditions and under neither in the corner. It is exact there too, and the only thing it needs is how much larger one arm's variance is than the other's.

corner · Nuisance

One comparison, and the two error bars it can be given. 60 rolling origins, a window of 60 observations, forecasts 4 steps ahead, at the persistence φ = 0.8256 where the two benchmarks have exactly equal population mean squared error. Each mark is one origin's difference in squared error; the horizontal line is their mean, 0.6522. The two vertical bars at the right are ±1.96 standard errors round that mean computed two ways — 0.5337 treating the differences as independent, 0.6880 allowing for the overlap between neighbouring forecasts. The null is true here by construction, so an interval that excludes zero is a mistake, and the narrow one does it far more often than the wide one.

Which forecast is better

Two forecasters, one series, and a difference in mean squared error. Whether that difference is real is a hypothesis test, its terms are not independent, and the standard error it needs is not the one a t-test computes.

evaluation · Forecast

The probe a trial has is the probe a trial got. What the two-chain test says when it is run on the trial's own difference in arm means, over 24 outcomes on one fourteen-unit set. The set is in 2 mirror components — that is enumerated, not inferred — so every quiet reading is a miss. 29% of them are quiet. The reason is in the enumerated set rather than in the run: how far the two components are apart on a given probe ranges from 0.001 to 4.938 of a within-component spread across these outcomes, a factor of several thousand. Both covariate probes — chosen before any outcome existed, and replaceable if they had been quiet — report the split. An outcome cannot be chosen and cannot be replaced.

A probe nobody chose

On a set that is definitively in two pieces, seven of twenty-four outcomes report nothing at all. Every covariate probe reports it. What separates them is not accuracy — it is that one of them can be chosen and the other is what happened.

after · Randomisation

Eight candidates, one of them exactly as good as the benchmark. The candidate set: moving averages of the last 1, 2, 3, 5, 8, 13, 21 and 34 observations, each drawn as its expected squared error divided by the benchmark's — the mean of all 60. The persistence is not chosen, it is solved for: at φ = 0.4895 the best candidate in the set, the average of 2, has exactly the benchmark's expected squared error, and every other candidate is worse by between 0.5% and 5.1%. So the null that no candidate beats the benchmark is true, with one candidate on its boundary. Everything a set comparison claims about its own error rate has to be measured here, because anywhere further inside the null every procedure flatters itself.

Eight forecasters and one benchmark

A set of forecasters is a multiplicity problem on top of a dependence problem, and the two do not separate. Eight windows of one series carry the multiplicity of two and a half independent comparisons; eight separate problems carry eight.

ranking · Multiplicity

The one candidate an effective sample size is right about. n/n_eff with the finite-sample inflation Σ(1 − |k|/n)ρ^|k| is not an approximation to tr(HΩ) for a fit with only an intercept — it is that trace, to machine precision, because the hat matrix of a constant column is 1/n everywhere and its trace against Ω is the mean of Ω. The quoted limit form n(1 − ρ)/(1 + ρ) is not even right about that one. And the average correction the table's fifteen candidates actually need is 3.318 per parameter, well below the scalar, so applying it to all of them over-charges every one.

One number for a table of candidates

An effective sample size is a real quantity, it is exactly right about one thing, and that thing is a mean. Substituted into Akaike's criterion it changes nothing at all, because the penalty it is meant to fix has no sample size in it.

effective · Dependence

Three readings, one verdict. Every tolerance of a fourteen-unit trial, with three comparisons on each. The first is between a chain started at an assignment and a chain started at its complement, which is what a mirror split separates. The second is between two chains started at the same assignment on different streams, which nothing about the set can separate — so a large reading there says the run is too short and not that the set is in pieces. The third is the same comparison on the statistic's absolute value, which is symmetric under the complement and therefore blind to the split by construction. The verdict is the pattern rather than any one line: the split is called only where the first fires and the other two do not, which happens at exactly the tolerances the enumeration calls disconnected — 0.8, 0.75, 0.7.

The statistic that changes sign

A test for an unreachable half needs a quantity that tells one half from the other. Every symmetric reading of a mirror pair is identical, and a magnitude is the natural thing to reach for.

reach · Randomisation

What studentising costs

Averaged over eight cells the studentised interval is 2.09 times as wide as the percentile one and covers 0.46 points better. At the block lengths the rules choose, the scale it divides by rests on two or three numbers.

student · Bootstrap

The crossing is in the dependence, not in the split. Regret of each rule as the design and the errors are made persistent at the same coefficient, scored on fresh rows because the closed form assumes exactly what is being taken away. An optimism theorem counts rows; when the rows repeat each other there are fewer of them than there are rows, the penalty is too small for the fit it is correcting, and the criterion starts buying coefficients it should not — its average winner grows from 3.31 coefficients to 3.90. The hold-out never used the theorem and overtakes at ρ ≈ 0.81. Schwarz's criterion, worst of the three on independent rows, is best on repeating ones — its heavier penalty is right for the wrong reason.

Where the two searches cross

The obvious dial between a criterion and a hold-out is how much of the sample to hold out, and moving it never changes the answer. The dial that does is one nobody chooses — how much each row repeats the one before it — and the two rules change places at about 0.81.

proxy · Forecast

Which weights are the inverse variances

There is an exact estimator when the two arms share a variance and another when every block has the same two counts, and between them they cover every trial anybody designs on purpose. In the corner where neither holds, both cover 98.45% instead of 95%, and the only estimator at its level is the one with no theorem behind it.

contrast · Nuisance

Unrepresentative in every respect but the one that matters. Three properties of the complete cases as the chance of being observed leans harder on the regressor, in closed form, at 35.0% of outcomes missing throughout. The mean of the regressor among the rows kept climbs from 0.0000 to 0.5528 against a population mean of zero, and the mean of the outcome from 0.0000 to 0.3980 above its own. The bias in the fitted slope is exactly zero at every one of the ten settings, because selection acting on the regressor alone leaves the conditional law of the outcome given the regressor untouched and least squares conditions on exactly that. The sample is wrong about almost everything and right about the one quantity being estimated.

Dropping the incomplete rows

Push the missingness until the rows that survive have a covariate mean of 0.543905 against a population zero and a variance of 0.5041 against one, and the fitted slope is still exactly right. Where the rule reads the outcome instead, the same sweep takes coverage to 2.42% at eight hundred rows.

missing · Missingness $Three answers to how much sample is left. What a set of inverse-probability weights leaves of the treated arm, by three routes, at six settings of the assignment rule. The integral 1/(π∫φ/e) reads the whole covariate space and falls from 0.9392 to 2.655e-3. Kish's effective size counted in samples of 600 falls only to 0.2861, because almost all of the integral's fall is in a region a sample of six hundred never draws from. And the fraction the variance of the weighted mean actually delivers is lower again — 0.1155 — because the variance is the average of one over the effective size and the effective size averaged is not the same number. At the widest overlap all three agree to 0.05%.$

How many observations a weight leaves

Kish's effective sample size is exact — for an outcome whose mean does not move with the covariates the weights are built from, the studentised variance reads 1.0680 where the formula says one. For the population's own outcome the same reading is 6.769, rising to 52.497.

weights · Weighting

Whichever dial made the set thin, the crossing is at the same thinness. Each curve is one dictionary, swept over eight tolerances at two hundred units; a point above the line is a set thin enough that walking beats hunting. The curves lie nearly on top of one another, which is the answer to whether the crossing is a fact about the tolerance or about the thinness it produces: the crossings sit between one admissible assignment in 176 and one in 268 for dictionaries of 3 to 6 functions. The mechanism is that a hunt costs exactly 1/p and a walk costs almost the same everywhere — between 82 and 394 evaluations per usable draw across the whole table — so the crossing is wherever 1/p reaches a number that does not move.

The set a dictionary leaves

A rule constrained on six functions at a loose tolerance leaves a set as thin as one constrained on three at a tight one. Both sampling methods cross over at the same thinness, and the tolerance where that happens moves by a factor of three.

dict · Assignment

What a chosen probe finds

On a chain of eight hundred draws the probe the earlier fields use misses 44% of the sets that are split. Its own residual off the rule's span misses 12%, for one least-squares fit.

aimed · Randomisation

The crossing barely moves. Both methods' costs in one unit — assignments evaluated per usable draw — as the tolerance tightens. A hunt costs 1/p and rises without limit: from 2.22 at a tolerance of 1.2 to 357.14 at 0.18. A walk costs its autocorrelation time and barely moves. The two cross at a tolerance of 0.190 at one swap and 0.195 at eight — the whole family of proposal sizes crosses inside a band of about two hundredths, because where the crossing is, the large proposal has already lost its advantage. A multi-swap proposal is worth a factor of 5.65 in the regime where the walk should not be used at all.

Where the gain is, and where the decision is

A bigger proposal is worth a factor of six at a loose tolerance and nothing at a tight one. The tolerances where it helps are the ones where a hunt costs two evaluations a draw, and the crossing barely moves.

blocks · Assignment

Where a walk is cheaper than a hunt. Both costs in the same unit. A rejection sampler evaluates 1/p assignments per independent draw and does not care how large the trial is; a walk evaluates one per step and yields an effective draw every τ steps, and τ is a property of the constraint and the statistic together. They cross at a tolerance of 0.194 standard deviations, where about one assignment in 396 is admissible — far tighter than any trial is designed at. And the walk does not remove the acceptance cost; it pays it once, hunting for somewhere to start.

Draws that repeat each other

A hunt costs 1/p evaluations per independent draw. A walk costs one per step and yields an effective draw every τ steps. Both are counted in the same unit, and the walk is dearer at every tolerance a trial is designed at.

joint · Reference

A bias against a variance, with the answer in between. How wrong one sample's reference distribution is, split into the two things it is wrong by. Sharing the multiplier over more rows keeps more of the dependence and closes the bias from 1.688 to 0.835; every row it is shared over also removes an independent sign from the 101 the sample started with, and the spread of the resulting quantile rises from 1.307 to 2.172. The distance a practitioner with one sample is actually exposed to is the two together, and it is smallest at ℓ = 5.

How long a block a multiplier shares

Sharing a sign over more rows keeps more of the dependence and leaves fewer independent signs to build a distribution from. The bias falls from 1.6885 to 0.8479 and the spread rises from 1.3073 to 2.1716, and the rejection rate walks straight through its nominal level on the way from 11.3% to 1.3%.

proxy · Reference

What the diagnostic says at two hundred units. The same test run 8 times on independent streams, at five tolerances of a two-hundred-unit trial, 40,000 steps each. At the loosest tolerance every run says the same thing — the walk reaches the whole set — and it keeps saying it as the set is thinned. Past a point the runs stop agreeing with each other: at the tightest tolerance here 6 of 8 report that the chains have not mixed and 2 report a split, which is a diagnostic disagreeing with itself rather than a property of the set. That disagreement is the honest answer at this size, and it is one nothing in this collection could give before: an enumeration stops at about twenty-four units.

The diagnostic at two hundred

Pointed at a trial size no enumeration reaches, the test gives three answers rather than one — and past a certain thinness it stops agreeing with itself, which is the honest reading and the one nothing could give before.

reach · Randomisation

Where the taper's case begins, and it is not where the algebra says. The block length at which a trapezoidal block's implied variance stops being more biased than a rectangular one's, against the length of the sample. Computed exactly — from the law's own autocovariances, with no sampling in it — the answer is 19.2 and does not depend on the sample at all. What a sample of 120 rows reports is 13.3, and the reported crossing walks out towards the exact one as the sample grows: 13.3, 15.0, 16.4, 18.0. The mechanism is that the autocovariances the window is applied to are themselves attenuated, worst at the longest lags, and the window that discards those lags loses less of them.

The error no window repairs

Every block window's best estimate of a long-run variance is wrong by about forty per cent at a hundred and twenty rows, and the largest part of that is not a bias at all. Choosing the window moves a twentieth of it.

crossing · Bootstrap

What 20 clusters of 20 correlated observations do to a 95% interval. Each study has 400 observations arranged as 20 clusters of 20. The lower points are the counted coverage of the usual interval, which treats them as 400 independent observations; the curve through them is 2Φ(1.96/√deff) − 1 with deff = 1 + 19ρ, computed before any data was drawn. At ρ = 0.81 the interval covers 36% rather than 95%. The upper points treat the cluster as the unit and need no variance components at all.

Two levels at once

A third level of grouping adds no new arithmetic and produces one number — the design effect — that decides how many independent observations a clustered study is worth. It is the same quantity the time-series field computes for autocorrelated data, arrived at from a completely different picture.

multilevel · Levels

When every null is true

A reality check assumes that every candidate in the set is exactly as good as the benchmark, which is a configuration nobody's data is ever in. Test a combination against its own parts and that configuration is not assumed — it is what the arithmetic makes true.

search · Multiplicity

What 240 observations are worth, by which question is asked. 8 rows and 10 columns with 3 observations in each cell — 240 in all, each belonging to one row and one column, neither nested in the other. the overall mean: variance 0.1703 against a naive 0.0060, a design effect of 28.4 and 8.5 effective observations; a difference between two rows: variance 2.0682 against a naive 0.0960, a design effect of 21.5 and 11.1 effective observations; a difference between two columns: variance 1.0845 against a naive 0.1200, a design effect of 9.0 and 26.6 effective observations.

Two groupings that cross

Pupils belong to a school and to a neighbourhood, and neither is nested in the other. There is then no design effect: the overall mean is worth 8.5 independent observations out of 240, a row difference 11.1 and a column difference 26.6, and which grouping matters depends on the question rather than on the study.

multilevel · Levels

The damage and the warning, against the same dial. Two readings at each persistence. In the darker colour, how often a regression between two independent series of 200 steps is called significant at 5%: 4.9% at φ = 0, 34.2% at 0.8, 52.4% at 0.9, 83.4% at a unit root. In the lighter, how often the standard unit-root test refuses a unit root on one of those series — the chance the analyst is told the series is stationary and may be regressed: 87.2% at φ = 0.9 and 31.9% at 0.95. At φ = 0.9 both are high at once, which is a correct diagnostic licensing a regression that is wrong half the time.

The cliff that is a slope

A regression between two independent series is called significant 4.9% of the time at no persistence, 52.4% at a lag-one correlation of 0.9, and 83.4% at a unit root. The rule the field offers asks whether the last of those holds, and at 0.9 the unit-root test correctly refuses one 87.2% of the time.

timeseries · Spurious

The points a slope rests on

Before any outcome is measured, a regression's design already says how many of its points the slope will rest on: n divided by the covariate's kurtosis, exactly. A hundred points of a normal covariate carry a slope on about 35 of them; a hundred of a lognormal one, on 7.6. The largest leverage follows the same tail — above one half in 15.7% of lognormal designs of a hundred and in none of the normal ones — and on a correct straight line, deleting the single highest-leverage point moves the slope by more than its standard error in 16.3% of lognormal datasets of a hundred and 0.1% of normal ones.

spread · Summary

What the two-regime residue was made of, persistent first. Under a break in the persistence at row 60 of 120 (0.95 then 0.65), the selection regret of six whitenings over 200 draws: restart, unscaled, residual persistences 0.04043; restart, unscaled, persistences given 0.03488; restart, unscaled, fitted with the line 0.03466; the law's recursion, residual persistences 0.03856; the law's recursion, fitted with the line 0.01640; the law's own covariance 0.01579. The whitening that restarts at the break and does not rescale its two regimes gives up 0.01910 against the oracle even when it is given the true persistences; the law's recursion, which divides each row by its regime's innovation scale and continues across the break, is the oracle when given them and within 0.00061 of it when they are fitted with the regression.

The residue was the weights

A regression whose errors change persistence at a known row was whitened as two autoregressions and still gave up 0.0246 against the law's own covariance. That residue was put down to estimating the two persistences from residuals, and the repair named was to fit them with the line. Fitting them with the line moves the persistent regime's estimate from 0.81 to 0.89 and recovers a fifth of the residue — exactly what being told the true persistences recovers. The rest was the transform: it never divided the two regimes by their own innovation scales, which differ sixfold. The law's own recursion, told the persistences, is the oracle to machine precision; fitted with the line, it estimates the persistent one at 0.945 and comes within 0.0006 of the oracle.

general · Dependence

A penalty that counts independent rows

Akaike's penalty charges a fit two error variances a coefficient because each coefficient is fitted from n independent rows. When the design and the errors both persist, a row is worth less than a row, and the honest charge is twice the coefficient's variance inflation. On a hundred and twenty rows at persistence 0.7 that charge is 34.7 against an optimism counted at 34.5, where Akaike's charges 10. Charged it, the criterion's regret at 0.95 falls from 0.128 to 0.032 with the persistences given and to 0.065 when they are read off the sample — and at 0.5, where the charge is right and Akaike's is half of it, Akaike's still selects better.

proxy · Forecast

The exceedances arrive together. 300 steps of a max-autoregression with dependence 0.75, drawn on a logarithmic scale because its marginal has no variance. The rule marks the 0.9 quantile: 30 of the 300 readings are above it and they fall into 5 clusters, the largest holding 11. The mean cluster holds 6.000, and its reciprocal — 0.167 — is the runs estimator of the extremal index, whose true value for this process is exactly 1 − 0.75 = 0.25. Every threshold method in the collection assumes exceedances are independent pieces of information; here 30 of them are 5.

The clustering the tail has

Every threshold method counts exceedances as though they were independent pieces of information, and in a dependent series they arrive in clusters. Ignoring that overstates a return level by the reciprocal of the extremal index — ×3.527 counted where the mean cluster holds four — and leaves a reported standard error 2.151 times too small.

extreme · Extremes

The rows are held fixed; only the clusters move. Counted coverage of four 95% intervals for a slope, at five cluster counts with the row count held at 300 throughout and a within-cluster correlation of 0.1, over 6000 draws apiece, with the sizes equal. The interval that counts rows covers 53.42% at 5 clusters — a second closed form says 2Φ(z/√D) − 1 = 54.44% for a design effect of 6.900, and reads nothing about clusters at all. The cluster-robust interval read against a normal covers 74.43% there and 94.20% at 100 clusters; read against a t on G − 1 it covers 85.08% and 94.47%. The number of independent things is the cluster count, and every quantity here is blind to how many rows were typed.

The count that is not the rows

Three hundred rows in five clusters of sixty carry 6.9000 times the variance an independent-rows calculation reports, and the interval that counts rows covers 53.42%. The same five unequal sizes laid out two ways give design effects of 9.3158 and 5.4652.

sandwich · Misspecification

Estimating P(Z > 5) = 2.8665×10⁻⁷ with plain draws and with four proposals. One seed each. Plain simulation draws nothing past 5 in 100,000 and estimates zero throughout. At 100,000 draws the proposal N(5, 1) reads 1.009 of the truth, N(9, 1) 1.041, N(4.5, 0.25²) 1.039 and N(5, 0.3²) 1.008. Values above 2.2 are drawn at the top edge.

The draws aimed at the tail

The chance a standard normal exceeds 5 is 2.8665×10⁻⁷, and a plain simulation needs 349 million draws to estimate it to within ten per cent. Draws aimed at the tail and weighted back need 565. Aimed slightly too narrowly, the same method has an infinite variance, an interval that covers 86.0% and gets worse with more draws, and an effective sample size that reads healthier than a proposal that works.

method · Seeds

Weights that balance a sample by construction. What three sets of weights leave of the standardised difference between the arms on each covariate, as a root mean square over 1200 samples of 600 units. The true propensity leaves 0.1317 and 0.1186 — a sampling error, since it is right about the population and knows nothing of the draw. A likelihood fit leaves 0.0770 and 0.0657, having absorbed part of the draw's imbalance as a side effect of fitting the treatment. Weights fitted so that each arm's weighted means are the sample's leave 1.4e-14 and 1.2e-14, which is the arithmetic's floor rather than a small number: the largest gap between a weighted arm mean and the sample mean in any draw is 9.8e-14.

A weight fitted to balance

Weights fitted so that each arm's weighted covariate means equal the sample's leave a difference of 1.4×10⁻¹⁴ between the arms and give the estimate a third of the variance of weights fitted by likelihood — 0.011883, within a relative 5.8% of the bound no estimator can beat. In the world where the assignment carries a square nobody named, the same exact balance leaves the square further apart than no weighting at all, and where the outcome carries it too the estimate is wrong by 0.6973 with an interval that covers 1.5%.

weights · Weighting

What a variance estimated from K units is worth. The between-unit mean square is a scaled chi-square on K − 1 degrees of freedom, so the estimator's whole distribution is decided by the number of units. At two units its interquartile range spans a factor of 13.03 and its ten-to-ninety range a factor of 171.3, and it comes out exactly zero on 26.7% of studies. The closed form and 3,000 simulated studies agree to 0.051 at every quantile.

A level with two units

A variance estimated from two units is a scaled chi-square on one degree of freedom. Its interquartile range spans a factor of thirteen, its ten-to-ninety range a factor of a hundred and seventy-one, and it comes out exactly zero on 26.7% of studies — so the design effect it decides runs from 1.00 to 7.01 against a truth of 4.69.

multilevel · Levels

A point ordinary on every axis

In a multiple regression each coefficient rests on its own number of points, n divided by the kurtosis of that covariate's residual after the others, exactly. Correlation between normal covariates inflates a coefficient's variance fivefold and leaves it resting on the same third of the points. What correlation does instead is hide the point that decides the coefficient: at a correlation of 0.9 it stands out on no single covariate's plot in 77.3% of designs of a hundred, and the largest leverage a normal design reaches depends on the number of points and covariates and on nothing else.

spread · Summary

Three charges and two counts. The optimism of an in-sample fit of the widest candidate on 120 rows, in units of its own error variance, over 1500 draws, by persistence. Counted on the 40 rows straight after the sample: 11.1, 21.5, 35.6, 51.5, 65.5, 87.8, 119.1. Counted on rows 600 further on: 9.3, 18.3, 33.0, 50.6, 67.0, 96.1, 159.0. Charged by the long-series inflation: 10.0, 19.3, 34.7, 54.4, 74.3, 114.2, 234.1. Charged by the same inflation summed over the sample's rows: 10.0, 19.2, 34.2, 53.1, 71.8, 108.2, 208.8. Charged by twice the trace of the hat matrix against the errors' correlation on the sample's own design: 10.0, 18.7, 32.0, 47.5, 61.5, 85.2, 132.2.

A charge the sample can afford

The inflation charge for dependent rows runs to twice the counted optimism at persistence 0.95, and the reason offered was that a series of n rows cannot be optimistic about movements slower than n rows. Summing the inflation over n rows does almost nothing: 234 becomes 209, against 119 counted. Evaluating the same variance on the sample's own design gives 132, and moving the fresh rows away from the sample raises the count to 159. So the charge can be made nearly right, and when it is, the regret doubles from 0.032 to 0.061. A selection wants a much larger charge than the optimism, and past about fifty per predictor it is no longer selecting.

proxy · Forecast

Each interval covers one question and not the other. Coverage of each interval for the overall mean, scored against both estimands, over 20,000 two-site studies of 10 observations apiece. The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and over-covers the two sites in hand at 98.25%. Both are correct; they are answers to different questions printed in the same place.

What a two-unit study should report

The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and is 11.6 times wider.

multilevel · Levels

A shock every pair shares

Sixteen pairs of series whose shocks are half common are worth 5.37 independent pairs to the panel's average statistic, and sixteen that share nine tenths are worth 1.44. Read against the 5% point computed for independent pairs, the pooled test then calls a panel with no relation tied 15.5% and 31.6% of the time; read against its own point, it finds a relation shared by every pair 22.3% of the time at nine tenths common, against one pair's 20.0%. Removing each date's cross-sectional average gives back the level and all sixteen pairs when every pair feels the shock equally, and leaves 13.3% false alarms when the loadings run from a half to one and a half.

cointegration · Spurious

A term built from the others

A regression's squared terms and interactions are columns built from its covariates, and each coefficient on one rests on the number of points divided by the kurtosis of what the built column adds — fifteen for the square of a normal covariate, nine for the product of two uncorrelated ones, three for a treatment's interaction with a normal covariate, which is the same as the covariate's own coefficient. A design of a hundred rests on more than the population says, 12.2 points behind a square against 6.7, because a hundred draws rarely hold the tail that sets the kurtosis. Centring changes nothing about the square's coefficient, to the last digit. And correlation between two covariates hides the point that decides each one's own coefficient while it reveals the point that decides their interaction.

spread · Summary

The weight each of a hundred outcomes carries in the fitted value at the 97.5th percentile of a lognormal covariate. One design of a hundred lognormal covariate values. The fitted value at a covariate value of 7.10 is a weighted sum of the outcomes, with weights fixed by the design. Under a line and its square it rests on 10.7 effective points, the largest carrying 19.1% of its variance. Under a natural spline with knots at 0.17, 0.57, 1.18, 3.95 it rests on 3.8, the largest carrying 41.1%.

A value read off a curve

No coefficient of a spline is read; what is read is the fitted curve's value at a covariate value of interest, and that value is a weighted sum of the outcomes with weights the design fixes before any outcome is seen. In the tail of a lognormal covariate a natural spline's value rests on 4.8 of a hundred points, fewer than the squared term's coefficient's 6.9, and the squared term's own value there rests on 10.2 — yet it is three times further from the truth, because its extra points are borrowed from where the curve has a different shape and buy no precision at all. At four hundred points, moving the last knot past the value raises its count threefold and halves its error; doubling the data does neither.

spread · Summary

Ten studies every study shares the control: the three combinations read as though independent, by the correlation. Counted on 400,000 null sets of ten one-sided studies at each correlation, every study shares the control. Fisher: 5.00%, 9.54%, 12.83%, 15.01%, 16.63%, 17.88%, 18.80%. Stouffer: 5.00%, 11.51%, 16.29%, 19.54%, 22.20%, 24.23%, 25.86%. Tippett: 4.98%, 4.86%, 4.68%, 4.45%, 4.17%, 3.72%, 3.27%. Smallest of three: 9.66%, 14.44%, 17.94%, 20.41%, 22.55%, 24.34%, 25.88% — at correlations 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6. Each combination is read at its critical value for independent studies.

Studies that share a control

Ten comparisons against one control arm of the same size are correlated at exactly one half, and combined as though they were independent, Stouffer's sum rejects 24.23% of sets in which nothing works, Fisher's 17.88%, while Tippett's smallest p-value falls to 3.72%. Repaired with the correlation the design states, each is exact again — and the smallest of the three is then inflated less than for independent studies, 7.05% against 9.66%, because sharing makes the three combinations agree. The two inflations the dependence was expected to stack do not stack.

testing · Uniformity

A cross-entropy fit of a normal proposal to P(Z > 5), stage by stage, fitting mean and spread or the mean alone. Each stage draws a thousand points from the current proposal, keeps the top tenth, and refits the normal to them, each weighted by the target density over the proposal's. Fitting the mean alone, the levels are 1.24, 2.99, 4.59, 5.00 and the fit reaches 5 at stage 4, ending at N(5.199, 1). Fitting mean and spread from the same seed, the spread goes 1.000, 0.456, 0.244, 0.096, 0.054, 0.043 over the first six stages, roughly halving each time, and after 30 stages the level has stalled at 3.728 with a spread of 1.0e-6, never reaching 5.

A proposal fitted to its own draws

The cross-entropy method fits an importance-sampling proposal from its own pilot draws, and asked to fit a normal's mean and spread to P(Z > 5) it fails before the question of variance arises: the spread halves at every stage, the level stalls near 3.7, and 95% of runs never reach the threshold, so the interval covers 1.1% of the time. Its ideal end point, the normal closest to Z conditioned past 5, is N(5.187, 0.181²), which has an infinite variance inside its own draws' reach and covers 93.7% at a thousand draws and 90.8% at ten thousand. Fitting the mean alone lands within a tenth of the best shift and covers 94.5%.

method · Seeds

Coverage of the cluster-robust slope interval at six, ten and twenty clusters of unequal size, by reference. 5,000 studies of 300 rows at each setting, within-cluster correlation 0.2. 6 clusters, equal sizes: 89.1%, 92.6%, 96.0%; 6 clusters, 5:1, big at the edges: 80.1%, 88.1%, 95.2%; 6 clusters, 5:1, big at the centre: 93.8%, 95.4%, 96.8%; 6 clusters, 15:1, big at the edges: 76.2%, 86.8%, 94.5%; 10 clusters, equal sizes: 91.1%, 93.0%, 95.8%; 10 clusters, 5:1, big at the edges: 86.3%, 90.3%, 94.7%; 10 clusters, 5:1, big at the centre: 94.5%, 95.7%, 96.4%; 10 clusters, 15:1, big at the edges: 84.9%, 89.6%, 94.8%; 20 clusters, equal sizes: 92.9%, 93.8%, 95.1%; 20 clusters, 5:1, big at the edges: 90.9%, 92.7%, 94.8%; 20 clusters, 5:1, big at the centre: 94.5%, 95.0%, 95.6%; 20 clusters, 15:1, big at the edges: 90.6%, 92.4%, 94.7% — for the usual variance on G − 2, the bias-reduced variance on G − 2, and the bias-reduced variance on its effective count.

An effective number of clusters

Ten clusters of thirty rows, a covariate spread evenly across them, and a cluster-robust interval for the slope read against a t on G − 2 = 8 covers 91.1%. The slope needed about five degrees of freedom, not eight, because the two outermost clusters carry 49% of its variance and the middle two 0.6%. A count computed from the sizes and positions before any outcome is seen — 4.45 here, 3.32 when the big clusters sit at the edges — with the bias-reduced variance covers between 94.5% and 96.8% in all twelve layouts measured, where the usual interval on G − 2 covers between 76.2% and 94.5%.

sandwich · Misspecification

A factor estimated from the panel

When sixteen pairs feel a common shock with loadings from a half to one and a half, removing the cross-sectional average leaves the pooled test rejecting an unrelated panel 13.3% of the time. Estimating the factor and each pair's loading instead brings that to 8.9% — and no further, because each loading comes from the same two hundred dates however many pairs there are — and it costs power against the average at the right critical value, 73.8% against 82.5%. Removing the shocks as observed restores 4.5% exactly and finds a shared relation 7.5% of the time: a series tied to its partner carries the partner's factor in its level, not its own shock.

cointegration · Spurious

Where the extremal index matters to a return level

A return level from a declustered fit depends on how often clusters arrive, which is the extremal index estimated from the same record. For a level reached once in two periods of a twenty-period record the index's error is a quarter to four fifths of the level's variance, and an interval that leaves it out covers 59.0% when clusters are long; put in, 90.0%. For a level reached once in a hundred periods its share is under two and a half per cent and adding it changes no interval at all — what fails there is the shape, fitted from however many clusters the dependence left: 82.3% coverage from forty-six clusters, 65.7% from nineteen.

extreme · Extremes

The bias a tolerance on the second moments leaves where a square drives both the assignment and the outcome. 600 samples a point. Bias at δ = 0, 0.02, 0.05, 0.1, 0.2, 0.5: −0.0074, 0.0500, 0.1353, 0.2782, 0.5473, 0.6983; coverage 89.7%, 89.5%, 72.4%, 26.5%, 1.0%, 1.0%. The tolerance spent in full at the outcome's slope on the square would give 2·δ·√2: 0.0566, 0.1414, 0.2828, 0.5657 at the four smaller values; balancing the means alone gives 0.6973.

A balance allowed a tolerance

Weights that balance the means exactly and the second moments only to within a tenth of a standard deviation were supposed to keep the second-moment repair and recover the samples exact balance loses. Where a square drives both the assignment and the outcome they keep neither half of the repair: the bias is 0.2782 against −0.0074 with exact balance, and the interval covers 26.5%, because the weights spend the whole tolerance — every second-moment gap sits at the edge of its band — and the outcome converts a gap of a tenth into bias at its own slope, 2·δ·√2 = 0.283. At thin overlap the same tolerance does recover samples: 89.0% have weights at all, against 62.0% with exact balance.

weights · Weighting

Named alongside it

The objects these essays reach for when they reach for this one.

Monte Carlo Reference distribution Autocorrelation Closed form Covariate balance Coverage Dependence Randomisation test Degrees of freedom Markov chain Monte Carlo Rerandomisation Confidence interval

All concepts