The collection

Every essay — page 12

Essays 265 to 288 of 613, in the same order.

More arms than two

Minimisation balances a trial by making the arms' counts even inside every factor level. With two arms there is one way to measure how uneven two counts are. With three there are several, they are all called minimisation, and they send different patients to different arms — while the ratio a trial was designed to deliver quietly disappears unless the score was told about it.

Four analyses of the same 3-arm trials, under a true null. 250 trials of 150 patients, 3 arms, minimisation with p = 0.85, 99 re-randomisations for each exact test. Two statistics — an F on the arms alone and an F on the arms after the balanced factors — against two reference distributions: the table the statistic is named for, and the distribution the allocation rule itself generates when the outcomes are held fixed and the rule is re-run. Only the first cell is wrong, and it is wrong in the direction that costs power rather than the one that manufactures findings: 0.0% where 5% is claimed. Either repair works — adjusting for what the rule balanced, or asking the rule what it would have done.

The analysis after three arms

An unadjusted analysis after a two-arm balancing rule rejects 0.6% of true nulls where it claims 5%. With three arms and a deterministic rule it rejects none at all — and the repair is the same repair, which is a sentence and a column in the model.

8 figures · Assignment, part 6

What a guesser gets, and what a guesser gets for nothing. 600 trials of 150 patients under a fully deterministic rule, with an investigator who knows the rule, the factors and every assignment so far. The guess rate falls with the number of arms — 87.8% at 2, 86.2% at 3, 81.0% at 4 — which reads like a trial getting safer and is not: what a guesser can trade on is the excess over the 50%, 33%, 25% they would get by naming an arm at random, and that goes the other way, from 1.76× chance at two arms to 3.24× at 4. The gap a guesser manufactures between the best and worst arm under a true null is 0.759, 0.733, 0.711 standard deviations — nearly unchanged.

Guessing one arm in three

A balancing rule is guessable because it is balancing. With three arms the next assignment is worked out less often than with two — and by more, relative to what a guesser gets for nothing, and the damage they can do is almost unchanged.

9 figures · Assignment, part 7

What a guesser gets from less than everything. 1,000 three-arm trials of 150 patients under minimisation on three factors with the preferred arm taken with probability 1, and five investigators guessing each next assignment from different information. Knowing the rule, the factors and every assignment so far: 86.3%. Knowing everything about the patients at their own centre and nothing about the others, and applying the rule's own score to that: 52.8%. Knowing only how many of their own centre's patients went to each arm: 55.5%. Knowing only the trial's arm totals, which a drug-supply system shows anyone who reorders: 54.5%. A guess at random gets 33.3%. The cruder the information, the less the guesser gets — except that the rule's own score, applied to a partial record, does worse than simply counting.

A guesser who sees the totals

The bound on how guessable a three-arm minimisation is assumes an investigator who sees everything. One who sees only the arm totals — what a drug-supply screen shows — names the next arm 54.5% of the time against a chance 33.3%, and manufactures 45% of the full guesser's bias.

4 figures · Assignment, part 8

The guess rate through a trial. Guess rates in each tenth of a trial of 150 patients on three arms with the rule taking its preferred arm with probability 0.8, over 1500 trials. knowing everything: 65.9%, 72.2%, 72.0%, 72.8%, 72.4%, 72.2%, 72.5%, 72.7%, 72.7%, 72.4%. a joint table learned in earlier trials: 53.4%, 55.4%, 54.8%, 55.1%, 54.3%, 53.8%, 54.9%, 54.8%, 54.1%, 54.8%. the centre's totals: 45.4%, 50.3%, 50.2%, 50.8%, 49.8%, 50.2%, 50.9%, 50.6%, 50.4%, 50.7%. following whichever has been right: 46.5%, 50.2%, 49.3%, 49.6%, 48.9%, 48.7%, 49.3%, 49.0%, 49.0%, 48.7%. a joint table learned in this trial: 36.4%, 43.8%, 45.0%, 46.2%, 46.7%, 46.8%, 48.1%, 48.5%, 47.9%, 48.9%. the centre's own factors, scored: 46.8%, 51.4%, 49.7%, 49.9%, 48.4%, 48.6%, 48.5%, 47.7%, 47.6%, 46.7%. Chance is 33.3%.

The guesser who learns

An investigator who watches a three-arm minimisation can tally which of several partial readings has been right so far and follow the leader. It does not help. Following the leader names 48.9% of assignments where the best single reading names 49.9%, and a table of what the three readings say together, learned inside the trial, starts at 36% and has not caught up by a hundred and fifty patients. Learned over earlier trials, that table reaches 54.5%. The full guesser's 71.8% stays out of reach of all of them, because the gap to it is information rather than practice.

3 figures · Assignment, part 9

The best of a set, and what the search costs

Comparing two forecasters is a test. Comparing eight is a multiplicity problem on top of a dependence problem, and the two do not separate: eight windows of one series carry the multiplicity of about two independent comparisons and eight separate problems carry eight, so a correction that charges for the number of models is wrong in both directions. The reference distribution has to be over the whole set — and once it is, the models nobody would have run turn out to cost more than the ones that were close.

Three quantities, and only one of them crosses zero. Two forecasts of an AR(1) — the last value carried forward and the mean of the last 60 observations — at 1 step ahead. The curve through zero is σ₁² − σ₂², the difference in expected squared error that a comparison of accuracy tests; it changes sign at φ = 0.4922. The two curves above it are σ₁² − σ₁₂ and σ₂² − σ₁₂, the quantities the two encompassing tests are about, and neither of them comes near zero anywhere: the smallest value either takes across the range is 0.008 times the variance of the series. All three are closed forms in φ, R and h with no simulation in them. Equal accuracy is one hypothesis about this picture and encompassing is another, and a set of numbers can satisfy either without the other.

What the other forecast adds

Two forecasters, one series, and two different questions about them. Which is more accurate has an answer that changes with the persistence of the series; whether either is redundant has an answer that never changes at all.

9 figures · Forecast, part 5

Eight candidates, one of them exactly as good as the benchmark. The candidate set: moving averages of the last 1, 2, 3, 5, 8, 13, 21 and 34 observations, each drawn as its expected squared error divided by the benchmark's — the mean of all 60. The persistence is not chosen, it is solved for: at φ = 0.4895 the best candidate in the set, the average of 2, has exactly the benchmark's expected squared error, and every other candidate is worse by between 0.5% and 5.1%. So the null that no candidate beats the benchmark is true, with one candidate on its boundary. Everything a set comparison claims about its own error rate has to be measured here, because anywhere further inside the null every procedure flatters itself.

Eight forecasters and one benchmark

A set of forecasters is a multiplicity problem on top of a dependence problem, and the two do not separate. Eight windows of one series carry the multiplicity of two and a half independent comparisons; eight separate problems carry eight.

9 figures · Multiplicity, part 5

Sixteen candidates nobody would have run, and what they cost. At φ = 0.65 one candidate in the original set is genuinely better than the benchmark, and the question is how often each procedure finds it. The added candidates are stale copies of the last value — read two, four, six … steps late — every one of them worse than the benchmark by at least 43%, and not one of them is ever the best candidate in a sample. The reality check goes from 35.5% to 0.0% as they are added, because its reference distribution has to assume every candidate is exactly as good as the benchmark and sixteen such assumptions is a critical value nothing reaches. The recentred version, which drops from the recentring the candidates the data has already ruled out — 15.1 of 24 of them — goes from 25.5% to 24.5%. The top line never moves: reporting the winner's own p-value cannot notice a change to a set it never looks at.

The models that were never in the running

A reference distribution for a set has to assume something about every candidate in it. Assuming that all of them are as good as the benchmark is what makes the reality check honest, and it is what sixteen hopeless candidates use to destroy it.

9 figures · Multiplicity, part 6

A distribution drawn from the null

Between nested models the ordinary comparison statistic has a null distribution centred at minus one and a 95% point of a quarter. A correction to its mean repairs the centre and leaves the shape; simulating the null repairs both.

6 figures · Bootstrap, part 4

Power as the recentring's threshold moves. The recentred set test's power at a persistence of 0.65, where one forecaster genuinely beats the benchmark, over 300 draws of 100 origins, as its threshold runs from zero to three times Hansen's. With the eight windows alone: 30.3%, 28.3%, 26.0%, 26.0%, 26.0%, 26.0%. With sixteen stale candidates added: 30.3%, 28.3%, 24.3%, 21.0%, 19.3%, 17.7%. The studentised reality check, which sets nothing aside, has 26.0% and 17.7%.

The threshold the recentring reads

A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.

3 figures · Multiplicity, part 7

What Hansen's threshold costs, by the number of origins. The power a threshold of zero keeps over Hansen's, on the same draws. with sixteen stale candidates: 7.5%, 7.3%, 6.8%, 5.3%, 3.7%, 6.0%. the eight windows alone: 2.7%, 3.8%, 4.7%, 5.3%, 3.7%, 6.0%, at 25, 50, 100, 200, 400, 800 origins. The price does not fall as the sample grows.

A threshold against its sample

A recentred test's threshold exists so that the test is right in the limit, and at a hundred origins it costs a fifth of the power against a stale set. The natural defence is that the price falls as the sample grows and the guarantee starts to earn it. Across twenty-five to eight hundred origins it does not: a threshold of zero keeps four to seven points more power at every size, and its size at the corner stays at the level from fifty origins on. What does change with the sample is the stale set, which stops costing anything by two hundred origins under Hansen's rule and by eight hundred under the most cautious.

3 figures · Multiplicity, part 8

What the design is asked to guarantee

Two halves of one question that were never separated: which parameters a design is for, and how much precision is enough. Protecting one parameter of a non-linear model over a range of its own values is a worst case of a ratio of determinants, and it is not a special case of either problem it is made of. Letting the experiment stop when it is precise enough is the adaptation with a theorem against it — the rule stops when its own noise estimate is low, so the interval it produces is short.

Two designs for one model, and the weights are not equal. Where the runs go, for the same two-parameter model at K = 1 and a ceiling of T = 10. The D-optimal design for both parameters is the familiar one: half the runs at 0.8333 and half at the ceiling. The design for the half-saturation constant alone moves the lower setting down to 0.6040 — where the response curve is still bending, which is where K is visible — and, unlike every design in the two fields before this one, it does not split the runs evenly: the weights are exactly 1/√2 and 1 − 1/√2, 0.7071 and 0.2929, at every K, V and T. The equal weights of D-optimality were a consequence of asking about both parameters at once, and nobody had to notice while that was the only question being asked.

An efficiency that is a ratio

A design chosen for a model is not a design chosen for the parameter somebody wanted. Asking for one of two parameters moves the runs, unbalances the weights, and costs the other question exactly 15.07% — at every setting, because it is algebra.

9 figures · Criterion, part 6

Four designs, scored on the one parameter that was wanted. Every design scored by its Ds-efficiency for K at 13 true values across a 16-fold range. The peaked curve is the subset design built at the guess K = 1: 100% there and 42.1% at the worst point of the range. The flat curve is the maximin-Ds design, never above 64.8% and never below 61.2%. Between them is the maximin design for the pair — a robust design, protecting something else, and worth 41.9% at worst here. The lowest curve is the D-optimal design at the guess, which is what an experimenter who wanted K and looked up a design for the model would actually run: 29.1% at the worst point, against 61.2% available.

Protecting one parameter over a range

A design for a non-linear model is optimal at a guess. A design for one of its parameters over a range of guesses is a worst case of a ratio of two determinants, and it is not a special case of either problem it is made of.

9 figures · Local design, part 7

One rule keeps its promise and the other keeps its budget. Both stopping rules at five requirements, 1,500 experiments each, with a first stage of 5. The upper curve is the two-stage rule: 97.1%, 96.2%, 95.9%, 96.1%, 96.0% — at or above 95% at every point, which is a theorem rather than a tendency, because its interval is built from a spread estimated before the stopping point was chosen. It pays 2.06×, 2.01×, 2.00×, 1.99×, 1.99× the observations that knowing σ would need. The lower curve is the rule that re-estimates after every observation: 94.5%, 89.3%, 91.0%, 91.5%, 94.3%, on 0.98×, 0.87×, 0.88×, 0.93×, 0.96×. The second rule is the one anybody would run and the first is the one whose claim is true.

Stopping when it is precise enough

An experiment that runs until its estimate is precise enough is the natural design and the one with a theorem against it. Its two-stage cousin keeps its promise exactly, for every unknown spread, and pays twice the observations for it.

8 figures · Stopping, part 5

Four intervals at one stopping time. 3,500 experiments under the sequential rule with a first stage of 5 and a required half-width of 0.4, which is a demand that knowing σ would meet with 24.0 observations and which the rule meets with 20.8. The rule's own interval covers 90.3%. Replacing the fixed width by a t interval on the same data gives 92.0%. Keeping the rule's own random sample size and drawing a fresh sample of that size gives 89.8% at the fixed width and 95.5% for a t interval — so the sample size being random costs nothing, and the sample size being chosen by the data the interval is built from costs the rest. The spread estimated at the stopping moment is 17.1% below the truth, which is the same fact one level down.

The interval after a stop it chose

A rule that stops when the estimated precision is good enough stops on the samples whose estimate was small. Its interval covers 90% and claims 95%, and a fresh sample of the same random size covers 95.4%.

8 figures · Stopping, part 6

A third stage, between the two rules. Four rules for one requirement: a half-width of 0.4 at 95%, which knowing σ would meet with 24.0 observations, after a first stage of 5. Each bar is the mean number of observations as a multiple of that, and the note beside it is the coverage. The two-stage rule spends 2.03× and covers 95.42%, computed exactly. The three-stage rule — a second stage to half of what the first spread asks for, the spread re-read on everything, a third stage to the rest — spends 0.86× and covers 88.42%; given 8.50 more observations in every experiment's last stage it covers 95.00% on 1.20×. The rule that re-estimates after every observation spends 0.88× and covers 90.5% over 4,000 drawn experiments; it is the only row that has to be drawn, because it is the only one whose stopping point shares observations with the mean.

The fifth a third stage never reaches

A three-stage stopping rule keeps the two-stage rule's independence and the sequential rule's budget. With a first stage of five it covers 88.4% — worse than the rule it was meant to repair — because in a fifth of experiments the second stage buys nothing.

5 figures · Stopping, part 7 $Three repairs at one requirement. Coverage against spend at a half-width of 0.4 and a first stage of 5, all exact. A floor on the second stage at 5, 8, 10, 12, 15, 18, 21, 24 observations covers 88.4%, 91.4%, 92.4%, 93.1%, 93.9%, 94.6%, 95.3%, 96.0% at 0.86, 0.94, 0.97, 1.00, 1.02, 1.05, 1.09, 1.14 times the oracle's spend. The second stage reading with t, with its fraction dialled from 0.2 to 1, covers 88.1%, 89.0%, 90.6%, 92.1%, 93.4%, 94.4%, 95.2%, 95.9%, 96.4% at 0.84, 0.91, 1.02, 1.17, 1.34, 1.52, 1.70, 1.89, 2.08 times. A constant at the end covers 88.4%, 89.4%, 90.4%, 91.3%, 92.1%, 92.9%, 93.6%, 94.2%, 95.0%, 95.7% at 0.86, 0.90, 0.94, 0.98, 1.02, 1.06, 1.10, 1.14, 1.20, 1.26 times.$

A floor that reads the first stage

A three-stage stopping rule fails at a first stage of five because a low first reading asks for nothing more, and the only floors that repair it were set near the answer, by somebody who already knew roughly what the spread was. Inflating the first reading by Stein's own factor needs no such knowledge, and it does cut the experiments left stuck from a fifth to a fifteenth. But those still stuck cover less than before, the rule needs a different dial at every requirement, and a floor of 26 observations covers 96.5% at the spend where the rule covers 92.1%, even met by a spread it was not set for.

3 figures · Stopping, part 8

Balancing what has no levels

Every balancing rule in the two fields before this one reads a level. Age, blood pressure and a baseline score have none, and the first thing that happens to them is that somebody invents some — a choice with a cost available in closed form before any data exists: a median split can see exactly 2/π of a normal covariate, so a rule that balances its two halves perfectly still leaves three fifths of a coin's imbalance. A rule that reads the number instead does not beat that by a factor; it beats it by a rate.

What a median split can see. A standard normal covariate with its median marked, and the mean of each category as a vertical rule: -0.7979, 0.7979. A rule that balances the categories is balancing those numbers and nothing else, so the part of the covariate it can act on is the variance between them — 0.6366 of the total, which at two categories is exactly 2/π because the two half-normal means are ±√(2/π). The rest, 0.3634, is variation inside the categories that the rule cannot see and does not touch: the assignment within a category is still a coin. Everything the next figure measures is a consequence of this one, and it is available before any unit has arrived.

A covariate with no levels

Every balancing rule on this site reads a level. Age and blood pressure have none, so somebody cuts them into categories — and a median split can see exactly 2/π of a normal covariate, whatever the rule does with the halves.

8 figures · Assignment, part 8

Two rates, not a factor. The standard deviation of the covariate imbalance under three rules, at five trial sizes, 260 trials each, on log axes. The upper line is a coin: its slope is -0.489, against a closed form of exactly −½. The middle line is minimisation on a median split; its slope is -0.519 — the same rate — because inside a category the assignment is still a coin, and what it buys is the constant, 0.654 of a coin's at n = 200. The lower line is the rule that reads x and maximises the information about the treatment effect: slope -0.987, nearly twice as steep. Its advantage is therefore not a number that can be quoted — it is 0.258 of a coin's at n = 50 and 0.065 at n = 800, and it keeps going.

The rule that reads the number

Stop categorising and let the rule read the covariate itself. What it should minimise is not an invented distance but the variance of the effect being estimated — and what comes back is not a better constant but a different rate.

9 figures · Assignment, part 9

Three analyses of the same trials, none of them wrong about the data. 320 trials at n = 60 with no treatment effect at all, so every rejection counted is a false one, and a covariate that drives the outcome with coefficient 1. The unadjusted comparison is at 5.94% after a coin — its level — and at 0.00% after the rule that reads the covariate: the design removed the imbalance and the analysis is still pricing it. Adjusting for the covariate gives 4.06%, and the rule's own reference distribution — hold the outcomes, re-run the rule 199 times, count — gives 3.13% against the 4.5% that 199 draws can deliver. The last of the three has to be told the assignment rule and nothing else, which is the one thing the experimenter certainly knows.

What the balanced trial is worth

A rule that reads the covariate removes three quarters of the imbalance. An analysis that does not know it happened prices the imbalance anyway, rejects one true null in two hundred instead of one in twenty, and finds a real effect less often than a coin-tossed trial does.

8 figures · Randomisation, part 3

Balancing more than one number

The criterion generalises to several covariates without a word changing, which makes the question what it is worth rather than whether it can be done. Each one added takes a share of the assignment's freedom, and the imbalance left in every one of them rises.

8 figures · Blocking, part 2

A number read with error

A rule that balances the recorded covariate balances the true one only as far as the recording reaches. At a reliability of 0.8 it leaves 45% of a coin's imbalance in the covariate that drives the outcome, back on the coin's 1/√n rate — and a median split turns out to be a recording with reliability 2/π.

3 figures · Assignment, part 10

A second reading or a second covariate

A rule that balances a recording balances the truth only as far as the recording reaches, and a second reading of the covariate raises that reach. So does a first reading of something else. At a reliability of 0.8 the first reading of a second covariate does more once that covariate matters a third as much as the first, and the threshold is exact: the square root of (1 − λ)/(1 + λ), with no trial size in it. A protocol's list of measurements turns out to be a small knapsack, and it is rarely filled by reading anything twice.

3 figures · Assignment, part 11

Searching among fitted models

The set field compares forecasts nobody estimated. A specification search compares a benchmark with a table of variants that all contain it, and two things change at once: every variant is behind before the search begins, by an amount with a closed form in the shape of the table, and the reference distribution for the winner can no longer be resampled from the data — it has to be generated from a model. The repair the nested case asks for, applied row by row, takes a table from its nominal level to a quarter.

What each variant loses before anything has been searched for. The mean loss differential of each of the eight variants against the benchmark, over 600 tables of 60 origins, with every fit given 71 rows. The series is an AR(1) and every variant adds a lag whose coefficient is zero, so in population the two forecasts are the same forecast and the difference drawn here is estimation noise and nothing else. The marked line is σ²(q₁ − q₀)/n = -0.01408, which is an expression in how many coefficients each model has and how many rows it was fitted on — it knows nothing about the series, the persistence or which lag the variant added, and every bar is within a fifth of it. This is the amount a reference distribution recentred at each column's own sample mean believes the candidates are already behind by.

A table of nested models

A benchmark and eight variants of it, each adding one thing. Every variant is behind before the search begins, by an amount that can be written down before the data exists — and the two most natural ways of reading the table are wrong in opposite directions.

8 figures · Forecast, part 6

A null with a model in it

The distribution to read the winner of a table against cannot be resampled from the data, because the data does not contain the null. It has to be generated from a model — which is the assumption the resampling was chosen to avoid.

8 figures · Bootstrap, part 5

Fields Threads Series Concepts Figure library Search