Normal Distribution
nothing is called 95% until it has been counted
The collection

Every essay — page 19

Essays 433 to 456 of 613, in the same order.

The diagnostic after the trial

The test for whether a balanced-assignment walk can reach the whole admissible set is run on a covariate function, before any outcome exists. Run instead on the difference in arm means it is the same test — under the sharp null the outcome is a fixed column — and it is about the statistic the p-value is actually built from. Two findings: an outcome is a probe nobody chose, and on a set that is genuinely split three in ten of them see nothing at all; and the published two-sided p-value is exactly right on half a reference distribution, while a one-sided one is not.

Two diagnostics, one answer, two different moments. The two-chain statistic on a covariate probe and on the trial's own difference in arm means, at seven tolerances of a fourteen-unit rule, against the enumerated truth. Both are quiet wherever the set is one set and both fire wherever it is not, at every tolerance — which is what says the outcome probe is the same test rather than a resemblance of it. The difference between them is not accuracy and it is not power. It is when: the covariate probe can be run before a single outcome exists, when a practitioner can still loosen the rule or change the sampler, and it can be run again on a different function if it comes back quiet. The outcome probe runs after the trial, on the one column the trial produced, and what it can do with a positive verdict is repair the p-value rather than the design.

The statistic the p-value is about

The test for whether a balanced-assignment walk reaches its whole set is run on a covariate function chosen before the trial. Run on the difference in arm means it is the same test, and it is about the number the trial publishes.

8 figures · Randomisation, part 12 The probe a trial has is the probe a trial got. What the two-chain test says when it is run on the trial's own difference in arm means, over 24 outcomes on one fourteen-unit set. The set is in 2 mirror components — that is enumerated, not inferred — so every quiet reading is a miss. 29% of them are quiet. The reason is in the enumerated set rather than in the run: how far the two components are apart on a given probe ranges from 0.001 to 4.938 of a within-component spread across these outcomes, a factor of several thousand. Both covariate probes — chosen before any outcome existed, and replaceable if they had been quiet — report the split. An outcome cannot be chosen and cannot be replaced.

A probe nobody chose

On a set that is definitively in two pieces, seven of twenty-four outcomes report nothing at all. Every covariate probe reports it. What separates them is not accuracy — it is that one of them can be chosen and the other is what happened.

8 figures · Randomisation, part 13 The one thing a trial always reports is the one thing that survives. How wrong three p-values are when they are computed over the half of the admissible set a single walk can reach, rather than over all of it, at a fourteen-unit trial where the whole set can be enumerated. The two-sided p-value on the difference in arm means — the number a trial publishes — is wrong by exactly nothing, at every row, to machine precision. That is not luck: the two components are complement pairs and the difference in arm means is exactly negated by the complement, so the distribution of its absolute value is the same on both. A one-sided p-value on the same statistic is out by as much as 0.112, and the largest response observed in the treated arm — a safety reading rather than an effect, and the one statistic here that is not odd under the complement — by as much as 0.172. The defect survived because the commonest thing anybody computes is the one quantity it cannot touch.

Half a reference distribution

A walk that reaches half its admissible set reports the two-sided p-value exactly right, to the last digit, for ever. A one-sided one it puts on the wrong side of five per cent about once in thirty.

8 figures · Reference, part 11 Where the set stops being one set. How many of the 3,432 equal splits of fourteen units a balancing rule admits, as the tolerance tightens, with the number of components single swaps leave it in. The set falls from 886 to 84 assignments, and somewhere in that fall it stops being connected: at 0.8 it is in 2 pieces and every assignment's complement is in the other one. Nothing about the rule changes at that point and nothing a chain reports changes either, which is the whole difficulty — the acceptance rate, the stationary distribution and the detailed balance are all in order on both sides of it.

Before the trial and after

The same diagnostic run at two moments answers two different questions. Before, a positive verdict changes the design. After, it changes which number gets reported — and only for the numbers the defect can reach.

9 figures · Assignment, part 10 How many starts it takes to land in two pieces. Averaged over the split designs at each of three tolerances, the chance that k starting assignments drawn uniformly from the admissible set — what hunting for a start by rejection gives — do not all fall in one component, which is the most any diagnostic built on comparing chains from those starts can detect. At 0.6: 28% with 2, 47% with 4, 63% with 8, 77% with 16, 88% with 32, 95% with 64, 99% with 128. At 0.8: 12% with 2, 21% with 4, 32% with 8, 46% with 16, 62% with 32, 77% with 64, 90% with 128. At 1.0: 4% with 2, 8% with 4, 13% with 8, 21% with 16, 33% with 32, 47% with 64, 64% with 128. The median split design needs 21, 109 and 439 starts at the three tolerances to expose its split with probability 95%. With an even allocation the mirror makes two starts enough whenever the split is a unit.

After the mirror is gone

The two-chain diagnostic starts its second chain at the complement of the first's start, which exists only when the arms are equal. At five treated against ten, enumerated on two hundred designs, a split admissible set is one big piece and some crumbs — the median puts 97.3% of its assignments in one component — so two random starts see the split on 12% of split designs and 95% needs a median of 109 starts. Worse, the rule fixes units in a connected set all the time, so a unit that never moves stops being evidence. A hunt that forces such a unit into the other arm finds 41 of 70 splits; 29 carry no unit that marks them.

4 figures · Randomisation, part 14

The other half of the dependence

What a balancing rule can remove of an interaction was measured across six marginals with one joint law of the ranks held fixed. Change the copula instead and the two exact zeros come apart for two different reasons. A median split's zero is arithmetic — a centred median split squares to a quarter identically — and holds under every copula there is. A mean's zero needs the copula to be symmetric under reflection as well as the covariate to be symmetric: it is exactly nothing under a Gaussian, t or Frank copula and 7.71% under a Clayton, at the same rank correlation and with a normal covariate throughout.

One zero is arithmetic and one is a symmetry. What two balancing rules remove of the interaction they are aimed at, on five joint laws of the ranks matched at a Spearman correlation of 0.4, with a normal covariate throughout. A rule holding a median split of each covariate removes exactly nothing of the product of the splits under every one of them, including the two that are not symmetric under reflection — and the reason is not a symmetry at all: a centred median split takes the values ±½, so its square is a quarter identically, and the interaction is orthogonal to both main effects whatever the joint law is. A rule holding the mean of each removes exactly nothing under the three radially symmetric copulas and 7.71% under the two that are not. Bars at the floor are exact zeros; the axis cannot draw 9e-32.

A zero that is arithmetic

A median split's exact zero was explained by a symmetry of the latent normal. It holds under a Clayton copula, which has no such symmetry, because a centred median split squares to a quarter identically.

8 figures · Criterion, part 14 The mean's zero is the copula's symmetry. Five copulas, each at a Spearman rank correlation of 0.4, with a normal covariate throughout — so nothing here is about the marginal, which is the whole of the earlier field. Horizontally: how far the copula's density is from its own reflection through the centre of the unit square, measured rather than read off the family's name. Vertically: what a rule balancing the mean of each covariate removes of their product. The three copulas at zero on the horizontal axis remove exactly nothing, to thirty decimal places. The two that are not symmetric remove 7.71%. A guarantee that held for six marginals turns out to have needed something the marginals could not have told anybody about.

The symmetry the marginals could not show

A mean's interaction zero needs the covariate to be symmetric and the copula to be symmetric under reflection. Six marginals could only ever test one of those, and the other is broken by the commonest kind of dependence there is.

7 figures · Criterion, part 15 Which tail the threshold is in. What a rule balancing a threshold at 1 on each covariate's own scale removes of the interaction between the two thresholds, on five copulas matched at a Spearman rank correlation of 0.4 with a normal covariate throughout. This rule never had a zero to lose — the earlier field establishes that under every marginal — so what is left is a size, and the size depends on where the dependence lives. A Clayton copula, whose density piles up in the lower tail, leaves 5.33%; the same copula turned over, so that it piles up in the upper tail where the threshold is, leaves 33.36%. Same rank correlation, same Kendall tau, same marginal, same threshold: 6.26 times the leak, decided by which end of the distribution the dependence and the cut are both in.

Which tail the cut sits in

The same copula and its reflection have the same rank correlation, the same Kendall tau and the same marginals. A balancing rule holding a threshold at a dose leaves 5.33% under one and 33.36% under the other.

8 figures · Criterion, part 16 A cut walked from the median into each tail. What a rule balancing a threshold on each of two normal covariates removes of the interaction between the two thresholds, with the cut moved from the 1st percentile to the 99th, on five copulas at a Spearman rank correlation of 0.4. Every copula reads exactly zero at the median. The three radially symmetric copulas are mirror images about it: the Gaussian peaks at 19.49% near the 95th percentile and is down to 13.44% at the 99th; the Frank peaks at 13.22% near the 85th and falls to 2.06%; the t is still near its highest, 37.45%, at the 99th. The Clayton climbs to 56.37% at the 1st percentile, in the tail its dependence is in, and never passes 5.39% in the other; its reflection is the same curve turned round.

A cut walked into the tail

Move a balancing rule's threshold off the median and the interaction it leaks rises from exactly zero — and then either turns back to nothing or climbs to a ceiling, depending on one property of the dependence. With both covariates cut, the whole regression collapses to two probabilities, and far out it settles on 2λ/(1 + λ) of the tail-dependence coefficient: 57.25% under a Clayton, 35.57% under a t, and zero under a Gaussian or a Frank however strongly the covariates are related.

5 figures · Criterion, part 17 A leak read off a cohort, at two cuts. The threshold rule's leak as a cohort of 1,000 units estimates it from its own two-by-two table: the bar is the middle eighty per cent of 600 cohorts' plug-in estimates, the open circle their median and the stroke the true leak, for five copulas at a Spearman rank correlation of 0.4. At the 90th percentile the estimate is centred on the truth and its middle eighty per cent spans about fourteen points: Gaussian 12.8% to 26.4% around 19.2%, t on four 21.5% to 35.7% around 28.9%, Frank 5.4% to 18.7% around 12.2%, Clayton 0.8% to 9.6% around 4.4%, Clayton turned over 35.9% to 49.5% around 43.1%. At the 99th it is not centred on anything: Gaussian 13.1% to 100.0% around 13.7%, t on four 17.5% to 72.9% around 37.4%, Frank 14.4% to 100.0% around 2.2%, Clayton 16.1% to 100.0% around 0.6%, Clayton turned over 37.0% to 75.7% around 56.3%, because on three of the five copulas between two cohorts in five and four in five have nobody past both cuts.

A leak read off a two-by-two table

The share of an interaction a threshold-balancing rule leaves alone needs one number from the data: how often a unit past the cut on one covariate is past it on the other. A cohort of a thousand estimates it to within a dozen points at the 90th percentile. At the 99th most cohorts have an empty cell in their table, and every one of those reads a leak of exactly one hundred per cent — a reading the bootstrap then confirms with an interval of zero width.

6 figures · Criterion, part 18

A block length chosen from the data

Two block windows were compared at each window's own best block length, which is the argmin of a quantity that needs the truth. Re-run on rules a practitioner could actually run, the ordering reverses: the taper wins at the best available length and at one estimated from the sample's own persistence, and the rectangle wins at a length written into a protocol and at the rule of thumb, at every sample size measured. And the whole argument is a third of the size of what estimating the length costs.

Three rules and a target none of them is aimed at. Which block length each rule picks, over 400 samples of 120 rows, for the tapered window. Two of the rules are points: a length written into a protocol is 8.00 on every draw and the rule of thumb is 4.00, because n to the one third does not read the data at all. The plug-in reads the sample's own persistence and lands at 14.36 with a standard deviation of 2.93. The length that would actually have been best on that draw averages 24.57 with a standard deviation of 16.23 and runs from 10 to 48 between its tenth and ninetieth percentiles. The target moves five times as much as the best estimate of it does, which is why no rule can be close to it and why the two that do not try are not merely worse — they are somewhere else.

The length nobody has

Every comparison of block windows in this collection is made at each window's own best block length. That length has a standard deviation of sixteen across draws and averages twenty-five. No rule is aimed at it.

8 figures · Bootstrap, part 15 Which window is better depends on who chose the block length. The margin between a rectangular block and a tapered one, on 400 samples of 120 rows, under four rules for choosing the block length. At the length that would actually have been best on each draw the taper is ahead by 2.12 points of a 35.8% error, at 22.1 paired standard errors; at a length estimated from the sample's own persistence it is ahead by 1.89. At the length this field's own figures use — eight — the rectangle is ahead by 2.04, and at the rule of thumb by 4.54. Every rule sees the same draws. What separates them is the length: the two rules that lose to the rectangle pick 4.00 and 8.00 where the best available is 24.57, and a tapered window at a quarter of the right length has thrown away most of what it was weighting.

An ordering that depends on the rule

The tapered block beats the rectangular one at the best available block length and at one estimated from the data. At a length written into a protocol, and at the rule of thumb, the rectangle wins — at every sample size measured.

8 figures · Bootstrap, part 16 The argument is a third of the size of the thing it is inside. Three quantities on one scale, in points of the error in a block resample's implied long-run variance, at 120 rows. The gap between the two windows at the best available block length — the whole subject of the comparison this field inherited — is 2.12 points. What the best rule a practitioner could actually run gives up against that same best length is 7.26, a factor of 3.42. What the rule of thumb gives up is 26.01. So the ordering between windows is worth establishing and is not worth arguing about, and the sentence that follows from it is not use the taper but estimate the block length, because that is where the points are.

What choosing the length costs

The gap between two block windows at the best available length is 2.12 points. What the best rule a practitioner could run gives up against that same length is 7.26. The argument is a third of the size of the thing it is inside.

8 figures · Bootstrap, part 17 A premium that shrinks and a payout that grows. The relative error of the trapezoid's implied variance with the block length read off the whole autocovariance sequence, minus its error with the length from the lag-one correlation alone, in points, at four sample sizes and 400 draws each. Under the first-order autoregression the plug-in's algebra was written for, reading the sequence costs 3.87 at 120, 2.37 at 240, 2.48 at 480, 1.15 at 960. Under the five-period moving average, which the plug-in reads as an autoregression at 0.8, it is 0.43 at 120, -1.72 at 240, -3.01 at 480, -3.77 at 960 — a tie at 120 rows and a gain that grows with the sample.

A length read off the whole sequence

The automatic block length reads one number off the sample — the lag-one correlation — and assumes the rest of the dependence is geometric. Reading the whole autocovariance sequence instead removes the assumption and puts the length where the law wants it. At a hundred and twenty rows it buys nothing, even where the assumption is wrong; it costs 3.87 points where the assumption is right, and by 960 rows that premium is 1.15 and the payout under a law the plug-in misreads is 3.77. Under long memory neither rule, nor the best length on each draw, gets within ninety per cent.

4 figures · Bootstrap, part 18 Five length rules, read on two intervals. The coverage of a 95% interval for the mean at the block length each of five rules picks, on the percentile interval and the studentised one built from the same resamples, under a five-period moving average and a first-order autoregression at 120 and 480 rows, 2,000 draws each. moving average at 120: percentile 88.5%, 85.3%, 86.4%, 86.3%, 88.8%; studentised 92.5%, 87.9%, 94.9%, 92.6%, 95.0%. moving average at 480: percentile 90.5%, 90.5%, 90.4%, 91.6%, 91.9%; studentised 91.2%, 91.2%, 94.5%, 93.3%, 95.0%. autoregression at 120: percentile 84.7%, 80.2%, 85.8%, 85.4%, 85.9%; studentised 90.3%, 83.0%, 92.0%, 90.9%, 94.3%. autoregression at 480: percentile 87.0%, 87.0%, 90.0%, 90.5%, 90.9%; studentised 88.6%, 88.6%, 93.0%, 93.0%, 94.2% (in the order eight, from the protocol, n to the one third, the plug-in, the whole sequence, the best on the draw). The rules spread over at most 5.6% on the percentile interval; the interval moves the same rule by up to 8.5%.

A ranking that belongs to the interval

Five ways of choosing a block length, read on the coverage of the interval each length builds rather than on the variance it implies. On the percentile interval the variance's verdict mostly survives: under a five-period moving average the plug-in and the whole-sequence rule tie at 120 rows and the sequence rule leads at 480. Build a studentised interval from the same resamples and the lead reverses — the plug-in's longer blocks cover 94.5% against 93.3% — because an interval that widens for its own uncertainty is paid for a long block where a percentile interval is punished for one. Under long memory no length and no interval covers two times in three.

5 figures · Bootstrap, part 19

A charge for a covariance's own dimension

The two charges that pick a band's width were derived for a regression coefficient, and a band's numbers are neither free nor entered the same way. Measured as an optimism against a second independent sample, a Bartlett band costs 0.374 of a log-likelihood unit a lag where a criterion charges one. The window's own weights are the mechanism — scale them and the charge scales with them, to within two per cent across three windows — and they are not the arithmetic: the level is three quarters of the sum, a different shape at the same sum costs more, and the charge rises with the law's persistence. Levying the right one moves every width in the table by a factor of four and the error by half a per cent.

The area under the window is what the band actually costs. The three windows' weight sequences at a width of 30 lags, drawn against the lag as a share of the window. A truncated window applies a weight of one to every lag inside it and zero outside, which is why its sum is the width and why every conventional charge is right for it — and it is a covariance matrix on almost no sample, so it cannot be used. The Bartlett window falls linearly to zero and its weights sum to exactly 15.000000000000004, which is half the width, at every width: Σ(1 − k/(L+1)) over k = 1 … L is L − L/2. The Parzen window sums to 11.13 here, three eighths of the width, and it gets there by holding a weight near one over the first few lags and then falling faster. A plug-in estimate multiplied by a weight below one is a shrunk estimate, and a shrunk estimate is worth less than a free one — which is the whole of why a charge levied per lag is a charge for parameters the window has already spent.

The charge nobody derived

A band of lags is charged one log-likelihood unit apiece, because that is what a regression coefficient costs. A band's numbers are not regression coefficients, and measuring what they actually cost puts the convention out by a factor of nearly three.

4 figures · Criterion, part 12 Four windows, one line, and one that is off it. The optimism measured for each window at a band of 30 lags, against what that window's weights sum to, on 2000 pairs of independent samples of 120 rows. The diagonal is where a window that spent exactly its summed weights would sit. Three of the points are one shape at three levels — the Bartlett window, its square and its cube, whose sums stand in the ratio 6 : 4 : 3 — and they lie on a line through the origin at 0.767 of the diagonal, with 0.033 between the highest and the lowest. Scaling the weights scales the charge by the factor the weights predict, which is what makes the weights the mechanism. The Parzen window has a comparable sum and a different shape, and it sits at 0.871: its weights stay near one over the first few lags, and the first few lags are where the information is. A weight sum treats every lag as equally informative and no sample does.

What a window leaves free

A Bartlett window's weights sum to exactly half its width, which is a candidate for what the band costs. Varying the weights without varying anything else says the weights are the mechanism; varying the shape at the same weight says they are not the arithmetic.

5 figures · Criterion, part 13 Cut the charge and the width follows it. The band width each charge picks, averaged over 400 draws of 120 rows under AR(1) at 0.8, with the standard deviation across draws beside it. Schwarz's charge — half a log n a lag, which is 2.39 here — picks 3.67. Akaike's picks 6.02. Charging the numbers the window actually leaves free, which is half the width, picks 10.12; charging what the optimism measures, 0.767 of that, picks 14.15. A charge and the width it buys are very nearly reciprocal, which is what a likelihood rising at a fixed rate a lag implies and is why the four answers span a factor of 3.86. The width that was actually best on the draw averages 13.90 and moves by 10.30 from draw to draw — three times as much as any rule's answer does.

A width that moves and an error that does not

Four charges give four widths a factor of four apart and four errors half a per cent apart. The derived charge wins, significantly, by a quarter of what was on offer — and none of the four is an estimate of anything.

4 figures · Criterion, part 14 What a lag of band costs, as the sample grows. The optimism one more lag of a Bartlett band adds, measured by fitting on one sample and scoring on an independent one, 2,400 pairs at each size, with two standard errors: 0.3745 ± 0.0176 at 120 rows, 0.3842 ± 0.0266 at 240 rows, 0.4523 ± 0.0392 at 480 rows, 0.4022 ± 0.0563 at 960 rows. The window's summed weights give one half a lag and the convention charges one.

The charge at four times the sample

A band of lags is charged a full unit apiece and is worth about three eighths of one at a hundred and twenty rows. The field's last essay said the error in that convention was harmless only because the error surface is flat, and that a steeper surface would give it a cost. A longer sample was the obvious place for one. It is not there: from 120 rows to 480 the error's spread across band widths falls by three quarters, and the derived charge's advantage over the convention falls from 5.5 standard errors to 1.7. What the sample does move is the charge itself, from 0.37 of a unit a lag towards the half a unit the window's weights predict.

4 figures · Criterion, part 15 Past the cap, long memory's surface is flat. The prediction error at every band width from four lags to eighty, above the lowest on the extended list, under long memory at d = 4/9 at three sample sizes, 600 draws each. At 120 rows the floor is at 12 lags and eighty is 0.0039 above it. At 480 the floor is at 30 lags and every width from twenty-four to eighty is within 0.0003 of it. At 960 it is at 48 and thirty to eighty lags are within 0.0000. The dashed line is the list's cap: the surface does not rise behind it, it levels off.

A cap that decides nothing

Every band-width list this field priced stopped at thirty lags, and long memory at a large sample is the one dependence whose best band wants to be wider. At 960 rows it does: the draw's own best width lies past the cap on 48.7% of draws. Lift the cap to eighty and the measured charge follows, from 28 lags to 46 — and the error moves by two ten-thousandths, in the wrong direction and inside its standard error, because the surface behind the cap is flat. The field's last conditional fails where it had its best chance.

5 figures · Criterion, part 16

The rate and the size of a disagreement

A sweep of what it costs to let every candidate choose its own tuning parameter reported a product and called it a cost. Separated — and the decomposition is exact, because a draw on which nothing disagreed carries exactly zero — the rate rises by half across the list and what a disagreement is worth does not move at all. A third quantity explains why: a disagreement costs something only when it changes which candidate the table selects, that happens on about an eighth of draws, and the eighth does not depend on the list.

One factor moves and the other does not. The two factors of the same average, each drawn against its own largest value so that they share an axis. The rate at which the fifteen candidates disagree about the tuning parameter rises from 28.6% at 4 values on the list to 43.3% at 8, a factor of 1.52. What a disagreement costs, given that there was one, is 0.00975 ± 0.00224 and 0.00848 ± 0.00113 at the same two points — 0.5 standard errors apart, and the paired comparison on the draws that disagree under both lists puts it the other way. The guess this field was written to test was that a longer list makes disagreements commoner and each one smaller. The first half is right and there is no second half.

A rate times a size

A sweep reported what it costs to let every candidate choose its own tuning parameter and found it flat across the list. It was reporting a product, and the two things multiplied together do not behave the same way at all.

4 figures · Order selection, part 12 What a disagreement costs, split on whether it decided anything. The regret from choosing the tuning parameter per candidate, on the draws where the candidates disagreed, split on whether the disagreement changed which candidate the table selects. Over 1200 draws at each list length: when the winner changes the regret is 0.02215, 0.03029, 0.03145; when it does not it is -0.00243, -0.00069, -0.00065 — negative, and small enough that it is inside two standard errors of nothing at every length. The whole of the cost lives in the first column, and the second column is not merely small but slightly the wrong sign: when the table's answer is unaffected, letting each candidate use its own window is a very slightly better rule than making them share one. So a disagreement about the tuning parameter is not a cost. A disagreement that changes the winner is.

The quarrel that changes the winner

A disagreement about the tuning parameter costs 0.031 when it changes which candidate the table selects and −0.0007 when it does not. The distance between the values disagreed about has nothing to do with it.

3 figures · Order selection, part 13 The quantity that does not depend on the list. The probability that letting each candidate choose its own tuning parameter changes which candidate the table selects — the product of the two moving shares — against the length of the list, over 1200 draws apiece. It is 14.2%, 11.9%, 12.3%: a spread of 2.2% across a list length that moves the disagreement rate by a factor of 1.52. This is the invariant the whole field turns on. Everything downstream of the winner — the coefficients, the regret, whatever a reader is going to quote — is a function of whether the winner changed, and how often that happens is not something the list controls. A longer list changes how often the candidates quarrel and not how often the quarrel matters.

How often it matters

The disagreement rate rises by half across the list and the share of disagreements that decide anything falls by nearly the same factor. Their product — how often the tuning list changes which candidate wins — sits at an eighth and does not move.

4 figures · Order selection, part 14 Which factor the sample settles. The field's decomposition at the full tuning list, by sample size, over 1,200 draws at each: how often fifteen candidates choose different values, what share of those disagreements change which candidate the table selects, and the product — how often the list changes the winner. The window: 120 rows 43.3%, 28.5%, 12.33%; 240 rows 17.5%, 14.8%, 2.58%; 480 rows 38.3%, 2.6%, 1.00%; 960 rows 11.6%, 2.2%, 0.25%. The order: 120 rows 48.9%, 21.6%, 10.58%; 240 rows 47.5%, 13.2%, 6.25%; 480 rows 46.1%, 5.1%, 2.33%.

What a longer sample settles

At a hundred and twenty rows, letting each candidate choose its own whitening changes which of fifteen candidates wins on about an eighth of draws, whatever the list, the table or the law. At four hundred and eighty rows it changes the winner on 1.0% of draws for a window and 2.3% for an order, and at nine hundred and sixty on a quarter of one per cent. The sample settles it through one factor only: how often the candidates disagree about the tuning value hardly moves, and the share of their disagreements that change the winner collapses.

3 figures · Order selection, part 15

FieldsThreadsSeriesConceptsFigure librarySearch