Normal Distribution
nothing is called 95% until it has been counted
Concept

Tuning parameter — where it appears

A dial a procedure is set by rather than a quantity it estimates — a window, a block length, an order. Choosing one from the same sample the procedure then reads is a selection problem inside a selection problem, and the criterion it reports becomes a minimum over a list.

Named by 25 essays across 11 fields — each of them below, with the objects they name alongside it.

One factor moves and the other does not. The two factors of the same average, each drawn against its own largest value so that they share an axis. The rate at which the fifteen candidates disagree about the tuning parameter rises from 28.6% at 4 values on the list to 43.3% at 8, a factor of 1.52. What a disagreement costs, given that there was one, is 0.00975 ± 0.00224 and 0.00848 ± 0.00113 at the same two points — 0.5 standard errors apart, and the paired comparison on the draws that disagree under both lists puts it the other way. The guess this field was written to test was that a longer list makes disagreements commoner and each one smaller. The first half is right and there is no second half.

A rate times a size

A sweep reported what it costs to let every candidate choose its own tuning parameter and found it flat across the list. It was reporting a product, and the two things multiplied together do not behave the same way at all.

apiece · Order selection The eighth was not a constant. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations between the candidates, over 800 draws apiece. The earlier field reports this flat at about an eighth across list length, on a table and a world it never varies. Vary how much the omitted coefficients are worth — one multiplier, with the table, the list, the law and the sample size all held — and it runs from 17.6% to 1.5%, a factor of 11.75. The world in which every candidate is true is the world in which the tuning list decides most; the world in which one candidate dominates is the world in which it decides nothing.

The eighth that was not a constant

How often a per-candidate tuning list changes which candidate wins is reported flat at about an eighth across list length. Vary how far apart the candidates are instead and it runs from 17.6% to 1.5%.

turnover · Order selection Three rules and a target none of them is aimed at. Which block length each rule picks, over 400 samples of 120 rows, for the tapered window. Two of the rules are points: a length written into a protocol is 8.00 on every draw and the rule of thumb is 4.00, because n to the one third does not read the data at all. The plug-in reads the sample's own persistence and lands at 14.36 with a standard deviation of 2.93. The length that would actually have been best on that draw averages 24.57 with a standard deviation of 16.23 and runs from 10 to 48 between its tenth and ninetieth percentiles. The target moves five times as much as the best estimate of it does, which is why no rule can be close to it and why the two that do not try are not merely worse — they are somewhere else.

The length nobody has

Every comparison of block windows in this collection is made at each window's own best block length. That length has a standard deviation of sixteen across draws and averages twenty-five. No rule is aimed at it.

feasible · Bootstrap Two instruments, two block lengths. The block length that would actually have been best on each draw, for each of the two error readings, averaged over 400 samples of 120 rows. For the rectangular window the implied long-run variance wants 18.92 and the 95% point wants 16.05; for the tapered window, 21.82 against 17.74. The quantile wants a shorter block under both windows — a ratio of 0.848 and 0.813. That is the mechanism the whole field turns on: a rule for choosing a block length is a way of guessing a target, and the two instruments do not have the same target. A rule tuned to one is systematically long for the other, and the two windows do not pay the same price for being long.

A length for each instrument

The block length that is best for an implied variance is 18.92; the one best for the 95% point of the same resamples is 16.05. A rule is a way of guessing a target, and there are two targets.

readout · Bootstrap Which window is better depends on who chose the block length. The margin between a rectangular block and a tapered one, on 400 samples of 120 rows, under four rules for choosing the block length. At the length that would actually have been best on each draw the taper is ahead by 2.12 points of a 35.8% error, at 22.1 paired standard errors; at a length estimated from the sample's own persistence it is ahead by 1.89. At the length this field's own figures use — eight — the rectangle is ahead by 2.04, and at the rule of thumb by 4.54. Every rule sees the same draws. What separates them is the length: the two rules that lose to the rectangle pick 4.00 and 8.00 where the best available is 24.57, and a tapered window at a quarter of the right length has thrown away most of what it was weighting.

An ordering that depends on the rule

The tapered block beats the rectangular one at the best available block length and at one estimated from the data. At a length written into a protocol, and at the rule of thumb, the rectangle wins — at every sample size measured.

feasible · Bootstrap What a longer block buys and what it costs. A trapezoidal block at 120 rows, with the error split into the two things it is made of. The bias falls with the block length, because a longer block attenuates less, and it flattens at 23.2% because the sample's own autocovariances are short whatever window is applied to them. The spread rises with it, because a longer block means fewer of them. Their sum in quadrature has a minimum at ℓ = 16, which is not where either of the two has one. The faint line is the rectangle's total error, for scale: it is above the trapezoid's from ℓ = 12 onwards.

Bias is not the whole of it

A window that reaches zero at its ends attenuates less and uses less of each block. The block length that minimises its bias is not the one that minimises its error, and comparing two windows at one length compares one of them mis-tuned.

crossing · Bootstrap What a disagreement costs, split on whether it decided anything. The regret from choosing the tuning parameter per candidate, on the draws where the candidates disagreed, split on whether the disagreement changed which candidate the table selects. Over 1200 draws at each list length: when the winner changes the regret is 0.02215, 0.03029, 0.03145; when it does not it is -0.00243, -0.00069, -0.00065 — negative, and small enough that it is inside two standard errors of nothing at every length. The whole of the cost lives in the first column, and the second column is not merely small but slightly the wrong sign: when the table's answer is unaffected, letting each candidate use its own window is a very slightly better rule than making them share one. So a disagreement about the tuning parameter is not a cost. A disagreement that changes the winner is.

The quarrel that changes the winner

A disagreement about the tuning parameter costs 0.031 when it changes which candidate the table selects and −0.0007 when it does not. The distance between the values disagreed about has nothing to do with it.

apiece · Order selection Two factors, opposite directions. The two factors the cost of a per-candidate tuning parameter is a product of, as the candidates are pulled apart, over 800 draws at each of 5 separations. How often the candidates disagree about the tuning parameter rises from 31.8% to 88.8%; the share of those disagreements that change which candidate the table selects falls from 51.6% to 1.7%. So the setting where the candidates quarrel most about the tuning parameter is the setting where the quarrel matters least, and a sweep that reads the rate and stops has read the factor pointing the wrong way.

Two factors pointing opposite ways

As the candidates on a table are pulled apart, they quarrel about the tuning parameter three times as often and the quarrel decides the winner thirty times less often. A sweep that reads the first factor has read the one pointing the wrong way.

turnover · Order selection The split decides the width. The width of the interval against the share of 200 observations spent on fitting rather than on calibrating, over 3000 draws. Spending more on the fit shrinks the residuals; spending more on calibration builds the interval at a less extreme order statistic. The two meet at 0.5, where the width is 4.0416 against 4.1603 at 0.1 and 4.3820 at 0.9. Full conformal, which spends the same 200 points on both jobs, is 3.9865 — so the whole cost of splitting is 1.38%.

What the split costs

Splitting a sample between fitting and calibrating looks like a trade against the guarantee, and it is not: coverage moves 0.63 points across nine splits and every reading sits on its own promise. The whole cost is 1.38% of width — and at sixty observations the width falls, rises and falls again.

conformal · Exchangeability Flat along a row, apart between them. The probability that a per-candidate tuning list changes the winner, at three list lengths on three candidate tables, over 800 draws in each of the nine cells. Along a row — the reading the earlier field takes — it moves by a factor of at most 1.21, so that field's invariant survives on every table. Down a column it moves by up to 1.98. The list length is the dial that does not move this number and the table is one that does, and the earlier field varied only the first.

A table and a list

A nested ladder of candidates differing by one coefficient was predicted to turn over more often at every list length. It turns over less at every one, and its list changes the winner half as often.

turnover · Order selection Cut the charge and the width follows it. The band width each charge picks, averaged over 400 draws of 120 rows under AR(1) at 0.8, with the standard deviation across draws beside it. Schwarz's charge — half a log n a lag, which is 2.39 here — picks 3.67. Akaike's picks 6.02. Charging the numbers the window actually leaves free, which is half the width, picks 10.12; charging what the optimism measures, 0.767 of that, picks 14.15. A charge and the width it buys are very nearly reciprocal, which is what a likelihood rising at a fixed rate a lag implies and is why the four answers span a factor of 3.86. The width that was actually best on the draw averages 13.90 and moves by 10.30 from draw to draw — three times as much as any rule's answer does.

A width that moves and an error that does not

Four charges give four widths a factor of four apart and four errors half a per cent apart. The derived charge wins, significantly, by a quarter of what was on offer — and none of the four is an estimate of anything.

dimension · Criterion The quantity that does not depend on the list. The probability that letting each candidate choose its own tuning parameter changes which candidate the table selects — the product of the two moving shares — against the length of the list, over 1200 draws apiece. It is 14.2%, 11.9%, 12.3%: a spread of 2.2% across a list length that moves the disagreement rate by a factor of 1.52. This is the invariant the whole field turns on. Everything downstream of the winner — the coefficients, the regret, whatever a reader is going to quote — is a function of whether the winner changed, and how often that happens is not something the list controls. A longer list changes how often the candidates quarrel and not how often the quarrel matters.

How often it matters

The disagreement rate rises by half across the list and the share of disagreements that decide anything falls by nearly the same factor. Their product — how often the tuning list changes which candidate wins — sits at an eighth and does not move.

apiece · Order selection The reversal is a property of the instrument. The margin between the two block windows under each of four rules, on three readings of the same resampled means, signed so that a positive bar is the tapered window winning. On the implied long-run variance the taper wins at the best available block length and at one estimated from the sample and loses at a length written into a protocol and at the rule of thumb — which is the reversal the earlier field's whole argument turns on, at 1.48 and 4.52 points. On the 95% point a test actually reads, the taper wins at all four, by 6.13 to 7.08 points. On the coverage the interval actually delivers, the taper wins at all four again, by 2.50 to 5.75 percentage points. Two of the four rules change sign between the first reading and the other two, and the two that change are exactly the two the earlier field's recommendation is about.

The reversal that was the instrument's

On an implied variance the rectangle wins at a protocol length and at the rule of thumb. On the 95% point a test reads, and on the coverage an interval delivers, the taper wins at all four rules.

readout · Bootstrap The argument is a third of the size of the thing it is inside. Three quantities on one scale, in points of the error in a block resample's implied long-run variance, at 120 rows. The gap between the two windows at the best available block length — the whole subject of the comparison this field inherited — is 2.12 points. What the best rule a practitioner could actually run gives up against that same best length is 7.26, a factor of 3.42. What the rule of thumb gives up is 26.01. So the ordering between windows is worth establishing and is not worth arguing about, and the sentence that follows from it is not use the taper but estimate the block length, because that is where the points are.

What choosing the length costs

The gap between two block windows at the best available length is 2.12 points. What the best rule a practitioner could run gives up against that same length is 7.26. The argument is a third of the size of the thing it is inside.

feasible · Bootstrap One window for the table, or one each. The regret of the same fifteen-candidate table under AR(1) at 0.8 over 150 draws, with the window attached three ways. Chosen once from the fullest candidate's residuals it gives up 0.02518. Chosen from each candidate's own residuals, with the covariance estimate still shared, it gives up 0.02799 — a paired cost of 0.00281 at 2.0 standard errors for the tuning parameter alone. Estimating the covariance per candidate as well costs 0.01087, so the objection already on record is about 3.9 times the size of the one that was not.

A window for every candidate

The window and the order a whitening needs are chosen once, from the fullest candidate, on an argument that was made about an estimated covariance. A tuning parameter is not a covariance, and the two cost different amounts.

together · Order selection The same dial, on a list that steps by one. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations, for both tuning parameters at a matched list length of 8, with errors from a first-order autoregression. The sieve order runs from 13.1% to 2.6%, a factor of 5.00; the whitening window, from 16.4% to 1.5%, a factor of 11.75. What is held is the number of options, the table, the law, the sample size and the seeds; what cannot be held is the size of a step, since an integer step and a geometric step are different amounts of change. The dial moves both, and it moves them by 2.35 times as much on one as on the other.

A step that is not a ratio

Run the separation sweep on a tuning list of integers rather than a geometric ladder and the two factors still point opposite ways. The invariant does not survive: along a row of integers the probability moves by 2.163 where along the geometric ladder it moves by 1.208.

turnover · Order selection A premium that shrinks and a payout that grows. The relative error of the trapezoid's implied variance with the block length read off the whole autocovariance sequence, minus its error with the length from the lag-one correlation alone, in points, at four sample sizes and 400 draws each. Under the first-order autoregression the plug-in's algebra was written for, reading the sequence costs 3.87 at 120, 2.37 at 240, 2.48 at 480, 1.15 at 960. Under the five-period moving average, which the plug-in reads as an autoregression at 0.8, it is 0.43 at 120, -1.72 at 240, -3.01 at 480, -3.77 at 960 — a tie at 120 rows and a gain that grows with the sample.

A length read off the whole sequence

The automatic block length reads one number off the sample — the lag-one correlation — and assumes the rest of the dependence is geometric. Reading the whole autocovariance sequence instead removes the assumption and puts the length where the law wants it. At a hundred and twenty rows it buys nothing, even where the assumption is wrong; it costs 3.87 points where the assumption is right, and by 960 rows that premium is 1.15 and the payout under a law the plug-in misreads is 3.77. Under long memory neither rule, nor the best length on each draw, gets within ninety per cent.

feasible · Bootstrap Every pair of sieve orders. Each cell is a tuning list of exactly two sieve orders, the row's and the column's, and its shade is how often fifteen candidate regressions choose different values from it, at a true null over 240 draws. Every pair holding order zero disagrees on no draw at all, because the criterion never picks zero under this law. Every adjacent pair of positive orders disagrees on between 17% and 25% of draws — 22.1% for one and two, 25.0% for eleven and twelve — and the rate falls as the two orders move apart, to 5.0% for one and twelve. The largest cell is 25.0%.

Two values the criterion cannot tell apart

Fifteen candidates disagree about a whitening's order on two draws in five when the list holds all thirteen orders — and on more than one in five when it holds only one and two. Every adjacent pair of positive orders disagrees about as often as that pair does, and every pair holding an order the criterion never picks disagrees on none. A list's disagreement is set by the pairs of values on it that the criterion cannot tell apart, not by how many values it has, and one candidate's own doubt about its pick predicts all fifteen candidates' disagreement with a rank correlation of 0.96.

lists · Order selection Which factor the sample settles. The field's decomposition at the full tuning list, by sample size, over 1,200 draws at each: how often fifteen candidates choose different values, what share of those disagreements change which candidate the table selects, and the product — how often the list changes the winner. The window: 120 rows 43.3%, 28.5%, 12.33%; 240 rows 17.5%, 14.8%, 2.58%; 480 rows 38.3%, 2.6%, 1.00%; 960 rows 11.6%, 2.2%, 0.25%. The order: 120 rows 48.9%, 21.6%, 10.58%; 240 rows 47.5%, 13.2%, 6.25%; 480 rows 46.1%, 5.1%, 2.33%.

What a longer sample settles

At a hundred and twenty rows, letting each candidate choose its own whitening changes which of fifteen candidates wins on about an eighth of draws, whatever the list, the table or the law. At four hundred and eighty rows it changes the winner on 1.0% of draws for a window and 2.3% for an order, and at nine hundred and sixty on a quarter of one per cent. The sample settles it through one factor only: how often the candidates disagree about the tuning value hardly moves, and the share of their disagreements that change the winner collapses.

apiece · Order selection What four length rules cover, 120 rows. The coverage of the t interval on the overlapping scale at 120 rows, when the block length is the shortest one predicted to reach 94.5% under each of four readings of the share, over 4,000 draws of each law. AR(1) at 0.7: geometric 96.2%, read to the chosen lag 93.2%, read to lag 6 93.7%, known 96.5%. a five-period moving average: geometric 96.5%, read to the chosen lag 93.5%, read to lag 6 94.5%, known 95.3%. a break, 0.95 then 0.65: geometric 95.1%, read to the chosen lag 93.0%, read to lag 6 92.5%, known 95.2%. the break turned over: geometric 94.8%, read to the chosen lag 92.5%, read to lag 6 92.2%, known 95.0%.

A share read off the sample

A block length chosen for an interval's coverage needs the share of the mean's variance the scale will find, and the published rule reads it from a first-order autoregression at the sample's lag-one value. Reading it from the sample's own autocovariances instead was supposed to survive series nobody had modelled. At 120 rows it covers 92.5% to 93.5% on all four laws — worse than the wrong shape on every one — because a centred series' autocovariances sum to exactly zero and the share they imply is too generous; read to lag forty it covers 84%. The assumed shape costs width, not coverage: 17% too wide where dependence stops at lag four, and nothing visible under a break, where the interval covers for a reason the share knows nothing about.

taper · Bootstrap The same dial, on a list that steps by one. The probability that a per-candidate tuning list changes which candidate the table selects, at each of 5 separations, for both tuning parameters at a matched list length of 8, with errors from a five-period moving average. The sieve order runs from 29.5% to 5.8%, a factor of 5.13; the whitening window, from 26.3% to 4.8%, a factor of 5.53. What is held is the number of options, the table, the law, the sample size and the seeds; what cannot be held is the size of a step, since an integer step and a geometric step are different amounts of change. The dial moves both, and it moves them by 1.08 times as much on one as on the other.

The same dial under a second law

On a first-order autoregression, separating the candidates moved the chance that a tuning list changes the winner 2.35 times as far on a whitening window as on a sieve order, and the difference was put down to the step: an integer step is worth a third of a geometric one. Rerun the whole sweep with errors from a five-period moving average and the difference is gone — the probability spans a factor of 5.53 on the window and 5.13 on the order, a ratio of 1.08 — and near the null both lists leave about five and a half candidates within reach, where the autoregression left 3.6 and 1.9. The two factors still point opposite ways under both laws. The step's size was a property of the law the lists were tuned on.

turnover · Order selection Power as the recentring's threshold moves. The recentred set test's power at a persistence of 0.65, where one forecaster genuinely beats the benchmark, over 300 draws of 100 origins, as its threshold runs from zero to three times Hansen's. With the eight windows alone: 30.3%, 28.3%, 26.0%, 26.0%, 26.0%, 26.0%. With sixteen stale candidates added: 30.3%, 28.3%, 24.3%, 21.0%, 19.3%, 17.7%. The studentised reality check, which sets nothing aside, has 26.0% and 17.7%.

The threshold the recentring reads

A recentred set test keeps a candidate at its own mean when the sample has placed it far enough below the benchmark, and how far is a threshold — Hansen's, about 1.75 on a hundred origins, chosen as a rate rather than tuned. Swept from zero to three times that, it is a dial with one side that costs almost nothing. At zero the test finds a genuinely better forecaster 30.3% of the time with or without sixteen stale candidates in the set; at Hansen's threshold it finds it 24.3% of the time with them, and at three times it 17.7%. Its size never passes 3.0%, even where every candidate sits exactly on the boundary.

ranking · Multiplicity Five length rules, read on two intervals. The coverage of a 95% interval for the mean at the block length each of five rules picks, on the percentile interval and the studentised one built from the same resamples, under a five-period moving average and a first-order autoregression at 120 and 480 rows, 2,000 draws each. moving average at 120: percentile 88.5%, 85.3%, 86.4%, 86.3%, 88.8%; studentised 92.5%, 87.9%, 94.9%, 92.6%, 95.0%. moving average at 480: percentile 90.5%, 90.5%, 90.4%, 91.6%, 91.9%; studentised 91.2%, 91.2%, 94.5%, 93.3%, 95.0%. autoregression at 120: percentile 84.7%, 80.2%, 85.8%, 85.4%, 85.9%; studentised 90.3%, 83.0%, 92.0%, 90.9%, 94.3%. autoregression at 480: percentile 87.0%, 87.0%, 90.0%, 90.5%, 90.9%; studentised 88.6%, 88.6%, 93.0%, 93.0%, 94.2% (in the order eight, from the protocol, n to the one third, the plug-in, the whole sequence, the best on the draw). The rules spread over at most 5.6% on the percentile interval; the interval moves the same rule by up to 8.5%.

A ranking that belongs to the interval

Five ways of choosing a block length, read on the coverage of the interval each length builds rather than on the variance it implies. On the percentile interval the variance's verdict mostly survives: under a five-period moving average the plug-in and the whole-sequence rule tie at 120 rows and the sequence rule leads at 480. Build a studentised interval from the same resamples and the lead reverses — the plug-in's longer blocks cover 94.5% against 93.3% — because an interval that widens for its own uncertainty is paid for a long block where a percentile interval is punished for one. Under long memory no length and no interval covers two times in three.

feasible · Bootstrap When a longer sample settles the tuning list. How often the tuning list changes which candidate the table selects, against the sample size, for the whitening window (solid) and the sieve order (dashed), on three tables of candidates, twelve hundred draws at each size. Every subset of four predictors: window 12.33% at 120, 2.58% at 240, 1.00% at 480, 0.25% at 960; order 10.58% at 120, 6.25% at 240, 2.33% at 480. The nested sequence of four: window 5.83% at 120, 1.25% at 240, 0.67% at 480, 0.25% at 960; order 5.83% at 120, 4.00% at 240, 2.25% at 480. Six subsets of two predictors each: window 11.75% at 120, 3.75% at 240, 5.75% at 480, 2.33% at 960; order 9.42% at 120, 6.67% at 240, 4.42% at 480. The nested table starts at half the full table's rate and reaches a small one a sample size sooner; the table whose candidates all have the same dimension is still changing its winner on one draw in forty-three at 960 rows.

The table decides when the sample settles it

Letting each candidate regression choose its own whitening changes which candidate wins on about an eighth of draws at 120 rows, and a longer sample settles it — for fifteen nested-and-crossed subsets, at 0.25% by 960 rows. A nested sequence of four candidates gets there a sample size sooner, from half the rate. Six candidates that all hold two predictors never get there: at 960 rows their tuning list still changes the winner on 2.3% of draws, nine times the full table's, because a disagreement among candidates of one size goes on deciding the winner as often at 960 rows as at 240.

apiece · Order selection What a quarrel between two orders costs. Every two-value list of autoregressive orders from one to twelve, read on the same 2400 draws of a first-order autoregression at a hundred and twenty rows: each cell is the average regret of choosing the order per candidate rather than once for the table, on the draws where the fifteen candidates disagree about it. The cost grows away from the diagonal and stays level along it. Next to the diagonal, a quarrel between neighbouring orders costs 0.00126 for one and two and 0.00088 for eleven and twelve; four orders apart it costs 0.00803 and 0.00802; eleven apart, 0.02436. Every list holding order zero is left off because the candidates never disagree about one.

A quarrel's cost is its width

When fifteen candidate regressions disagree about a whitening's autoregressive order, the disagreement costs in proportion to how far apart the two orders are — 0.0027 of regret for every order of distance, a line that explains 96% of the cost across sixty-six pairs — and not at all by which orders they are: a quarrel between eleven and twelve costs what one between one and two costs, though one of those holds the truth. Fitted on pairs, the line predicts every longer list's cost. The window's quarrels follow no such line.

lists · Order selection

Named alongside it

The objects these essays reach for when they reach for this one.

Monte CarloSelection effectModel selectionDependenceInformation criterionRegretLong-run varianceBandwidth selectionBlock bootstrapBlock lengthTaperingWhitening

All concepts