Every essay — page 13
Searching among fitted models
The set field compares forecasts nobody estimated. A specification search compares a benchmark with a table of variants that all contain it, and two things change at once: every variant is behind before the search begins, by an amount with a closed form in the shape of the table, and the reference distribution for the winner can no longer be resampled from the data — it has to be generated from a model. The repair the nested case asks for, applied row by row, takes a table from its nominal level to a quarter.
The weight that is a vector
Two forecasts have a best combination and one number describes it. Eight have a best combination too, and the vector describing it puts nothing at all on the forecast with the smallest mean squared error.
When every null is true
A reality check assumes that every candidate in the set is exactly as good as the benchmark, which is a configuration nobody's data is ever in. Test a combination against its own parts and that configuration is not assumed — it is what the arithmetic makes true.
One test for eight nulls
A combination of eight forecasts can be tested against its parts one comparison at a time or all at once. All at once is a Wald test on the vector of eight encompassing means — which has seven degrees of freedom, not eight, because the means sum to zero on every origin when weighted by the combination. On sixty origins it rejects a true null 26.6% of the time, Hotelling's correction brings that to 15.1%, and handed its own true critical value it is less powerful than Bonferroni over the largest of the eight against both wrong combinations measured: 12% against 18%, and 47% against 96%.
What the procedure may not read
Two restrictions that were cheap until now. A design for a non-linear model may not read the parameter values, and where the model has two of them the guess is a point in a rectangle: the weights that were exactly 1/√2 become a function, and a design robust in one coordinate turns out to guarantee no more than one built at a single point. A stopping rule may not read the mean it will report — and there is an exact way to obey that, at a price in the width of the interval rather than in the number of observations.
The guess with two numbers in it
Every optimal design for a non-linear model is optimal at a guess. Where the model has one parameter that moves the settings, that guess is a number and everything about it comes out in closed form; where it has two, three constants become functions and one of them becomes zero.
The worst case in two directions
A design that protects a range of one parameter is robust. Protect the range of one parameter while holding the other at a guess and the design is still robust, still has a guarantee, and guarantees no more than a design that protects nothing at all.
The rule that cannot see the mean
A sequential rule stops when its own estimate of the spread is small, which is more often on the samples whose spread came out low — so the interval afterwards is short. There is a way to keep updating the estimate and stop being able to see the mean at all.
What the blindfold costs
The exactly-covering rule pays for it in the width of the interval, and the block size is a dial between two costs that run in opposite directions. And on an interval whose width was fixed in advance, the same repair buys nothing at all.
A variance that follows the response
Every design for the two-rate compartmental model assumed the same noise at every sampling time. Let the noise grow with the response, as a combined additive and proportional error does, and the design for the slow rate runs to the tail of the curve, where the response and so the noise are smallest: at three additive standard deviations of proportional error at the peak, the constant-noise design is worth two thirds of the right one. The earlier essay expected a weight function to break the model's scale invariance. A variance that follows the response leaves it exact to six decimals; only one that follows the clock breaks it. And tailoring the design to the noise makes it more fragile to the rates: its worst case over the rectangle falls from 9.2% to 3.3%.
The shape the covariate enters by
Every balancing rule in the two fields before this one optimises one over the variance of the treatment estimate in a model where the covariate enters linearly — which is not an assumption about the analysis, it is what the criterion is. Balancing the mean of a covariate removes exactly 2/π of the imbalance in a median split of it, a quarter of a threshold in the tail, and nothing at all from a quadratic. The ranking of the rules reverses between shapes, and against one of them every rule here is worse than a coin.
Balanced on the wrong function
A rule that reads a covariate's numbers halves the variance of the treatment estimate, if the covariate enters the outcome as a straight line. If it enters as a threshold the rule is worth a fifth of that, and if it enters as a curve every rule here is worse than a coin.
A threshold in the tail
How much of a threshold's imbalance a balanced covariate removes is a correlation, and the correlation is a closed form. At the median it is exactly 2/π — the same 2/π a median split throws away — and two standard deviations out it is an eighth.
Three functions of one number
A rule that balances the covariate is exposed to every shape the outcome might have. A rule that balances three functions of it costs two points of variance against the shape the first was built for and takes the worst case from a coin's to about half of it.
The analysis and the shape
An unadjusted analysis after a rule that read the covariate is too cautious — by a third against a linear outcome, by nothing at all against a quadratic. And an adjustment for the wrong function recovers almost none of the precision the right one would.
How many functions a trial can hold
A rule that reads more functions of one covariate protects more shapes, and every function it reads costs the shapes it already held: the imbalance it leaves in a linear outcome falls like one over the trial size, with a constant that runs from about 3 for one function to about 16 for six. At fifty units that cost eats the protection, and the worst case over four shapes sits between 37% and 42% of a coin's from two functions to six. At four hundred it falls from 42% to 6%. By six functions the worst shape, at every size, is one the rule is reading.
The list it was chosen on
Six functions of one covariate looked like 720 designs, one per order of addition. The rule reads a span and not a sequence, so they are 63, and at a hundred units the best four leave 12.1% of a coin's imbalance against the designer's first four at 31.5%. Those best four contain no cube and are, function for function, the four shapes the worst case was scored on. Read on six shapes it was not told about, the same design leaves the cube 77.0% of a coin's imbalance, and the designer's four, which lost by a factor of two and a half, leave it 28.1%.
A search with no fixed point
The search field compares a benchmark with variants that all contain it. Take the fixed point away — let the benchmark be one candidate among sixteen, selected by the same data as its rivals — and the closed form for what every candidate is behind by survives intact, because it was never about nesting but about counting parameters. Three readings of one true null give 2.5%, 8.0% and 76.8%; Bonferroni is three times too strong on one table and not strong enough on another; and searching the benchmark makes the table harder to reject with rather than easier.
When the benchmark is a candidate
A specification search with a benchmark nailed down is the case with a closed form. Take the nail out — let the model that would have been reported be one of sixteen, chosen by the same data as its rivals — and the same true null is read three ways, at 2.0%, 7.8% and 76.2%.
The displacement is a parameter count
A nested variant is behind its benchmark out of sample before anything is searched for. The closed form for how far turns out to have nothing about nesting in it — only two integers and a window length — and it prices a table where no candidate contains any other.
Residuals that keep their own variance
A reference distribution for a search has to be generated from a fitted model, and the generator draws residuals. Four ways of drawing them keep four different things — and the one this site has reached for three times repairs nothing at all here.
The corner the test is calibrated at
"No candidate is better than the benchmark" is not a null but a face of a region, and a reality check is calibrated at one corner of it. Fill the table with candidates that are hopeless rather than equal and the test finds a genuine improvement 0.0% of the time.
The displacement on a heavy tail
A correct candidate with one more parameter is behind out of sample by a closed-form amount, and the closed form rests on one Gaussian fact: the expected leverage of a fresh row is q/(R − q − 1). On predictors that are not Gaussian it becomes whatever the leverage is, and two routes that share no arithmetic agree on what that is. With a finite fourth moment it barely moves — 1.5% more per parameter at eight degrees of freedom, 4.5% at five, 7.3% on an exponential. At three degrees of freedom it is 28% more, and the estimate of it will not settle: four batches of twenty thousand draws disagree by more than a quarter.
An excess and its window
The parameter-count displacement is a Gaussian fact, and on heavy-tailed predictors it runs a few per cent over. Swept across windows from 20 to 640 rows, the excess is about the predictors' excess kurtosis divided by the window: a law with a fourth moment of nine on 80 rows is 7.5% over, and on 640 rows under one. At three degrees of freedom it falls only as the root of the window, from 44% to 7.6%. And the instability once blamed on that law was not there: with the fresh row integrated out, the batches agree to a point and a half, and the mean exists.
Choosing what the rule reads
A balancing rule cannot tell one basis from another with the same span, so choosing what to hand it is choosing a subspace, and everything about it is exact geometry: what a rule removes of a shape is that shape's squared multiple correlation on the span. The maximin basis over a list of shapes is then a finite problem with an exact answer — and over a whole subspace the guarantee is exactly zero rather than small, which is why the list cannot be avoided. Randomising which functions the rule reads is worth twice the best fixed choice.
A basis is a subspace
A balancing rule cannot tell one basis from another with the same span, so choosing what to hand it is choosing a subspace — and then what it removes of any outcome shape is a projection, computable exactly, with no trial anywhere in it.
Which shapes are worth protecting
Choosing a basis by its worst case is a finite problem with an exact answer. The answer has no tie in it, which a maximin optimum is supposed to have — and the tie comes back, along with twice the guarantee, when the basis is drawn rather than chosen.
Where the guarantee is exactly zero
An experimenter who declines to name the shapes, and asks instead to be protected against anything in a class, is asking for a number that is not small but zero. Bounding the class is unavoidable, and the two ways of doing it choose different bases.
When the constraints run out
Every function added to a basis is a constraint the assignment has to satisfy with the same units. At sixteen units and a stated tolerance the admissible assignments run 3,874, then 1,006, then 314, then none — and the count is exact, because the assignment space is finite.