Every essay — page 7
The spread, and its own uncertainty
Partial pooling estimates the population spread from the group means and then uses it as though it were known. It is not known: eight groups leave τ anywhere in a range that spans a factor of several, and on a third of eight-group datasets the usual estimate of it is exactly zero. Carrying that uncertainty rather than dropping it is the difference between an interval that covers 95% and one that covers 79%.
What the plug-in forgets
The shrinkage weight needs a population spread, and the population spread has to be estimated from eight numbers. Empirical Bayes estimates it, substitutes it, and proceeds as though it were known — and the interval that comes out covers 79% rather than the 95% it claims.
A prior on the spread
Integrating over the population spread means putting a prior on it, which sounds like the objection rather than the repair. The prior's effect is measurable, it is invisible where the groups are clearly different, and the reflex choice for a scale parameter turns out not to have a posterior at all.
When the spread estimates to zero
The usual estimate of a population spread is a difference of two positive quantities, clamped at zero. On a third of eight-group datasets with a real spread in them the difference comes out negative, the estimate is exactly zero, and every group is pooled completely on data that said no such thing.
The interval that integrates
A credible interval for one group in a hierarchy has to average over every value the population spread might take. That averaging is what makes it cover — 95.2% against the plug-in's 78.8% — and it costs 31% more width, a heavier tail, and a mixture rather than a normal.
A control borrowed from the last trial
A trial of fifty patients an arm can borrow its control group from an earlier trial of two hundred by treating the two control means as draws from one population with a spread between trials. With two trials there is one difference to learn that spread from, so the prior on it decides how much is borrowed — and its tail decides whether the borrowing stops when the two trials disagree. A half-normal prior of scale 0.05 buys 17.4 points of power and, at a drift of 1.5 standard deviations, still declares a treatment with no effect a success 31.8% of the time. A half-Cauchy of the same scale buys 13.6 points, peaks at 11.0% and falls back to 3.4%.
A history that agrees with itself
Borrowing a control from sixteen earlier trials that agree with each other should be safer than borrowing from one, and it is the opposite. Sixteen agreeing trials estimate the spread between trials as small, and a small spread estimated confidently is a licence to pool: under a half-Cauchy prior that let go of a single disagreeing trial, a current control 0.6 standard deviations from a sixteen-trial history turns a treatment with no effect into a success 80.7% of the time, against 10.5% at worst with one earlier trial. Moving a fifth of the prior onto a vague component brings the worst case to 16.9% and keeps power at 91.0%, against 70.5% without borrowing.
A history that has been moving
Earlier control arms borrowed as exchangeable draws are pooled towards the history's middle, and when the control mean has been rising the current trial sits at its end. A drift of 0.02 standard deviations a trial — a step between neighbouring trials of under a third of either one's standard error — turns a treatment with no effect into a success 9.9% of the time from sixteen earlier trials, and a drift of 0.05 overstates the effect by 0.130 standard deviations from eight. A line in time removes the bias exactly, at every slope, and pays for it in borrowing: from eight earlier trials it borrows 175 control patients where the exchangeable model borrows 439.
A history that stepped
When sixteen earlier control arms stepped once — the first eight a fifth of a standard deviation below the last — borrowing them as exchangeable flatters a treatment with no effect by 0.063 and stops borrowing, and fitting a line through them penalises it by 0.052. A model with a change point is unbiased at every step size, borrows more the larger the step, 512 patients at a fifth, and puts two thirds of its belief on the right place for the change. Against a steady drift it is the one that fails, so each of the three is safe only against the shape of history it assumes.
Hierarchy past one number
The same weighted average, on quantities that are not means. A slope, where how much a group borrows is decided by the arrangement of its x values rather than by how many it has. A proportion, where the pooling has to happen on a scale the data does not live on. A second level of grouping, whose arithmetic turns out to be the effective sample size of the time-series field arrived at from the other direction.
The slope that borrows
Pooling a mean makes it look as though how much a group borrows depends on how much data it has. Pool a slope instead and the illusion breaks — ten groups with ten observations each can borrow anything from 28% to 91%, decided entirely by where those ten observations were placed.
Pooling a proportion
A proportion cannot be shrunk on its own scale — an estimate would leave the interval, and how much information a count carries depends on where it sits. Move to log-odds and the approximation works, at the price of a group that saw nothing having no estimate at all until the correction supplies one.
Borrowing towards a line
A group shrunk towards the average of all groups is being compared with groups it has nothing in common with. Fit a group-level predictor and it is shrunk towards what the predictor says a group like it should be — which halves the spread left to borrow against and takes a quarter off the squared error.
Two levels at once
A third level of grouping adds no new arithmetic and produces one number — the design effect — that decides how many independent observations a clustered study is worth. It is the same quantity the time-series field computes for autocorrelated data, arrived at from a completely different picture.
Two groupings that cross
Pupils belong to a school and to a neighbourhood, and neither is nested in the other. There is then no design effect: the overall mean is worth 8.5 independent observations out of 240, a row difference 11.1 and a column difference 26.6, and which grouping matters depends on the question rather than on the study.
A level with two units
A variance estimated from two units is a scaled chi-square on one degree of freedom. Its interquartile range spans a factor of thirteen, its ten-to-ninety range a factor of a hundred and seventy-one, and it comes out exactly zero on 26.7% of studies — so the design effect it decides runs from 1.00 to 7.01 against a truth of 4.69.
What a two-unit study should report
The fixed-effect interval covers the mean of the two sites in hand 96.37% of the time and the population mean 54.77%. The random-effects interval covers the population mean 94.96% — exactly its level, from one degree of freedom — and is 11.6 times wider.
A prior on the spread between two sites
Integrate the between-site spread of a two-site study under a half-normal prior of scale 1 and the interval for the overall mean is 3.01 wide and covers 94.63% when the true spread is 1 — the supplied interval again, under another name. At a true spread of 3 it covers 68.38%. Every prior tried covers 95% averaged over its own draws and none covers it at every spread; each holds up to about its own scale, because two sites move the posterior median of the spread by less than a third of what the truth moves.
Series that move together
Two random walks regressed on each other are called related three times in four, which is why the time-series field ends in a warning. The exception it names and does not measure is here: when the pair is genuinely tied, the fitted relation converges at rate 1/n rather than the usual 1/√n, and the test that separates the two cases cannot be read against any table that exists.
The regression that is not spurious
Two random walks regressed on each other are called significantly related three times in four, so the time-series field ends in a warning. The exception it names and does not measure is here — and when the pair is genuinely tied, the fitted relation converges at rate 1/n rather than the usual 1/√n.
The test with no table
The statistic that separates a real long-run relation from a spurious one is computed as a t and is not a t. At two hundred observations its 5% point is −3.38 where the t table says −1.65, and reading it against the table calls two unrelated random walks cointegrated 70.5% of the time.
The model that corrects its error
A cointegrated pair can always be written as a mechanism — today's change in y depends on yesterday's disagreement between y and its long-run relation with x. The coefficient of that disagreement is recovered from data that never saw it — and on unrelated series the same fit produces one a t table would call real 41% of the time.
The cost of differencing a pair
Differencing two cointegrated series makes every standard error honest and throws away the one thing known about where they are going. The error-correction model forecasts better by exactly what a closed form says — and at four hundred observations it is better on four series in five and worse on average.
A relation supplied in advance
When theory supplies the long-run coefficient, the gap between two series can be tested directly and the search the Engle–Granger test pays for disappears: its 5% point moves from −3.38 to −2.89, and at two hundred observations the slowest return found four times in five goes from a half-life of 5.05 steps to 6.88. The price is in what a rejection means. With a gap that halves in three steps, a coefficient a fifth too large is confirmed 99.1% of the time and one half again too large 81.9%, though every such gap contains a random walk — and at a thousand observations the second is confirmed more often, not less.
A slow return across many pairs
One pair of two hundred observations is found tied four times in five only if its gap halves within 5.1 steps. Sixteen independent pairs sharing the same half-life, tested by the average of their Engle–Granger statistics, are found four times in five up to a half-life of 15.0 — and at a half-life of twelve, 99.8% of the time against one pair's 20.0%. What the widened test rejects is a statement about the panel: with half the pairs tied it rejects 55.4% of the time, with a quarter 21.2%, and a pair-by-pair reading of a panel in which all sixteen are tied finds 2.65 of them, against 0.78 when none are.
A shock every pair shares
Sixteen pairs of series whose shocks are half common are worth 5.37 independent pairs to the panel's average statistic, and sixteen that share nine tenths are worth 1.44. Read against the 5% point computed for independent pairs, the pooled test then calls a panel with no relation tied 15.5% and 31.6% of the time; read against its own point, it finds a relation shared by every pair 22.3% of the time at nine tenths common, against one pair's 20.0%. Removing each date's cross-sectional average gives back the level and all sixteen pairs when every pair feels the shock equally, and leaves 13.3% false alarms when the loadings run from a half to one and a half.
A factor estimated from the panel
When sixteen pairs feel a common shock with loadings from a half to one and a half, removing the cross-sectional average leaves the pooled test rejecting an unrelated panel 13.3% of the time. Estimating the factor and each pair's loading instead brings that to 8.9% — and no further, because each loading comes from the same two hundred dates however many pairs there are — and it costs power against the average at the right critical value, 73.8% against 82.5%. Removing the shocks as observed restores 4.5% exactly and finds a shared relation 7.5% of the time: a series tied to its partner carries the partner's factor in its level, not its own shock.