Field

A cut point, at a correlation

The geometry of a balancing dictionary over two dependent covariates is closed for polynomials and was taken to be asymptotic for cut points, because a threshold's Hermite coefficients never terminate. Conditioning on the second variable closes it exactly: every mixed inner product is ρ^j times a one-variable answer and the only two-dimensional object left is an orthant probability, which at the median is (2/π) arcsin ρ. The truncation that was feared falls geometrically in the correlation rather than algebraically in the order — and the interaction guarantee a correlation destroys for powers survives it exactly for median splits, because a two-valued function squares to a constant.

The expansion that never terminates. The Hermite coefficients of a median split, in magnitude, against the reference j to the power −3/4, anchored at the first one. Every even order is exactly zero because sign is an odd function, and every odd order is not, so no truncation is exact — where a polynomial of degree d is exact at any order past d. Summed, the tail past J falls like 1/√J: sixty orders still leave 6.6% of the variance outside. That statement is what made a cut dictionary's geometry unavailable in closed form, and it is a statement about the function against itself. What it is not is the accuracy of an inner product between two correlated variables, where every term past J carries a factor of ρ^m as well.

A cut is not a polynomial, and it does not have to be

A threshold's expansion never terminates, which is why a balancing dictionary's geometry was closed for powers and taken to draws for cut points. Conditioning on the second variable closes it for both.

Sheppard's arcsine, by two routes. Corr(sign X, sign Y) as the covariates' correlation runs from zero to one, drawn twice. One route is a sixty-four-node quadrature of the orthant probability over the correlation — the general construction, which works at any pair of cut points; the other is (2/π) arcsin ρ, which is elementary and works only at the median. They agree to 3.3e-16 at every one of 81 correlations, which is what licenses the quadrature everywhere else. The curve is above the diagonal at small ρ and below it at large: two signs agree with probability ½ + arcsin(ρ)/π, so a correlation of 0.5 gives exactly ⅓ and a correlation of 0.8 gives 0.5903.

The arcsine that closes it, and the error that was overstated

Two median splits of a correlated pair agree with probability ½ + arcsin(ρ)/π, exactly. And the truncation the field was avoiding falls geometrically in the correlation, not algebraically in the order.

The guarantee that survives a correlation, and the one that does not. What a balancing rule handed both main effects removes of the pure interaction between them, as the covariates become dependent. For median splits it is exactly zero at every correlation, because sign(x)² = 1: the interaction sign(X)sign(Y) is orthogonal to sign(X) and to sign(Y) whatever ρ is. For the product of the raw covariates it is 4ρ²/(1+ρ²)² — 64.00% by ρ = 0.5, rising to all of it at perfect correlation. A cut away from the median sits between them and is not small: 23.01% at a cut of one. The zero is not a fact about interactions. It is a fact about a dictionary whose functions square to a constant, which a polynomial one does not.

The zero that survives a cut

A rule holding both main effects removes half of a pure interaction between correlated powers and exactly none between correlated median splits. The guarantee that a correlation destroyed was never about interactions.

What a companion with a known expectation can remove from the variance of a simulated 5% rate. For a rate of 0.05, a companion used linearly removes at most 22.4% of the variance however well it correlates. At a correlation of 0.9 it removes 18.1%, the companion's own rate at the same cut 38.2%, and the best function of the companion 48.1%; at 0.99, 21.9%, 77.0% and 82.8%. The companion's rate overtakes the linear companion at a correlation of 0.64.

A rate needs a rate beside it

A simulated rejection rate is an average of indicators, and a companion with an exactly known expectation can remove at most φ(c)²/p(1 − p) of its variance when it is used as a straight line — 22.4% for a 5% rate, 7.2% for a 1% rate — however closely it tracks the statistic. The ceiling is the share of a normal's information a cut keeps. Used as its own rate at the same cut, the same companion removes 38.2% at a correlation of 0.9 and 77.0% at 0.99. On the t test's size for lognormal data it was worth 1.43 times the draws used linearly and 2.32 times used as a rate at a matched level.

A companion cut into many

A simulated rate can be estimated within bands of a companion whose distribution is exact, and the bands weighted back by their exact probabilities. One band boundary at the target's threshold is the companion's own rate; infinitely many are the best any function of the companion can do, which for a 5% rate at a correlation of 0.9 removes 48.1% of the variance. Eight bands placed where the target's conditional probability is changing remove 46.8%; eight at equal probabilities remove 29.9%, and need thirty-two to catch up. On the lognormal t test's size, thirty-two bands of the logarithms' t statistic were worth 2.99 times the draws and two hundred and fifty-six, 2.14.

What two exact companions are worth to a simulated rate: the lognormal t test's size post-stratified three ways, by the number of cells. To first order, bands of the logarithms' t statistic are worth at most 2.77 times the draws; a 32 × 32 grid in the pair 4.68; bands of a score fitted to the pair on a thousand samples 4.64 at sixteen bands and 4.69 at sixty-four. Counted over runs of 4,000 samples, the grid of 1,024 cells was worth 1.97, with 94 cells a run left empty.

A companion with two coordinates

The logarithms' t statistic, cut into bands, is worth at most 2.77 times the draws to a simulation of the lognormal t test's size. It is one function of two exactly known statistics, the logarithms' mean and spread, and the pair is worth 4.64 when a score fitted to it on a pilot of a thousand samples is cut into sixteen bands whose probabilities are integrated exactly. A grid of cells in the pair needs a thousand cells to reach the same and, counted on runs of four thousand samples, is worth 1.97. Band weights estimated from a pilot instead of computed are worth less than the pilot's draws would have been had they simply been counted — exactly nothing at a pilot the size of the run.

Approximate weights against the size of the run. What sixteen bands of the logarithms' t statistic are worth to a simulation of the lognormal t test's lower-tail size, in multiples of the draws, against the size of the run, with the band weights from Student's t exactly and from three approximations to it. Exact weights are worth 2.709 at every size. Fisher's one-term expansion is worth 2.708 at four thousand draws and 2.523 at a million, and breaks even with counting at 23.7 million draws. The variance-matched normal is worth 1.849 at four thousand and breaks even at 14,698. The standard normal is worth 0.762 at a thousand and breaks even at 669: a bias that does not shrink is multiplied by every draw added.

Weights from an approximate law

A companion whose law is only approximately known can still be cut into bands, with each band's weight computed from the approximation. The weight's error is a bias that does not shrink, and the run multiplies it: the stratified rate is worth p(1 − p)/(v + Nb²) times the draws. Reading Student's t on nineteen degrees of freedom as normal breaks even at 669 draws; the normal with the t's variance at about fifteen thousand; Fisher's one-term expansion at twenty-four million. What decides the bias is how the weight errors line up with the band rates, not how large they are — Fisher's is wrong by sixty per cent in one band and biases the rate by six hundredths of one.

All essays