a-cut-at-a-correlation

Sheppard's arcsine, by two routes

Corr(sign X, sign Y) as the covariates' correlation runs from zero to one, drawn twice. One route is a sixty-four-node quadrature of the orthant probability over the correlation — the general construction, which works at any pair of cut points; the other is (2/π) arcsin ρ, which is elementary and works only at the median. They agree to 3.3e-16 at every one of 81 correlations, which is what licenses the quadrature everywhere else. The curve is above the diagonal at small ρ and below it at large: two signs agree with probability ½ + arcsin(ρ)/π, so a correlation of 0.5 gives exactly ⅓ and a correlation of 0.8 gives 0.5903.

A cut point, at a correlationwide15 views

What else it draws

The same object, drawn to answer the other questions the essays put to it.

The error in ⟨c(X), c(Y)⟩ from truncating Mehler's sum at J orders, for a median split at four correlations, against the exact answer. The truncation leaves the sum over m > J of a_m b_m ρ^m, which Cauchy–Schwarz bounds by ρᴶ⁺¹√(tail·tail) — so the coefficients' tail sets the constant and the correlation sets the rate. At ρ = 0.2 sixteen orders are exact to the last bit a double carries; at 0.5 the error is 7.1e-8; at 0.95 it is still 1.5e-2 and thirty-two orders are needed. The caveat this replaces was the ρ = 1 statement, carried to correlations a trial actually has.

What a balancing rule handed both main effects removes of the pure interaction between them, as the covariates become dependent. For median splits it is exactly zero at every correlation, because sign(x)² = 1: the interaction sign(X)sign(Y) is orthogonal to sign(X) and to sign(Y) whatever ρ is. For the product of the raw covariates it is 4ρ²/(1+ρ²)² — 64.00% by ρ = 0.5, rising to all of it at perfect correlation. A cut away from the median sits between them and is not small: 23.01% at a cut of one. The zero is not a fact about interactions. It is a fact about a dictionary whose functions square to a constant, which a polynomial one does not.

For a rate of 0.05, a companion used linearly removes at most 22.4% of the variance however well it correlates. At a correlation of 0.9 it removes 18.1%, the companion's own rate at the same cut 38.2%, and the best function of the companion 48.1%; at 0.99, 21.9%, 77.0% and 82.8%. The companion's rate overtakes the linear companion at a correlation of 0.64.

The ceiling is φ(c)²/(p(1 − p)) at the cut c whose tail is p: 63.7% for a median rate, 34.2% at 10%, 22.4% at 5%, 7.2% at 1% and 1.1% at 0.1%. It is reached only by a companion perfectly correlated with the statistic.

The t statistic of the data, which rejects its true mean more often than 2.5% in the lower tail, against the t statistic of the logarithms, which is exactly Student's t. Of 1,500 samples, 27 are rejected by both, 57 by the data's test alone and 1 by the logarithms' alone.

The rejection rate being simulated is 6.14%. Over 200 simulations of 4,000 samples, the logarithms' t statistic used linearly was worth 1.43 times the draws (its correlation predicts 1.31); the logarithms' own rejection at 2.5%, 1.60 (predicted 1.55); at 6.5%, a level matched to the target's from a pilot run, 2.32 (predicted 2.23).

The best any function of the companion can do is 48.1%; its own rate at the same cut, 38.2%. Strata spent across the band where the conditional probability changes reach 95% of the best at 8 strata; equal-probability strata need 32. At eight strata the two remove 46.8% and 29.9%.

The conditional probability climbs from 1% to 99% between companion values 0.70 and 2.95. Equal-probability cuts put 6 of their 7 cuts below that stretch where the probability is already flat; the band scheme puts all 7 inside or at its edges.

Over one hundred simulations of 4,000 samples, strata across the lower band were worth at most 2.99 times the draws, at 32 strata, and 2.14 at 256, where 69 strata a run were empty; strata at equal probabilities were worth at most 3.03, at 32.

To first order, bands of the logarithms' t statistic are worth at most 2.77 times the draws; a 32 × 32 grid in the pair 4.68; bands of a score fitted to the pair on a thousand samples 4.64 at sixteen bands and 4.69 at sixty-four. Counted over runs of 4,000 samples, the grid of 1,024 cells was worth 1.97, with 94 cells a run left empty.

Samples of twenty from a lognormal with σ = 0.5, shaded by the share whose t test on the data rejected below its 2.5% point: darkest at three quarters or more, then a quarter, then 2%. Along the ray on which the logarithms' own t statistic is −2 the share runs from 43.3% to 100.0%, rising as the spread falls: the t statistic of the logarithms cannot tell those samples apart, and the pair can.

First-order worth of sixteen score bands with exact weights: 3.88 from a pilot of 100, 4.48 from a pilot of 250, 4.64 from a pilot of 1,000, 4.85 from a pilot of 4,000, 4.85 from a pilot of 16,000. The best the logarithms' t statistic does is 2.77; a 32 × 32 grid, filled, 4.68.

With exact weights the bands are worth 4.64 times a run of four thousand. With weights estimated from a pilot of 1,000 they are worth 0.30, of 4,000 they are worth 1.00, of 16,000 they are worth 2.43, of 64,000 they are worth 3.78, of 256,000 they are worth 4.39, of 1,024,000 they are worth 4.57; the same pilots added to the run's draws are worth 1.25, 2.00, 5.00, 17, 65, 257.

What sixteen bands of the logarithms' t statistic are worth to a simulation of the lognormal t test's lower-tail size, in multiples of the draws, against the size of the run, with the band weights from Student's t exactly and from three approximations to it. Exact weights are worth 2.709 at every size. Fisher's one-term expansion is worth 2.708 at four thousand draws and 2.523 at a million, and breaks even with counting at 23.7 million draws. The variance-matched normal is worth 1.849 at four thousand and breaks even at 14,698. The standard normal is worth 0.762 at a thousand and breaks even at 669: a bias that does not shrink is multiplied by every draw added.

Where it is used

7 essays draw this figure, each at the numbers its own argument is about, so the same picture answers 7 different questions.

All 80 figures

FieldsThreadsSeriesConceptsAll essaysSearch