A cut walked into the tail

Which tail the cut sits in priced a balancing rule that holds a threshold on each of two covariates, at one threshold. The cut sat at 1 on a standard normal scale — the 84th percentile — and five copulas at one rank correlation put the rule’s leak anywhere from 5.33% to 33.36%, the two ends being the same copula turned over. That essay named the obvious sweep and declined it: move the cut from the median outwards and watch the leak grow from the median split’s exact zero. It would need a second axis in every figure, and the pair of Claytons stood in for it.

The sweep turns out to say something the single cut could not. The leak does grow from zero, on every copula. What it does next divides the five into two kinds, and the division is a property of the dependence that a rank correlation does not record and that the earlier field never needed to name: whether the covariates are still related out in the tail, where a threshold a clinician would actually write down tends to sit.

Five curves from one zero #

Two standard normal covariates, five copulas each matched at a Spearman rank correlation of 0.4, and a rule balancing a threshold on each. The cut is placed at seventeen percentiles from the 1st to the 99th, and at each one the quantity is the same as before: the share of the interaction between the two thresholds — the centred product of the two indicators — that balancing the two indicators removes.

Every copula reads exactly zero at the median, which is the median split’s arithmetic: a cut at the median is a median split. From there the five go five ways.

The Gaussian rises to 19.49% near the 95th percentile and is back down to 13.44% at the 99th. The Frank rises less and turns sooner: 13.22% near the 85th and 2.06% by the 99th. The t on four degrees of freedom is still close to its highest, 37.45%, at the 99th. The Clayton climbs to 56.37% at the 1st percentile, in the tail its dependence lives in, and in the other tail never passes 5.39%. Its reflection is the same curve turned round.

So the 84th percentile the earlier essay used is not a typical point on any of these curves. On the Gaussian and the t it is partway up a slope; on the Frank it is at the summit; on the Clayton pair it is near the bottom of one and two thirds of the way up the other. A comparison of copulas at one cut is a comparison of where five different curves happen to be at that cut, and the curves are what a protocol actually chooses among when it chooses where to put its threshold.

Two probabilities are the whole regression #

Once both covariates are cut, there is very little left for a copula to do. Each indicator is on with probability $p$ and both are on together with probability $p_{11}$, and those two numbers determine every moment the regression uses.

Write $a = A - p$ and $b = B - p$ for the centred indicators. Because $A$ is zero or one, $a^2 = p^2 + A(1 - 2p)$, so the covariance of the product $ab$ with either indicator is $(1 - 2p)\,c$ with $c = p_{11} - p^2$, the covariance of the two indicators. The rule removes

$$\frac{2\,(1-2p)^2\,c^2}{\bigl(p(1-p) + c\bigr)\,\bigl(p^4 + 2p^3(1-2p) + p_{11}(1-2p)^2 - c^2\bigr)}$$

of the product’s variance, and nothing else in the joint law enters.

It is the same geometry the dictionary field computes by a Hermite series for a Gaussian pair, with everything about the Gaussian removed. Three things are in that expression before any copula is named. The factor $(1 - 2p)$ is zero at the median, which is the arithmetic zero again, arriving from a different direction. If the indicators are independent then $c = 0$ and the rule removes nothing at any cut, because the product of two independent centred indicators is uncorrelated with each. And a copula reaches the leak only through $p_{11}$, the chance that both covariates are past the cut — which is the copula evaluated on its diagonal. Two dependences with the same diagonal have the same threshold leak at every cut, whatever their correlation, family or behaviour off the diagonal.

Complementing both indicators negates $a$ and $b$ and leaves their product alone, so the same formula reads a cut in the lower tail through the chance that both covariates are below it. The leak is a statement about whichever tail the cut is in, read through how often the two covariates are in it together.

Fifteen dependences on one line #

The formula makes a prediction that can be checked directly. Fix the cut, and the leak is a function of $p_{11}$ alone — or, more readably, of the conditional share $p_{11}/p$: how often the second covariate is past the cut, given that the first is. Every copula at every correlation should then sit on one curve.

They do. With the cut at the 95th percentile, five copulas at rank correlations of 0.2, 0.4 and 0.6 put the conditional share anywhere from 6.38% to 63.36% and the leak anywhere from 0.62% to 71.91%, and all fifteen lie on the curve the formula draws, to within $10^{-10}$. At independence the conditional share is just $p$, five per cent, and the curve starts there at zero.

This is the cleanest available statement of what the earlier field found by contrasting a copula with its reflection, and of why the mean’s zero needed a second symmetry the marginals could not supply. A rank correlation summarises the dependence across the whole square; a threshold rule is sensitive to one number, the conditional exceedance at its own cut. The two are related, which is why the leak rises with the correlation within a family, and they are not the same thing, which is why the families separate: at a rank correlation of 0.4 the conditional share at the 95th percentile is 8.38% under the Clayton and 42.99% under its reflection, and those two copulas agree to four decimal places on every rank statistic in common use.

It also says what an analyst would need in order to know the leak in advance. Not the copula’s family and not its parameter: one probability, the chance that a unit past the threshold on one covariate is past it on the other. That is a quantity a baseline table could in principle report, and almost never does.

Out past the data: a ceiling or nothing #

The sweep stops at the 1st percentile because a grid of cells cannot place a cut much further out. The formula needs no grid. Fed the joint tail probability exactly — in closed form for the Clayton and the Frank, and by a one-dimensional integral of the conditional law for the Gaussian and the t — it follows the cut out to one unit in a trillion.

As the share past the cut $p$ goes to zero, $c \approx p_{11}$, the product’s variance is about $p_{11}$, and $p(1 - p) + c \approx p + p_{11}$. The leak tends to

$$\frac{2\lambda}{1+\lambda}, \qquad \lambda = \lim_{p \to 0} \frac{p_{11}}{p},$$

and $\lambda$ is the copula’s tail-dependence coefficient: the limiting chance that one covariate is extreme given that the other is. It has closed forms for these families, none of which the grid knows anything about. The Clayton at a rank correlation of 0.4 has a lower coefficient of $2^{-1/\theta} =$ 0.4011, so its leak settles on 57.25% — and gets there quickly, reading 57.21% by one unit in a thousand. The t on four degrees of freedom has $\lambda = 2\,t_5\bigl(-\sqrt{5(1-\rho)/(1+\rho)}\bigr) =$ 0.2163 in both tails, so its leak settles on 35.57%.

The two copulas without tail dependence have $\lambda = 0$, and their leak must go back to nothing however strongly the covariates are related — which is the copula’s version of a fact the field on extremes meets from the other side, that what a tail does cannot be read off the middle of a distribution. The Frank does so promptly: 12.20% at one unit in ten, 0.23% at one in a thousand, and under a hundredth of a point by one in a hundred thousand. The Gaussian does so slowly, which is the familiar sluggishness of normal extremes reappearing in a balancing rule: 19.22% at one in ten, 5.94% at one in a thousand and still 0.32% at one in a million. A Gaussian copula’s covariates are asymptotically independent, but at any cut a protocol could specify they are a long way from behaving as though they were.

The t does something the limit alone would not predict. Its leak passes the ceiling on the way out — 37.42% at one unit in a hundred — and comes back down to 35.57% from above. The 99th-percentile value the sweep reports is therefore not a step towards the limit but the top of a hump the limit sits under, and a protocol that cut a t-dependent pair at the 99th percentile would see a larger leak than one that cut at the ten-thousandth.

A ceiling that does not need a correlation #

The limit depends on the rank correlation only through $\lambda$, and how $\lambda$ moves with the correlation differs between the two families that have one.

The Clayton’s ceiling rises steeply: 1.57% at a rank correlation of 0.1, 57.25% at 0.4 and 83.89% at 0.7. The t’s rises more gently from a much higher start: 18.38% at 0.1, 35.57% at 0.4, 59.02% at 0.7. The two cross between rank correlations of 0.2 and 0.3, below which the t’s far tail leaks more than the Clayton’s even though the Clayton’s dependence is concentrated in exactly the tail being cut.

The reason the t starts high is that its tail dependence does not need a correlation at all. At $\rho = 0$ a t copula on four degrees of freedom still has $\lambda = 2\,t_5(-\sqrt 5) =$ 0.0756, because its two covariates share one random scale and are extreme together for that reason alone. A threshold rule cut far into such a pair removes 14.05% of the interaction between the two thresholds, at a rank correlation of zero, where the median split removes exactly nothing and a Gaussian pair would leak nothing at any cut.

That is worth stating as a caution against the most natural shortcut. Uncorrelated covariates are often treated as covariates whose interaction a balancing rule cannot touch. For a median split that is true and needs no assumption at all. For a threshold it is true only if the covariates are also not extreme together, and nothing in a correlation coefficient — Pearson, Spearman or Kendall — says whether they are.

The ordering a protocol inherits #

A protocol does not choose a copula; nature does. What a protocol chooses is the cut, and the cut decides which copulas are the dangerous ones.

The comparison is the threshold’s counterpart of the field that put a factor of two between copulas at one skewed covariate, with the cut rather than the marginal as the dial. At the 70th percentile the five sit within seven points of each other — the reflected Clayton at 10.53%, the t at 7.62%, the Frank at 7.25%, the Gaussian at 6.63% and the Clayton at 3.78% — and the Frank is above the Gaussian. At the 90th the spread is nearly forty points and the Gaussian, at 19.33%, has passed the Frank at 12.05%. At the 99th the reflected Clayton removes 56.37% and the Clayton 0.59%, the Gaussian has fallen back to 13.44% and the Frank to 2.06%: fifty-six points from top to bottom.

So the question which dependence is worst for a threshold rule has no answer independent of the threshold. Near the median every dependence is benign and the differences are a few points. Past the 90th percentile the differences are the whole story, and which copula is at the bottom depends on whether its dependence is in the tail being cut, which is the earlier essay’s finding, and on whether it has any tail dependence at all, which is this one’s.

It also says where the earlier field’s six-fold ratio came from. At the 84th percentile the Clayton pair sits at 5.33% and 33.36% because the two curves have separated by then but neither has reached its end. At the 99th the same pair is 0.59% and 56.37%, a ratio of about ninety-five. The six-fold gap was a reading part way along a divergence that does not stop until the cut is at the end of the scale.

What a trial that cuts at a dose can do #

The practical setting is the one the earlier essay gave for preferring a threshold to a median split: a cut that means something — a dose above which a treatment changes, a laboratory value that defines a condition. Those cuts are usually in a tail, because the condition is usually unusual.

The quadrant formula turns that into something checkable. Before a trial, the leak depends on one quantity per cut: how often a unit past the threshold on one covariate is past the threshold on the other. A historical cohort with both covariates measured gives a direct estimate of it — a two-by-two table at the cut, not a fitted copula — and the formula turns the estimate into the share of the interaction a balancing rule would leave unprotected. That is the whole of the calculation, and it uses no model of the dependence away from the cut.

What the formula cannot do is extrapolate past the cohort. An estimate of the conditional exceedance at the 99th percentile from a cohort of a few hundred rests on a handful of units, and the far-tail figure shows that the families which agree near the middle disagree most exactly there. Between the 90th percentile and the 99th the Gaussian’s leak falls by six points and the t’s rises by eight, at the same rank correlation, so a reading taken where a cohort has data says little about one taken where it does not.

Three routes to each number #

Every number in the sweep was computed three ways, and the agreement between them is what makes the formula a finding rather than a fit.

The first route is the one the earlier essays used: the Gram solve of the threshold product on the two indicators, on a grid of each copula’s density over the latent normal scale. The second feeds the quadrant formula the same grid’s own $p$ and $p_{11}$; the two share the weights and nothing else, and they agree to $10^{-10}$ at every one of the eighty cuts off the median. The third feeds the formula the copula’s exact diagonal, available in closed form for the Clayton, its reflection and the Frank, and shares nothing with the grid at all; it agrees to within 0.03 of a point, which is the grid’s own accuracy at a cell width of a twentieth.

The cuts are placed on cell boundaries of the grid rather than at the exact percentile, so that each indicator is a partition of the cells and not a cell split in two. The percentiles quoted are therefore the boundary nearest each one, which moves the 84th-percentile reading of the earlier essay by less than a point and leaves every shape unchanged.

One defect was found on the way out to one in a trillion, and it is the kind that produces a confident wrong answer. The Frank’s diagonal involves the logarithm of one plus a number that becomes smaller than a double can resolve; written literally it reads exactly zero past about one in ten billion, the formula fed $p_{11} = 0$ divides zero by zero, and the leak came back as one hundred per cent — a Frank copula, with no tail dependence at all, apparently removing the whole interaction. The repair is the library’s own log1p and expm1, which compute the same expression without forming the sum; with them the Frank reads zero all the way out, as its $\lambda = 0$ requires.

Still open: a leak read off a two-by-two table #

The calculation this essay makes possible has not been made. The quadrant formula needs the conditional exceedance at the cut, and a trial planning to balance on a clinical threshold could estimate it from prior data and attach an interval to the leak — and the interval would be wide, because the estimate rests on the few units past the cut on both covariates.

How wide, at the cohort sizes trials actually have, is a measurement that can be made: draw cohorts of a few hundred from each copula, estimate $p_{11}/p$ at the 90th and 99th percentiles, and push the estimate and its interval through the formula. The prediction is that at the 90th the interval is narrow enough to separate the families and at the 99th it is not, and that the 99th — the cut where the leak varies most between copulas — is exactly where a cohort is least able to say which copula it came from. If that holds, the honest advice for a tail threshold is a bound, not an estimate. Nothing here measures it.

A second open question is a pair of different cuts: a threshold on one covariate and a median split on the other, which is a common design when only one covariate has a clinical cut. The formula generalises to unequal $p$ on the two sides, and whether the median split’s zero protects the mixed product is a short calculation that has not been done.

The claim, and where it stops #

This essay takes how a threshold-balancing rule’s leak depends on where the threshold sits. The claims are that with both covariates cut the rule’s share of the interaction is a closed function of the two quadrant probabilities, so the copula enters only through its diagonal; that at a Spearman rank correlation of 0.4 the leak is exactly zero at the median for every copula, peaks at 19.49% near the 95th percentile under a Gaussian and 13.22% near the 85th under a Frank, and reaches 37.45% under a t and 56.37% under a Clayton at the edge of the sweep; that as the cut runs into the tail the leak tends to $2\lambda/(1 + \lambda)$ of the tail-dependence coefficient — 57.25% and 35.57% for the two families that have one and zero for the two that do not — with the t passing its limit on the way; and that the ordering of the five copulas by leak depends on the cut, the Frank and the Gaussian changing places between the 70th and 90th percentiles.

What stays out, named as a decision: real covariates. The five copulas span the possibilities rather than represent any of them, and which one a pair of baseline covariates resembles in its tail is an empirical question these measurements cannot answer. And a dependence that changes with the cut — a mixture, or a copula whose tail dependence differs from its body’s. The formula covers it, since it needs only the diagonal; the figures here do not.

The boundary against the field’s first essay is that it established the median split’s zero as arithmetic and this essay finds the same arithmetic as one factor in a formula that covers every cut. Against the parity field and its skewed covariates, it holds the marginal fixed and normal throughout, so every effect above belongs to the joint law of the ranks. And against the field on extremes, whose maxima converge slowly for the same reason the Gaussian’s leak decays slowly here, it borrows the tail-dependence coefficient and asks only what it does to a balancing rule.