Two routes to every number

There are two ways to get a number in this subject. Derive it, or simulate it. Each has a failure mode the other does not, and using only one leaves no way to detect it.

A derivation can be implemented wrongly and will then produce a smooth, plausible, confident wrong answer for every input.

A simulation can be right and useless, converging slowly enough that the answer in hand is not the answer it is heading towards — or drawing from a distribution subtly different from the one intended.

Neither route can check itself. Together they check each other, and that is the whole method.

What agreement bounds, and what it does not #

Two routes agreeing to four significant figures is not a proof, and it is worth being precise about the class of errors it does rule out.

An error present in both routes has to live in what they share — the specification of the quantity, the constants it is defined by, the convention for what is being integrated. An error in either implementation shows up as a disagreement, because the two implementations have nothing in common.

So the pair checks the code against itself and neither one checks the definition. That is why a third route of a different kind is worth more than a third implementation of the same kind: uniformity is a property the quantity must have whatever its definition says, so it tests the specification rather than the arithmetic.

The round trip #

Every distribution function on this site has a quantile function, and they are required to invert each other.

The forward direction is a series or a continued fraction. The backward direction is a rational approximation refined by Halley’s method, or a bisection. They share no arithmetic whatsoever — different algorithms, different constants, different failure modes.

Across the normal, Student’s t at six degrees of freedom, and chi-square at four, at a dozen probabilities from one in a million to one minus one in a million, the worst relative disagreement is 2.7 × 10⁻¹¹.

That agreement is evidence rather than tautology, and it earns its keep: the first version of the incomplete beta function returned the complement in one branch, and the round trip found it immediately. Nothing about the wrong version looked wrong — it returned numbers between zero and one, monotone, with the right endpoints.

Exact against simulated #

The same discipline applies to the site’s central claim.

Coverage of an interval for a proportion is computed exactly, by summing over the n + 1 possible outcomes with their binomial probabilities. It is also computed by simulation, by building twenty thousand intervals and counting.

At n = 30 and p = 0.2 the exact answer is 96.386% and the simulation gives 96.445%, which is within half of the simulation’s own standard error.

Neither of those routes could have found its own error. The exact sum depends on the binomial probabilities and the interval formula; the simulation depends on the random generator and the same interval formula. The interval formula is common to both, but everything else is independent — so a bug in the binomial arithmetic, or in the generator, or in the summation logic, shows up as a disagreement.

The density against its own distribution function #

A third instance, and the one where the check had to be chosen carefully.

The obvious test of a probability density is that it integrates to one. Applied to Student’s t on three degrees of freedom over a wide range, it comes up 1.0 × 10⁻⁵ short — and that shortfall is not an error. It is the tail, which for t(3) decays slowly enough that a range of ±60 leaves real mass outside.

A check that treated it as an error would have to be loosened until it tested nothing.

So the density is integrated numerically and compared with the difference of its own distribution function over the same finite range. Two unrelated pieces of arithmetic, required to agree to ten digits, with the missing tail accounted for rather than ignored — and the size of that tail is itself asserted, since it is the quantity this site is largely about.

Uniformity as a third route #

For tests, the second route is different in kind and stronger than either.

A test’s rejection rate can be checked against α. That is one point of one distribution. Requiring the p-values to be uniform constrains every quantile at once, and catches the errors a rate check sails past — a wrong degrees of freedom, a one-sided slip, a variance with the wrong denominator.

It is the best example on the site of a general principle: a check that constrains a whole distribution is worth far more than one that constrains a summary, and usually costs the same.

And the checks are checked #

Everything above is an assertion that passes. An assertion that has never rejected anything proves nothing at all, so the site’s gate ends by feeding the machinery input it must refuse:

• a generator returning the same value however it is seeded
• a quantile requested at probability exactly zero and exactly one
• p-values halved by a one-sided/two-sided slip, which the uniformity check must detect
• a percentile bootstrap for the maximum of a uniform, whose coverage must be shown collapsing

Each of those must throw. If any of them stopped throwing, the corresponding check has quietly stopped working, and the build fails — which is the only way to know that a green tick still means something.

What a single route conceals #

Two failure modes, each invisible from inside the route that produced it.

A derivation implemented wrongly produces smooth, plausible, confidently wrong output. It has no noise, so it looks more trustworthy than a simulation; it is monotone where it should be; its endpoints are right. The incomplete beta function that returned the complement in one branch had all of those properties and was wrong by a large amount over half its domain.

A simulation drawing from the wrong distribution converges beautifully to the wrong answer. The standard error shrinks as expected, repeated runs agree, and everything about the diagnostics says the computation is working — because it is. It is computing something else.

Neither route contains the information needed to detect its own failure. That is not a comment on care; it is structural.

Choosing routes that share as little as possible #

The value of a cross-check is proportional to how independent the two routes are, and it is worth being deliberate about that rather than assuming it.

The distribution-function round trip is strong because the forward and backward directions share nothing: a continued fraction one way, a rational approximation refined by Halley the other, different constants, different failure regimes.

The coverage comparison is weaker, because both routes use the same interval formula. A bug in the interval formula would appear in both and cancel. What it does check is everything else — the binomial arithmetic, the summation, the generator, the counting logic — and it is worth knowing exactly what it does not cover.

The uniformity check is strongest of all, because it compares the implementation against a fact derived from the definition of a p-value rather than against another implementation. There is no shared code at all.

Ranking checks by shared-code surface is a useful habit, and it usually reveals that the most reassuring-looking check is the least informative.

When the two routes disagree #

The interesting case, since the essay so far describes agreements.

A disagreement is a fact, not a verdict. Three things it can mean, and the order to check them in.

One implementation is wrong. The commonest, and usually the newer one.

The comparison is not like for like. A finite integration range against an infinite one; a continuity correction applied on one side; a definition that differs by an offset. The density check on this site is exactly this case resolved properly — the numerical integral was compared against the distribution function over the same finite range, and the residual was identified as the tail rather than called an error.

The disagreement is real and interesting. Rare, and the reason to look. A simulation that stably disagrees with a derivation may be revealing an assumption the derivation made and the simulation did not.

The failure to avoid is loosening the tolerance until the disagreement disappears, which converts a signal into silence and is the most common response.

The cost, and why it is small #

The whole discipline sounds expensive and is not.

The closed forms this site’s distributions are computed from are about three hundred lines, most of it two standard continued fractions. The round trip is a loop over a dozen probabilities. The coverage comparison is one simulation beside a sum that already existed.

Against that: it caught a wrong branch in the beta function, it established that the generator and the summation agree, and it makes every number on the site checkable by someone who trusts neither route individually.

That last property is the point. A reader does not have to take the derivation on faith or the simulation on faith. They have to believe that two independent computations agreeing to ten digits are unlikely to be wrong in the same direction — which is a much easier thing to believe, and is the closest this subject gets to proof.

The routes this site does not have #

Completeness requires listing where the discipline is not applied, since a site claiming two routes everywhere would be overstating.

The paradox generators have one route. Simpson’s reversal, the base-rate curve and regression to the mean are direct arithmetic with no independent implementation to check against — though regression to the mean is the exception, where the simulated shift is checked against its closed-form prediction.

The forking-paths simulation has one route. There is no closed form for the distribution of the minimum of twenty correlated p-values, so the measurement stands alone. What it does carry is a comparison against the independent case, which bounds it — and the assertion requires the measured rate to fall below that bound, which would catch a simulation that had lost the correlation.

The power curves have one route, though they could have two: the power of a t test has a closed form via the non-central t distribution, and implementing it would give exactly the kind of check this essay argues for. That is a gap, and it is recorded as one.

Listing the gaps is part of the discipline. A site that claimed the property uniformly and had it in two thirds of cases would be making the more comfortable claim and the less useful one.

Why ten digits and not three #

A question the round-trip figure invites: agreement to 2.7e-11 is far beyond anything the subject needs, so why demand it.

Because the tolerance is not about the application, it is about the detection. A bug in a continued fraction typically produces an error of order 1e-3 or larger — a wrong branch, a missing term, an off-by-one in an index. Demanding 1e-10 leaves an enormous margin between “correct” and “any realistic error”, so the check has essentially no false negatives.

Loosening the tolerance to something the application would notice, say 1e-4, would let through a class of real bugs while still looking like a check. That is the worst outcome: a green tick that means less than it appears to.

The general rule: set the tolerance by what a bug would look like, not by what the application needs. They are different numbers and the first is usually much tighter.

Three failures the routes caught, all in one week #

The discipline is easy to state and its value is hard to argue for in the abstract, so here are the three defects it found in this site’s own libraries during a single phase of work. All three had been present since the foundation, all three produced plausible numbers, and none of them made anything look wrong.

A quantile that returned its own bracket. The chi-square and t quantiles were computed by bisecting a fixed interval. Ask the chi-square for its 0.999 point on 4,095 degrees of freedom and the answer is beyond the bracket, so bisection converges on the endpoint and returns 1,000 — a number whose actual cumulative probability is zero, returned without complaint. The t quantile did the same, returning −400 where the true value is −318,310.

A series that gave up quietly. Underneath the quantile, the incomplete gamma function summed a series with an iteration cap of 300. The series needs about √a terms, so from a few thousand degrees of freedom upward it was truncating: at a = 16,383 it wanted 992 terms, stopped at 300, and returned 0.4905 for a probability that is exactly 0.5. A one per cent error, arrived at silently.

A generator whose seeds were not independent. The seeding formed a state from the seed and discarded two steps, which is not enough to separate seeds that start two apart — and acrossSeeds uses seeds one apart, so the first draws of consecutive trials were related. The seed essay has the measurement.

What found each of them, and what did not #

The instructive part is which route did the catching, because it was not the same one twice.

The first two were found by the round trip: compute a quantile, put it back through the distribution function, and require the original probability. A cdf and a quantile computed by unrelated arithmetic agreeing to ten digits is evidence; the moment the bracket capped, the round trip disagreed in the second digit. No amount of checking the quantile against itself would have shown anything.

But the round trip had been passing for two years, because it only tested degrees of freedom up to fifty and probabilities down to 0.005 — every one of them inside the bracket. The check was correct and its coverage was the problem, which is a different failure from a wrong check and needs a different remedy: extending the range until it included the region where the method could plausibly break.

The third was not found by any existing route at all. It was found by asking what the reproducibility check would have rejected, and noticing that “the same seed gives the same stream, a different seed gives a different one” is satisfied by streams that differ and are still correlated. The check was too weak rather than wrong, and nothing inside the check could have revealed that.

Three lessons, in increasing order of how uncomfortable they are.

A second route catches arithmetic errors. That is the easy case and the one the discipline was built for.

A second route only catches them where it is run. Both bracket failures lived in the range the round trip did not visit, and a check’s range is a choice somebody made once and nobody revisits.

And no route catches a missing assumption. The seeding defect was not an error in any computation; it was a property nobody had thought to require. The only thing that surfaced it was asking, of each check in turn, what input would make it fail — and then supplying that input. Every check on this site now has one, and the two that were added because of this are a generator known to be broken and the seeding this site used to use.

The route the prose did not have #

One gap remained after all of that, and it is worth recording because it was the largest.

The figures on this site are checked by their own assertions and the libraries are checked against each other. The numbers typed into the essays were checked by neither. They were copied by hand from a computation that ran once, and nothing compared them with the code afterwards.

That gap became visible the moment the seeding was fixed, because the fix moved every simulated number on the site at once — three sentences in three different essays went stale in a single commit, and they were found by grepping for the old digits. Grepping works when somebody already knows a change is coming, and not otherwise.

So the prose now has a route too. Each quoted number is recomputed by the call that produced it and required to render to the same characters that appear in the essay. It is the same discipline applied one level out, and it closes the last place on this site where a number was asserted rather than checked.