Normal Distribution
nothing is called 95% until it has been counted
What makes it checkable

Two routes to every number

A site about probability that only simulates has one route to each answer and no way to tell a right one from a plausible one. Every important number here is computed twice, by arithmetic that shares nothing, and the two are required to agree.
11 min read 6 figures Two routes to a numberCount it, do not claim it

There are two ways to get a number in this subject. Derive it, or simulate it. Each has a failure mode the other does not, and using only one leaves no way to detect it.

A derivation can be implemented wrongly and will then produce a smooth, plausible, confident wrong answer for every input.

A simulation can be right and useless, converging slowly enough that the answer in hand is not the answer it is heading towards — or drawing from a distribution subtly different from the one intended.

Neither route can check itself. Together they check each other, and that is the whole method.

Student's t on 5 degrees of freedom, against the normalThe two-sided 95% critical value is 2.571 for t(5) and 1.960 for the normal — 31% wider. Using the normal at this sample size makes every interval too short by that much.00.1000.2000.3000.400-4-2024standard errors from the meandensityt 2.57z 1.96solid: t · dashed: normal31% wider at 5 df
Fig. 1 Two distribution functions, each checked against its own quantile function to better than ten digits.

The round trip

Every distribution function on this site has a quantile function, and they are required to invert each other.

The forward direction is a series or a continued fraction. The backward direction is a rational approximation refined by Halley’s method, or a bisection. They share no arithmetic whatsoever — different algorithms, different constants, different failure modes.

Across the normal, Student’s t at six degrees of freedom, and chi-square at four, at a dozen probabilities from one in a million to one minus one in a million, the worst relative disagreement is 2.7 × 10⁻¹¹.

That agreement is evidence rather than tautology, and it earns its keep: the first version of the incomplete beta function returned the complement in one branch, and the round trip found it immediately. Nothing about the wrong version looked wrong — it returned numbers between zero and one, monotone, with the right endpoints.

Coverage of four nominal 95% intervals, n = 30Computed exactly by summing over all 31 possible counts, not simulated. The Wald interval drops to 26.0% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted
Fig. 2 The exact route: a sum over every possible outcome, carrying no simulation error.
Coverage of a 95% interval for a mean, n = 8Measured over 20,000 samples. The t interval covers 95.0% and the z interval 91.3%. The difference is the price of pretending the standard deviation was known.t interval, 7 df95.0%± 0.3 at 2 s.e.z interval91.3%± 0.4 at 2 s.e.20,000 samples, one seed eachnominal 95%counted, not assumed
Fig. 3 The simulated route, with the Monte Carlo standard error reported beside the estimate.

Exact against simulated

The same discipline applies to the site’s central claim.

Coverage of an interval for a proportion is computed exactly, by summing over the n + 1 possible outcomes with their binomial probabilities. It is also computed by simulation, by building twenty thousand intervals and counting.

At n = 30 and p = 0.2 the exact answer is 96.386% and the simulation gives 96.445%, which is within half of the simulation’s own standard error.

Neither of those routes could have found its own error. The exact sum depends on the binomial probabilities and the interval formula; the simulation depends on the random generator and the same interval formula. The interval formula is common to both, but everything else is independent — so a bug in the binomial arithmetic, or in the generator, or in the summation logic, shows up as a disagreement.

The normal density at sigma = 1.00The bands hold 68.27%, 95.45%, 99.73% of the mass. Those figures are integrals of the curve drawn, not the memorised 68-95-99.7.00.1000.2000.3000.400-4-202standard deviations from the meandensity68.27%95.45%99.73%the bands are integrated, not recalledsigma = 1.00
Fig. 4 The density integrated over a finite range and compared with the difference of its own distribution function.

The density against its own distribution function

A third instance, and the one where the check had to be chosen carefully.

The obvious test of a probability density is that it integrates to one. Applied to Student’s t on three degrees of freedom over a wide range, it comes up 1.0 × 10⁻⁵ short — and that shortfall is not an error. It is the tail, which for t(3) decays slowly enough that a range of ±60 leaves real mass outside.

A check that treated it as an error would have to be loosened until it tested nothing.

So the density is integrated numerically and compared with the difference of its own distribution function over the same finite range. Two unrelated pieces of arithmetic, required to agree to ten digits, with the missing tail accounted for rather than ignored — and the size of that tail is itself asserted, since it is the quantity this site is largely about.

20,000 p-values from a true null, n = 12Flat, as it must be: under the null a p-value is uniform on (0,1). The Kolmogorov–Smirnov distance from uniform is 0.0165 (p = 0.13). That flatness is the check that catches an error a single rejection rate would miss.05001e+30.2000.4000.6000.800p-valuecountflatKS distance 0.0165a p-value that is not uniform is not a p-value
Fig. 5 A check that constrains every quantile at once rather than a single summary.

Uniformity as a third route

For tests, the second route is different in kind and stronger than either.

A test’s rejection rate can be checked against α. That is one point of one distribution. Requiring the p-values to be uniform constrains every quantile at once, and catches the errors a rate check sails past — a wrong degrees of freedom, a one-sided slip, a variance with the wrong denominator.

It is the best example on the site of a general principle: a check that constrains a whole distribution is worth far more than one that constrains a summary, and usually costs the same.

Where the bootstrap works and where it does notUniform data on [0, 1]. For the mean the percentile bootstrap covers 93.6%. For the maximum it covers 0.0%, because a resample can never contain a value larger than the largest one observed, so the interval cannot reach above it.the sample mean93.6%nominal 95%the sample maximum0.0%nominal 95%3,000 samples of 40the same procedure, two statisticsresampling cannot see past the data
Fig. 6 One of the refusals: a procedure whose coverage must be shown collapsing, or the check has stopped working.

And the checks are checked

Everything above is an assertion that passes. An assertion that has never rejected anything proves nothing at all, so the site’s gate ends by feeding the machinery input it must refuse:

  • a generator returning the same value however it is seeded
  • a quantile requested at probability exactly zero and exactly one
  • p-values halved by a one-sided/two-sided slip, which the uniformity check must detect
  • a percentile bootstrap for the maximum of a uniform, whose coverage must be shown collapsing

Each of those must throw. If any of them stopped throwing, the corresponding check has quietly stopped working, and the build fails — which is the only way to know that a green tick still means something.

What a single route conceals

Two failure modes, each invisible from inside the route that produced it.

A derivation implemented wrongly produces smooth, plausible, confidently wrong output. It has no noise, so it looks more trustworthy than a simulation; it is monotone where it should be; its endpoints are right. The incomplete beta function that returned the complement in one branch had all of those properties and was wrong by a large amount over half its domain.

A simulation drawing from the wrong distribution converges beautifully to the wrong answer. The standard error shrinks as expected, repeated runs agree, and everything about the diagnostics says the computation is working — because it is. It is computing something else.

Neither route contains the information needed to detect its own failure. That is not a comment on care; it is structural.

Choosing routes that share as little as possible

The value of a cross-check is proportional to how independent the two routes are, and it is worth being deliberate about that rather than assuming it.

The distribution-function round trip is strong because the forward and backward directions share nothing: a continued fraction one way, a rational approximation refined by Halley the other, different constants, different failure regimes.

The coverage comparison is weaker, because both routes use the same interval formula. A bug in the interval formula would appear in both and cancel. What it does check is everything else — the binomial arithmetic, the summation, the generator, the counting logic — and it is worth knowing exactly what it does not cover.

The uniformity check is strongest of all, because it compares the implementation against a fact derived from the definition of a p-value rather than against another implementation. There is no shared code at all.

Ranking checks by shared-code surface is a useful habit, and it usually reveals that the most reassuring-looking check is the least informative.

When the two routes disagree

The interesting case, since the essay so far describes agreements.

A disagreement is a fact, not a verdict. Three things it can mean, and the order to check them in.

One implementation is wrong. The commonest, and usually the newer one.

The comparison is not like for like. A finite integration range against an infinite one; a continuity correction applied on one side; a definition that differs by an offset. The density check on this site is exactly this case resolved properly — the numerical integral was compared against the distribution function over the same finite range, and the residual was identified as the tail rather than called an error.

The disagreement is real and interesting. Rare, and the reason to look. A simulation that stably disagrees with a derivation may be revealing an assumption the derivation made and the simulation did not.

The failure to avoid is loosening the tolerance until the disagreement disappears, which converts a signal into silence and is the most common response.

The cost, and why it is small

The whole discipline sounds expensive and is not.

The closed forms in lib/dist.js are about three hundred lines, most of it two standard continued fractions. The round trip is a loop over a dozen probabilities. The coverage comparison is one simulation beside a sum that already existed.

Against that: it caught a wrong branch in the beta function, it established that the generator and the summation agree, and it makes every number on the site checkable by someone who trusts neither route individually.

That last property is the point. A reader does not have to take the derivation on faith or the simulation on faith. They have to believe that two independent computations agreeing to ten digits are unlikely to be wrong in the same direction — which is a much easier thing to believe, and is the closest this subject gets to proof.

The routes this site does not have

Completeness requires listing where the discipline is not applied, since a site claiming two routes everywhere would be overstating.

The paradox generators have one route. Simpson’s reversal, the base-rate curve and regression to the mean are direct arithmetic with no independent implementation to check against — though regression to the mean is the exception, where the simulated shift is checked against its closed-form prediction.

The forking-paths simulation has one route. There is no closed form for the distribution of the minimum of twenty correlated p-values, so the measurement stands alone. What it does carry is a comparison against the independent case, which bounds it — and the assertion requires the measured rate to fall below that bound, which would catch a simulation that had lost the correlation.

The power curves have one route, though they could have two: the power of a t test has a closed form via the non-central t distribution, and implementing it would give exactly the kind of check this essay argues for. That is a gap, and it is recorded as one.

Listing the gaps is part of the discipline. A site that claimed the property uniformly and had it in two thirds of cases would be making the more comfortable claim and the less useful one.

Why ten digits and not three

A question the round-trip figure invites: agreement to 2.7e-11 is far beyond anything the subject needs, so why demand it.

Because the tolerance is not about the application, it is about the detection. A bug in a continued fraction typically produces an error of order 1e-3 or larger — a wrong branch, a missing term, an off-by-one in an index. Demanding 1e-10 leaves an enormous margin between “correct” and “any realistic error”, so the check has essentially no false negatives.

Loosening the tolerance to something the application would notice, say 1e-4, would let through a class of real bugs while still looking like a check. That is the worst outcome: a green tick that means less than it appears to.

The general rule: set the tolerance by what a bug would look like, not by what the application needs. They are different numbers and the first is usually much tighter.

Where this goes next

20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like The seed is part of the figure Every other site in this fleet draws from a deterministic rule, so a figure either is or is not what it claims. Here the figures are samples, and a sample can be right by luck. That changes what a figure has to carry. 0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted What the 95% refers to An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time. 05001e+30.2000.4000.6000.800p-valuecountflatKS distance 0.0165a p-value that is not uniform is not a p-value A p-value that is not flat is not a p-value Under a true null, p-values are uniform. That is stronger than saying the test rejects 5% of the time, it constrains the whole distribution rather than one point of it, and it catches implementation errors that a rejection rate sails past.