Normal Distribution
nothing is called 95% until it has been counted
What makes it checkable

The seed is part of the figure

Every other site in this fleet draws from a deterministic rule, so a figure either is or is not what it claims. Here the figures are samples, and a sample can be right by luck. That changes what a figure has to carry.
10 min read 5 figures One run is an anecdoteTwo routes to a number

A figure drawn from a deterministic rule either is or is not what it claims. A figure drawn from a sample can be right by luck, and that difference decides how this site is built.

Twenty samples of 40, every one of them genuinely normalEach panel is a quantile-quantile plot of 40 draws from a normal distribution. The worst point in the worst panel sits 1.52 standard deviations off the line. Anything a reader would reject here would be a false alarm.20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like
Fig. 1 Twenty panels from twenty stated seeds. Every one is reproducible byte for byte on any machine.

Why nothing here calls the system generator

If a figure used the ordinary random number generator, every rebuild would produce a different picture. That has three consequences and all of them are fatal.

Every page would change on every build. The version history would be noise, and a real change would be impossible to spot in a diff full of shifted coordinates.

No assertion could be trusted twice. A figure whose assertion passed today might fail tomorrow with no edit in between, and the only way to make the build reliable would be to loosen every tolerance until it stopped testing anything.

Nobody could reproduce a number. A caption saying the coverage is 87.6% would be unverifiable — a reader running the same code would get something else, and could not tell a disagreement from a different draw.

So everything random here comes from a counter-based generator with a stated seed. The same seed gives the same stream on every machine and every build, and a figure is byte-identical each time it is rendered.

Two measurements of the same thing, correlated 0.60Pick the worst 15% on the first measurement and their average rises by 0.55 on the second. Pick the best and theirs falls by 0.59. No treatment was given to anybody.first measurementsecond+0.55-0.59predicted 0.64 from the correlation alonenobody was treated
Fig. 2 A figure that plots one draw against another, which is exactly where a poor generator shows its structure.

What the generator has to be

The generator is PCG — a 64-bit linear congruential step whose output is permuted before being returned — written out in about thirty lines rather than imported.

The permutation is the part that matters. A bare linear congruential generator passes a one-dimensional uniformity test easily and fails in two dimensions: consecutive pairs fall on a small number of parallel planes, so a scatter plot of one draw against the next shows visible stripes.

That failure would be invisible in a histogram and disastrous here, because half the figures on this site plot one draw against another — the QQ panels, the pairs in regression to the mean, every scatter.

So the gate checks both: the uniforms are flat in one dimension, and the pairs are flat in two. The second is the test a bad generator fails, and it is the reason for the thirty lines.

Twenty 95% intervals for a proportion that really is 0.350 of the twenty miss the true value. The 95% is a property of the procedure across repetitions — no single interval has a 95% chance of anything, because it either contains 0.35 or it does not.the truth, 0.350.00.20.40.60.80 of 20 missedthe 95% belongs to the procedure, not to one interval
Fig. 3 Where a single run is the point, twenty of them are shown, because the distribution of pictures is the thing being conveyed.

One run is an anecdote

The second consequence of figures-as-samples is about what a figure is allowed to claim.

A single simulation is one draw from a distribution of simulations. If the claim is about behaviour — that an interval covers 95% of the time, that a test rejects at its stated rate, that an estimator is unbiased — then one run cannot support it, however convincing it looks.

Two responses run through the site.

Where the claim is about a procedure, the assertion runs across many seeds and reports what held for all of them. Every coverage number here is a count over thousands of trials, or an exact sum over the sample space where one is available.

Where a single run genuinely is the point, the figure shows twenty of them. That is the QQ panel figure and the twenty intervals: the reader needs to see the distribution of pictures, because their intuition about how much a single one wanders is unreliable and cannot be corrected by a caption.

20,000 p-values from a true null, n = 12Flat, as it must be: under the null a p-value is uniform on (0,1). The Kolmogorov–Smirnov distance from uniform is 0.0165 (p = 0.13). That flatness is the check that catches an error a single rejection rate would miss.05001e+30.2000.4000.6000.800p-valuecountflatKS distance 0.0165a p-value that is not uniform is not a p-value
Fig. 4 Twenty thousand seeded runs. A generator that ignored its seed would produce a spike here rather than a flat histogram.

The check that a generator is being used at all

Worth stating because it is the failure that would silently invalidate everything.

A generator that ignored its seed — returning the same stream regardless — would make every acrossSeeds result a thousand copies of one run. Every coverage number would come out as exactly 0% or exactly 100%, and every figure would still render.

The gate therefore requires two things together: the same seed gives the same stream, and a different seed gives a different one. Either alone is satisfiable by broken code.

That pairing is a small instance of the general rule this site’s checks are built on: an assertion that cannot fail proves nothing, so each one is accompanied by input it must refuse.

Coverage of four nominal 95% intervals, n = 30Computed exactly by summing over all 31 possible counts, not simulated. The Wald interval drops to 26.0% and is jagged everywhere; Clopper–Pearson never falls below 95% and pays for it in width.0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted
Fig. 5 An exact computation with no randomness in it at all, against which the simulated route is checked.

What reproducibility is not

It is not accuracy. A seeded simulation reproduces its own answer exactly, including when that answer is wrong.

Reproducibility buys one specific thing: it removes randomness as an explanation for a disagreement. If a reader recomputes a number here and gets something different, the code differs, the arithmetic differs, or one of the two is wrong — and the possibility that it was just a different draw has been eliminated.

That is exactly the property that makes the second discipline work. Every simulation here has a closed form beside it, and the comparison between them is only meaningful because the simulated side does not move.

What a counter-based generator is

The distinction from the generators most people have met is worth drawing, because it is what makes the reproducibility here practical rather than fussy.

A classical generator keeps a state and steps it forward. Getting the thousandth number requires generating the first nine hundred and ninety-nine, and two parts of a program drawing from the same generator interfere with each other — adding a figure changes every figure after it.

A counter-based generator computes its output from an index, by putting the index through a strong scrambling function. The thousandth number is available directly, streams from different seeds are independent by construction, and adding a figure changes nothing else.

That property is what makes acrossSeeds honest. Each trial gets its own stream from its own seed, so trials are independent, and a trial’s result does not depend on how many trials preceded it or on what else the build was doing.

Reproducible does not mean deterministic in the useful sense

A distinction worth making, because “seeded” is sometimes taken as a guarantee it is not.

Seeding fixes which sample is drawn. It does not make the sample representative, and it does not stop a figure from having drawn an unusual one.

So a seeded figure showing a striking result is still a single draw, and the fact that it reproduces exactly says nothing about whether it is typical. This is the trap of a carefully-seeded illustration: it looks rigorous and it is one anecdote, permanently preserved.

The protection is the second discipline rather than the first. A claim is checked across many seeds, or by an exact computation with no seeds at all — and where a striking single run appears in a figure, it is labelled as an illustration rather than as evidence.

Choosing a seed is a decision

A small practice worth naming, because it is where reproducibility quietly becomes dishonest.

If a figure is regenerated with different seeds until one looks good, the seed has become a free parameter fitted to the outcome. The figure is reproducible, the number is honest, and the selection has done exactly what the winner’s curse describes.

Every seed on this site is a small fixed integer chosen before the figure was looked at, and the sequences are contiguous — acrossSeeds(n, f, 1500) uses 1500 through 1500 + n. There is no seed on this site that was chosen after seeing what it produced.

That is not verifiable by a reader, which is the honest limitation. What is verifiable is that the assertions hold across the whole contiguous range, which is the property a hand-picked seed would not have.

Why this matters more here than elsewhere

Every site in this fleet asserts that its figures compute what they claim. On a deterministic subject that assertion is checkable by anyone who runs the code once.

Here it needs one more thing. A figure that passed its assertion on the seed it shipped with might fail on another, and if the seed were unstated nobody could tell the difference between a robust claim and a lucky one.

Stating the seed makes the difference checkable: run it again with the next seed, and the next. That is the whole argument for the thirty lines of generator, and it is why the first check in the site’s gate is that seeding works at all.

What reproducibility costs elsewhere

The practice here is unusual for a website and ordinary for scientific computing, and the comparison is instructive.

A published analysis that cannot be re-run produces numbers nobody can check. That has been the subject of a decade of argument in several fields, and the agreed minimum is now: the code, the data, and the seed.

The seed is the part most often omitted, and its omission is the most consequential, because without it a disagreement between the author’s number and a reader’s is uninterpretable. It might be a bug, a version difference, or a different draw, and there is no way to tell which.

This site’s figures are a small case of the same problem. The difference is that here the whole build is the analysis, so the discipline is enforced by the build rather than by intention: a figure that used an unseeded generator would change on every build and the change would be visible in the diff.

That is worth noting as a general technique. Making irreproducibility visible is more reliable than prohibiting it, because the prohibition depends on care and the visibility does not.

What is not reproducible here

Honesty about the boundary, since the claim is strong.

The figures are byte-identical across builds. The numbers in the prose were computed by the same libraries and are quoted by hand, so a change to a library could leave a stale number in an essay that no check would catch.

Several numbers on this site are quoted in prose: the 87.6% coverage, the 0.104 tail ratio, the 31% reversal region. Each is produced by an assertion in the site’s gate, so a change large enough to matter would fail the build — the assertion states the threshold, not the exact value.

That is a partial protection rather than a complete one. A number that drifted from 87.6% to 88.1% would pass the gate and be stale in the prose. The honest description is that the claims are checked and the digits are not, and a reader relying on a specific digit should re-run the computation rather than trust the sentence.

Where this goes next

0.4000.6000.80010.2000.4000.6000.800the true proportionactual coverage of a nominal 95% intervalnominal 95%Wald — the textbook oneWilsonAgresti–CoullClopper–Pearsonsummed over all 31 outcomesno interval is 95% until it is counted Two routes to every number A site about probability that only simulates has one route to each answer and no way to tell a right one from a plausible one. Every important number here is computed twice, by arithmetic that shares nothing, and the two are required to agree. 20 independent samples, 40 points eachall twenty are normalthis is what the noise looks like What normal actually looks like A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there. 05001e+30.2000.4000.6000.800p-valuecountflatKS distance 0.0165a p-value that is not uniform is not a p-value A p-value that is not flat is not a p-value Under a true null, p-values are uniform. That is stronger than saying the test rejects 5% of the time, it constrains the whole distribution rather than one point of it, and it catches implementation errors that a rejection rate sails past.