The seed is part of the figure

A figure drawn from a deterministic rule either is or is not what it claims. A figure drawn from a sample can be right by luck, and that difference decides how this site is built.

Both halves of that caption are load-bearing, and they pull in opposite directions. The panels differ from one another because forty draws never lie on the line: the tails swing furthest, several panels bend away from the diagonal at one end, and the worst point in the worst panel is more than half a standard deviation off it. Shown any single panel alone, a reader would be entitled to doubt what generated it, and would be wrong. That scatter is the honest content of the figure — it is what forty observations actually look like, and a figure that drew one tidy panel instead would be showing something that does not happen.

But the differences are not the rendering’s. Rebuild the site and the same twenty panels come back, every excursion in the same place, because each panel’s seed is written into the figure’s own specification rather than taken from the machine. So the figure is a statement with a truth value — this is what these forty draws from this law look like — rather than a picture of whatever one run happened to produce. The rest of this essay is about the machinery that buys that property, and about the second thing it buys, which is a check that can fail.

Why nothing here calls the system generator #

If a figure used the ordinary random number generator, every rebuild would produce a different picture. That has three consequences and all of them are fatal.

Every page would change on every build. The version history would be noise, and a real change would be impossible to spot in a diff full of shifted coordinates.

No assertion could be trusted twice. A figure whose assertion passed today might fail tomorrow with no edit in between, and the only way to make the build reliable would be to loosen every tolerance until it stopped testing anything.

Nobody could reproduce a number. A caption saying the coverage is 87.6% would be unverifiable — a reader running the same code would get something else, and could not tell a disagreement from a different draw.

So everything random here comes from a counter-based generator with a stated seed. The same seed gives the same stream on every machine and every build, and a figure is byte-identical each time it is rendered.

And it is what makes a check able to fail #

There is a fourth consequence, and it is the one this site turns on.

A figure drawn from the system generator can only be checked for properties true of every draw — that a curve is positive, that a count is between zero and n. A figure drawn from a stated seed can be checked against the numbers it actually produced.

That is the difference between an assertion that cannot fail and one that can. A gate reading “this cell is 0.002%” is a refutable claim about a specific picture; the same gate against a fresh draw every build would either be vacuous or would fire at random.

Reproducibility here is not a convenience for the reader. It is what makes the figures testable at all.

What the generator has to be #

The generator is PCG — a 64-bit linear congruential step whose output is permuted before being returned — written out in about thirty lines rather than imported.

The permutation is the part that matters. A bare linear congruential generator passes a one-dimensional uniformity test easily and fails in two dimensions: consecutive pairs fall on a small number of parallel planes, so a scatter plot of one draw against the next shows visible stripes.

That failure would be invisible in a histogram and disastrous here, because half the figures on this site plot one draw against another — the QQ panels, the pairs in regression to the mean, every scatter.

So the gate checks both: the uniforms are flat in one dimension, and the pairs are flat in two. The second is the test a bad generator fails, and it is the reason for the thirty lines.

One run is an anecdote #

The second consequence of figures-as-samples is about what a figure is allowed to claim.

A single simulation is one draw from a distribution of simulations. If the claim is about behaviour — that an interval covers 95% of the time, that a test rejects at its stated rate, that an estimator is unbiased — then one run cannot support it, however convincing it looks.

Two responses run through the site.

Where the claim is about a procedure, the assertion runs across many seeds and reports what held for all of them. Every coverage number here is a count over thousands of trials, or an exact sum over the sample space where one is available.

Where a single run genuinely is the point, the figure shows twenty of them. That is the QQ panel figure and the twenty intervals: the reader needs to see the distribution of pictures, because their intuition about how much a single one wanders is unreliable and cannot be corrected by a caption.

The check that a generator is being used at all #

Worth stating because it is the failure that would silently invalidate everything.

A generator that ignored its seed — returning the same stream regardless — would make every acrossSeeds result a thousand copies of one run. Every coverage number would come out as exactly 0% or exactly 100%, and every figure would still render.

The gate therefore requires two things together: the same seed gives the same stream, and a different seed gives a different one. Either alone is satisfiable by broken code.

That pairing is a small instance of the general rule this site’s checks are built on: an assertion that cannot fail proves nothing, so each one is accompanied by input it must refuse.

What reproducibility is not #

It is not accuracy. A seeded simulation reproduces its own answer exactly, including when that answer is wrong.

Reproducibility buys one specific thing: it removes randomness as an explanation for a disagreement. If a reader recomputes a number here and gets something different, the code differs, the arithmetic differs, or one of the two is wrong — and the possibility that it was just a different draw has been eliminated.

That is exactly the property that makes the second discipline work. Every simulation here has a closed form beside it, and the comparison between them is only meaningful because the simulated side does not move.

What a counter-based generator is #

The distinction from the generators most people have met is worth drawing, because it is what makes the reproducibility here practical rather than fussy.

A classical generator keeps a state and steps it forward. Getting the thousandth number requires generating the first nine hundred and ninety-nine, and two parts of a program drawing from the same generator interfere with each other — adding a figure changes every figure after it.

A counter-based generator computes its output from an index, by putting the index through a strong scrambling function. The thousandth number is available directly, streams from different seeds are independent by construction, and adding a figure changes nothing else.

That property is what makes acrossSeeds honest. Each trial gets its own stream from its own seed, so trials are independent, and a trial’s result does not depend on how many trials preceded it or on what else the build was doing.

Reproducible does not mean deterministic in the useful sense #

A distinction worth making, because “seeded” is sometimes taken as a guarantee it is not.

Seeding fixes which sample is drawn. It does not make the sample representative, and it does not stop a figure from having drawn an unusual one.

So a seeded figure showing a striking result is still a single draw, and the fact that it reproduces exactly says nothing about whether it is typical. This is the trap of a carefully-seeded illustration: it looks rigorous and it is one anecdote, permanently preserved.

The protection is the second discipline rather than the first. A claim is checked across many seeds, or by an exact computation with no seeds at all — and where a striking single run appears in a figure, it is labelled as an illustration rather than as evidence.

Choosing a seed is a decision #

A small practice worth naming, because it is where reproducibility quietly becomes dishonest.

If a figure is regenerated with different seeds until one looks good, the seed has become a free parameter fitted to the outcome. The figure is reproducible, the number is honest, and the selection has done exactly what the winner’s curse describes.

Every seed on this site is a small fixed integer chosen before the figure was looked at, and the sequences are contiguous — acrossSeeds(n, f, 1500) uses 1500 through 1500 + n. There is no seed on this site that was chosen after seeing what it produced.

That is not verifiable by a reader, which is the honest limitation. What is verifiable is that the assertions hold across the whole contiguous range, which is the property a hand-picked seed would not have.

Why this matters more here than elsewhere #

Every site in this fleet asserts that its figures compute what they claim. On a deterministic subject that assertion is checkable by anyone who runs the code once.

Here it needs one more thing. A figure that passed its assertion on the seed it shipped with might fail on another, and if the seed were unstated nobody could tell the difference between a robust claim and a lucky one.

Stating the seed makes the difference checkable: run it again with the next seed, and the next. That is the whole argument for the thirty lines of generator, and it is why the first check in the site’s gate is that seeding works at all.

What reproducibility costs elsewhere #

The practice here is unusual for a website and ordinary for scientific computing, and the comparison is instructive.

A published analysis that cannot be re-run produces numbers nobody can check. That has been the subject of a decade of argument in several fields, and the agreed minimum is now: the code, the data, and the seed.

The seed is the part most often omitted, and its omission is the most consequential, because without it a disagreement between the author’s number and a reader’s is uninterpretable. It might be a bug, a version difference, or a different draw, and there is no way to tell which.

This site’s figures are a small case of the same problem. The difference is that here the whole build is the analysis, so the discipline is enforced by the build rather than by intention: a figure that used an unseeded generator would change on every build and the change would be visible in the diff.

That is worth noting as a general technique. Making irreproducibility visible is more reliable than prohibiting it, because the prohibition depends on care and the visibility does not.

The check that had never rejected anything #

The gate above claims that two-dimensional flatness is “where a bad generator shows”. That claim went two years without being shown a bad generator, and it does not survive the experience.

RANDU is the standard example of the failure the permutation is supposed to prevent. It shipped on IBM mainframes through the 1960s and 70s, it is the generator that produced a decade of unreliable Monte Carlo work, and its defect is not subtle: because its multiplier 65539 is 2¹⁶ + 3, every consecutive triple satisfies

$$x_{k+2} = 6x_{k+1} - 9x_k \pmod{2^{31}}$$

exactly. Not approximately, and not usually. Across 100,000 draws the relation holds for every single triple, which means the triples occupy fifteen planes in the unit cube and none of the volume between them is ever generated.

Fed to this site’s two-dimensional check, RANDU scores 113.5 against a 0.999 point of 148.2. It passes. The generator whose name is a byword for lattice structure walks through the gate that exists to catch it, because pairs are the wrong dimension: RANDU’s pairs really are close to flat, and all of the structure is in the triples.

So the check was extended to three dimensions, where the separation is not close. On a 16-cell grid the site’s generator scores 4,000 against a 0.999 point of 4,380, and RANDU scores 158,432 — with 256 of the 4,096 cells never visited at all. The plane-counting relation is asserted directly beside it, so the reason is recorded and not merely the symptom.

The general lesson is the one this site keeps arriving at from different directions. An assertion is evidence only if it would have rejected the thing it was written against, and the only way to know is to feed it that thing. A gate that has never refused anything is indistinguishable from a gate that cannot.

The assumption underneath every number here #

The second finding is worse, because it was not in a check that failed — it was in an assumption no check had been written for.

acrossSeeds(n, f, seed0) runs a trial per seed and uses seeds one apart. Every coverage figure, every error rate and every power curve on this site is a count over that function, and all of them depend on the streams from consecutive seeds being independent of one another. The existing check tested something much weaker: that seed 7 and seed 8 produce different streams. Different is nearly free. Two streams can differ in every value and still be strongly related.

They were. The seeding took a seed, formed the state 2·seed + 1, and discarded two steps of the generator — and two steps is not enough to separate initial states that start two apart. Taking the first draw from each of 20,000 consecutive seeds and running the site’s own two-dimensional test on consecutive pairs gives 177.8 against a 0.999 point of 148.2. That could be luck; it was not. Ten blocks of seeds out of ten came in over the threshold, and at 200,000 seeds the statistic reaches 1,162.9. A statistic that grows with the sample is structure, not noise.

The repair is to mix the seed rather than to use it raw — one pass of SplitMix64 before the first step — which decorrelates any spacing instead of the one spacing that happened to be tested. Strides of 1, 2, 3, 7, 256 and 1,024 now all land in the 79 to 116 range where they belong.

Two things are worth saying plainly about the consequences.

It changed the answers very little. A coverage estimate built from consecutive seeds and one built from seeds 7,919 apart agreed to well inside Monte Carlo error, which is why the defect survived: nothing on the site looked wrong, because nothing on the site was materially wrong.

That is not a reason it was acceptable. The site’s claim is not that its numbers happen to be about right; it is that the machinery underneath them has been checked. A known flaw in the one function every measurement passes through is a flaw in the claim, whatever its magnitude — and the honest version of “it did not matter much” is that nobody knew whether it mattered until it was counted.

Both findings arrived the same way, and neither came from a failing build. They came from asking what each check would have caught, which is a different question from whether it passes.

What is not reproducible here #

Honesty about the boundary, since the claim is strong.

The figures are byte-identical across builds. The numbers in the prose were computed by the same libraries and are quoted by hand, so a change to a library could leave a stale number in an essay that no check would catch.

Several numbers on this site are quoted in prose: the 87.6% coverage, the 0.104 tail ratio, the 31% reversal region. Each is produced by an assertion in the site’s gate, so a change large enough to matter would fail the build — but the assertion states a threshold, not the exact value, so a number drifting from 87.6% to 88.1% would pass it and sit stale in the sentence.

That gap was real until the seeding fix above closed it the hard way. Mixing the seed moved every simulated number on the site at once: the forking-paths rate from 58% to 57%, the winner’s-curse inflation from 2.05× to 2.07×, the power at n = 10 from 13% to 14%. All three were found by grepping the essays, which works exactly once and only for someone who already knows a change is coming.

So the digits are now checked too. Each number quoted in prose is recomputed by the call that produced it and required to render to the same characters that appear in the essay; a number that moves fails the build, names the essay, and prints what the sentence should now say. Mirroring the producing call matters more than it sounds — a check that recomputes a number its own way tests nothing but its own arithmetic, and two of the first drafts of these checks disagreed with the prose only because they had used a different grid and a different seed.

The remaining honest limitation is coverage rather than principle: the check knows about the numbers it has been told about. A sentence that quotes a library result nobody added to the table is exactly as unprotected as every sentence used to be.