A score that rewards lying

Ask a scoring rule where it wants a forecaster to put its number. At a true probability of a quarter, the Brier score’s expected charge is smallest at 0.2499999983, the logarithmic score’s at 0.2500000054 and the spherical score’s at 0.2499999994 — three rules, found by searching over the whole unit interval rather than by differentiating a formula, each landing on the truth to within the resolution of the search. Across eight true probabilities the worst departure among them is 10⁻⁸, which is the search’s own step, and the most any of them could be improved on by lying is 8.3·10⁻¹⁷.

Now ask the same question of the score that looks like the plainest of the four. An absolute-error score — the distance between a probability and an outcome, unsquared, which is what somebody reaches for on the grounds that squaring is arbitrary — has its expected charge minimised at 0.0000000010, which is the lower bound of the search. Not near a quarter. At the end. And the saving is 0.125000 of a point, which is exactly an eighth.

That is the standard case, and it is worth having counted rather than quoted. What follows it is less standard and corrects a guess this field started with. A skill score — a proper score expressed as a fraction of a reference — is proper against a fixed reference, and it is still proper when the reference is estimated from the outcomes of the very sample being scored. What breaks it is a reference computed from the forecaster’s own reports, and the direction that breakage takes is available in one line before any search is run.

Propriety is a search, not a shape #

A rule is proper when a forecaster’s expected charge is smallest at the probability it actually believes. That is a statement about an optimum, so it is checked by finding the optimum. The search below is a coarse grid refined by golden section, deliberately written so that it reports the better of the two endpoints as well as the best interior point — because a search that assumes an interior minimum will report the inside of whatever bracket it was handed, and one of these four rules has no interior minimum at all.

The expected charge for reporting r when the truth is p is p·s(r, 1) + (1 − p)·s(r, 0), and for the absolute-error score that is

$$p\,(1 - r) + (1 - p)\,r \;=\; p + r\,(1 - 2p)$$

which is linear in r. A linear function on an interval is minimised at an endpoint, always, and which endpoint is decided by the sign of 1 − 2p. Below a half the coefficient is positive and the optimum is 0; above a half it is negative and the optimum is 1. There is no true probability whatsoever at which an absolute-error score wants to be told the truth, except the two ends and the exact half where the line is flat.

There is a respectable reason somebody reaches for it, and knowing what that reason is makes the failure less surprising. An absolute-error loss applied to a numerical prediction elicits the median of the predictive distribution, in the same way a squared-error loss elicits the mean; both are sensible losses and each is right for a different summary. The trouble is that the outcome being predicted here is 0 or 1, so its median is 0 or 1, and a rule that asks for the median of a Bernoulli is asking for a certainty. The score is not broken by carelessness — it is doing exactly what it does elsewhere, applied to a variable whose median carries none of the information the forecast was for. The same substitution of one summary for another is what makes a variance and a quantile different targets rather than two readings of one spread, and here the substitution happens silently because nobody wrote the word median down.

At a true quarter the truthful report costs 0.375000 and reporting zero costs 0.250000. The three proper rules charge the truthful report 0.187500, 0.562335 and 0.209431 respectively, and none of them can be improved on. The difference between the two families is not that one is more forgiving; it is that the graph of one of them is a straight line.

The same rule tells a different lie above a half #

The direction of the lie flips at a half, which is the part that makes the failure hard to spot in a record. A forecaster gaming an absolute-error score does not push everything down or everything up. It pushes everything outward, and the two halves of its record look like opposite errors.

At a true 0.7 the absolute-error optimum is 1 and the saving is 0.120000. At a true 0.3 the optimum is 0 and the saving is the same 0.120000, by symmetry. So the forecaster that gives this score what it asks for reports nothing but zeros and ones, and its record — every forecast at an extreme, roughly the right proportion of them ones — is a record whose mean forecast can be exactly the base rate. It would pass the crudest calibration check going.

It would not pass a reliability diagram, and that is worth saying because it locates the defect. The lie is visible in the shape of the calibration curve, which is precisely what holding a forecaster’s mean report at the base rate leaves for a diagram to find. What an improper score does is create the incentive; what a diagram does is catch it after the fact. The two are not substitutes, and the order matters — an evaluation that scores first and diagnoses second has already paid the forecaster for the distortion it is about to notice.

The gain has a closed form and it is exactly an eighth #

The saving from lying is not a number that has to be simulated. Reporting the truth costs 2p(1 − p); reporting the nearer end costs min(p, 1 − p); so the gain is

$$2p(1 - p) - \min(p,\, 1 - p)$$

which is zero at 0, at ½ and at 1, and takes its maximum of exactly ⅛ at p = ¼ and again at p = ¾.

The searched values match the closed form to 10⁻⁹ at every truth tried: 0.045000 at a true 0.05, 0.080000 at 0.1, 0.125000 at 0.25, 0.120000 at 0.3 and 0.000000 at 0.5. Two routes, one grid search and one algebraic expression, sharing no arithmetic.

The shape of that curve says something the maximum alone does not. The gain is largest in the middle of each half, which is exactly where an informative forecaster spends most of its time — a forecaster that only ever says 0.02 or 0.98 has little to gain, and one whose reports cluster around a quarter has the most. The rule pays best for corrupting the forecasts that carry the most information.

Counted, the liar wins every record there is #

The population arithmetic says a lie is profitable in expectation. Whether it wins on a record of a realistic length is a separate question, because a difference in expected score can be swamped by sampling. Here it is not, and not by a small margin.

Two hundred records of two thousand forecasts each, from an honest forecaster with the full signal, beside a liar reporting 1 wherever the honest posterior is above a half and 0 wherever it is below — which is precisely what the absolute-error optimum instructs. The truthful forecaster loses 0.074862 ± 0.000393 on that score, against a closed form of 0.074984 obtained by integrating the pointwise gain above over the forecaster’s own law. And it loses on 200 of 200 records: not one draw in two hundred goes the other way.

On the Brier score the same pair reverses completely. The truthful forecaster gains 0.066459 ± 0.000384 against a closed form of 0.066495, and the liar wins 0 of 200. One pair of forecasters, one set of records, two scoring rules, and a unanimous verdict in each direction. The counted figures and the integrals agree to within the standard error of the count in both cases, which is what makes the two routes a check rather than a restatement.

Both figures have an integral beside them and neither integral shares any arithmetic with its count. The closed form for the absolute-error gap is the pointwise gain 2p(1 − p) − min(p, 1 − p) integrated against the forecaster’s own signal distribution; the closed form for the Brier gap is the analogous integral with a squared charge, and both are quadrature over a grid the simulation never touches. That is the two-route discipline doing the job it exists for: a counted difference of 0.074862 could be a bug in the sampling, and an integral of 0.074984 could be a bug in the algebra, and the two agreeing to within the standard error of the count is the coincidence neither explanation gets for free.

The unanimity is the useful part. A difference in mean score is the kind of thing a forecast comparison has to test rather than read, because two hundred records of a difference could be noise. Two hundred out of two hundred is not a result that needs a test attached to it, and it is worth reporting the win count alongside the mean for exactly that reason.

Propriety is necessary and it is not sufficient #

Establishing that three rules are proper does not narrow the choice to one, and the field’s first essay measured the reason: two proper scores rank the same pair of forecasters in opposite orders.

The hedged forecaster scores 0.476965 and the loud one 0.501048 under the logarithmic rule, against 0.154504 and 0.149979 under the Brier rule — the order swaps. Both rules are proper, so both are safe from the failure this essay is otherwise about, and they still do not agree about which of two miscalibrated forecasters is better. Propriety guarantees that a forecaster cannot profit by misreporting its own belief. It guarantees nothing whatsoever about how two forecasters who are both wrong are to be ranked against each other.

So the choice of score is doing two separate jobs and only one of them has a right answer. Ruling out the absolute-error score is settled by the arithmetic above. Choosing between the Brier and logarithmic scores is a decision about what a confident mistake should cost, and it is the same kind of decision as choosing the charge a model selection levies per parameter — one that looks technical, is answered by convention, and moves the winner.

What the liar destroys is the ordering #

There is one more thing the improper score costs, and it is not in the score at all.

A monotone distortion of a forecast leaves the ranking of events untouched, which is why three forecasters with different reliabilities read the same 0.868312 there. That protection has a condition on it: the relabelling has to be strictly increasing. The absolute-error optimum is not — it is a step function, mapping every forecast below a half to 0 and every one above to 1, so the ordering inside each half is discarded entirely and what survives is a two-valued ranking.

That is a genuine loss of information rather than a relabelling, and it is not measured here: the liar’s own area under the curve is not among the numbers this essay computes, and it should not be inferred from the three that are. The liar with two answers computes it. What can be said is the structural part. A score that rewards lying does not merely mis-price a forecaster; the behaviour it rewards destroys the one property that survives every monotone distortion, and a record of zeros and ones cannot be repaired by any recalibration afterwards, because the information the repair would need has already been thrown away.

A proper score with a reference in its denominator #

The subtler failure is the one this field guessed wrong about, and the wrong guess is worth stating because it is the natural one.

A skill score reports a forecaster’s score as the fraction of a reference score it removes: 1 − score/reference. The honest forecaster’s Brier score is 0.141479 against a reference of 0.234237, so its skill is 0.396000. The tempting worry is that computing the reference from the same sample breaks propriety — that a forecaster could move the denominator by moving its own reports, and that estimating a climatology in-sample is therefore an incentive problem.

That worry is wrong, and it is wrong for a reason available in one line. A reference computed from the outcomes — the observed base rate of the events, and its variance — does not contain the forecaster’s reports at all. Given the outcomes it is a constant, so the skill score is an affine decreasing function of the Brier score, and an affine decreasing function of a proper score is optimised exactly where the proper score is. Estimating the reference from the sample costs precision and costs nothing else.

What breaks propriety is a reference read off the forecaster’s own average probability: taking f̄(1 − f̄) as the climatology on the grounds that it is what the forecaster itself thinks the base rate is. Now the denominator moves when the reports move, and the derivative at the truthful report is computable before any search. The numerator’s derivative is zero there, by propriety. The denominator’s is (1 − 2f̄)·df̄/dt, which is strictly positive whenever the base rate is below a half. So the derivative of the whole ratio at the truth is strictly positive, and the optimum is somewhere else.

Swept over shifts of the probit of every report, the fixed-reference skill score is optimised at a shift of exactly zero, which is the propriety it inherits. The own-average version is optimised at 0.162452, where the reported skill reads 0.408633 against the honest 0.396000 — a gain of 0.012633, or 3.19% of the honest figure, bought with a Brier score that is 0.002035 worse. The distortion moves the mean forecast from 0.374449 to 0.414458, walking the reference towards a half where it is largest.

This is the same defect as a benchmark that is chosen by the same data as its rivals, one level down: a comparison is only as sound as the thing in its denominator being fixed independently of the thing in its numerator. It is also a reason to be careful about what a skill figure is being compared with. A skill score is a comparison against a benchmark expressed as a ratio, and everything that makes a benchmark’s place in the table decide what a comparison concludes applies here with a multiplication on top: a denominator that moves by a few per cent moves every reported skill in the table, in a way no single forecaster’s own numbers reveal. The fixed reference has none of that, which is the argument for it and is stronger than the propriety argument, since propriety was never actually at stake.

And it is a defect the forecaster does not have to intend — a mild systematic shift, from any cause, raises reported skill while lowering real accuracy, and nothing in the reported number says which happened.

What is claimed, and the direction not measured #

Three claims, each checked two ways. The Brier, logarithmic and spherical rules are optimised at the truth, found by search over the whole interval at eight true probabilities, to 10⁻⁸ — with a refusal beside it, since a check that never rejects proves nothing: the same search applied to the absolute-error rule must land at an end, and does. The gain from lying on that rule matches 2p(1 − p) − min(p, 1 − p) to 10⁻⁹ at every truth, and counted over two hundred records it is 0.074862 against an integral of 0.074984 with the liar winning all two hundred. And the Brier skill score is proper against a fixed or an outcome-estimated reference and improper against a report-estimated one, with the optimum located at 0.162452 and the direction predicted before the sweep was run.

The improperness of the skill score is measured over a level shift only. The distortion family swept is one-dimensional: the same amount added to the probit of every report. A forecaster free to move both the level and the confidence multiplier could do better, and how much better is not measured. That is a deliberate limit rather than an oversight — one dimension is enough to establish that the optimum is not at the truth, which is the whole of what improperness means, and a larger search would report a bigger number without changing the claim.

What is not addressed at all is the practical question of which reference a real evaluation should use. Nothing here says a sample climatology is a good idea; it says the objection to it is not propriety. The precision cost of estimating a denominator from the same events, and how it interacts with the record lengths at which an honest forecaster’s diagram already looks broken, is a different measurement and is not made here.