Normal Distribution
nothing is called 95% until it has been counted
Concept

Debiased estimator — where it appears

An estimator with its known finite-sample bias subtracted, where that bias has a closed form computable before the data are seen. For an estimated reliability the correction is the sampling variance of each group's event rate, which depends on the forecasts alone and not on the outcomes.

Named by 2 essays across one field — each of them below, with the objects they name alongside it.

Also named here as expected calibration error — the same set of essays touches all of them, so they are one junction rather than several.

One forecaster, one sample, and the bins it was read in. A reliability diagram of 500 forecasts in 10 equal-count bins, drawn over the curve the same forecaster has in population. The forecaster is honest, so its population curve is the diagonal exactly and every departure the points show is sampling. The sample's reliability term reads 0.002262 and its expected calibration error 0.0358, against a true reliability of 0.000000. Both are properties of this binning as much as of this forecaster: at 5 bins and at 50 the same honest forecaster reports 0.001429 and 0.014100.

A curve that is a binning

A forecaster with no miscalibration in it at all reads 0.001429 at five bins and 0.014100 at fifty, on the same five hundred forecasts. The closed form is K/n times the forecaster's own irreducible score, and subtracting it returns zero.

calibrate · Calibration How long an honest forecaster looks broken for. The calibration error shown by a forecaster with no miscalibration in it at all, at five record lengths, drawn against one over the square root of the length so that the closed form is a straight line through the origin. It is 0.1252 at fifty forecasts and 0.0090 at ten thousand, against a closed form of √(2K/πn) times the mean root bin variance which gives 0.1257 and 0.0089. The threshold drawn across it is 0.02, a figure routinely read as evidence that something is wrong; the mean falls under it at 1976 forecasts and the 95th percentile at about 4111. Below that, an honest forecaster and a miscalibrated one are being told apart by a statistic that is mostly the sample size.

The miscalibration a perfect forecaster shows

A forecaster whose true reliability is exactly zero shows a calibration error of 0.1252 on fifty forecasts and 0.0090 on ten thousand. Every one of 1,200 blameless hundred-forecast records exceeds the 0.02 routinely read as evidence of a problem, and the mean does not fall under it until 1,976 forecasts.

calibrate · Calibration

Named alongside it

The objects these essays reach for when they reach for this one.

BinningExpected calibration errorForecast calibrationProbability forecastReliability diagramReliability termSampling variationBase rateBrier scoreCalibration testChi-squareClimatological forecast

All concepts