Normal Distribution
nothing is called 95% until it has been counted
Concept

Significance level — where it appears

The probability of rejecting a true null hypothesis that a test is designed to allow, fixed before the data are seen. A test holds its level when its actual rejection rate under the null does not exceed it, and the limit or threshold it produces is read off at that probability.

Named by 2 essays across one field — each of them below, with the objects they name alongside it.

Expected loss of three upper limits against the level each is set at, fifteen exponential observations, penalty 20. A failure costs 20 standard deviations of margin. At the conventional 97.5% the t limit's loss is 2.143, Hall's 1.922 and the fitted family's 1.530. Each at its own best level: the t limit 1.263 at 99.95%, Hall's 1.862 at 98.50%, the family 1.308 at 99.25%. The best fixed multiple of the standard error costs 1.255.

The level a limit should be set at

When a failure costs twenty times a standard deviation of margin, a t limit on fifteen exponential observations set at the conventional 97.5% has an expected loss of 2.143. Set at 99.95%, the same formula's loss is 1.263 — within 1% of the best any fixed multiple of s/√n can do, and ahead of both Hall's transformation and the fitted gamma family at their own best levels. Hall's is the worst at every level on every source. Choosing the level does more than choosing the construction, and the correction that reads the sample is the one no level rescues.

tails · Student Expected loss of three upper limits against the level each is set at, when a failure is charged by its size, fifteen exponential observations, penalty 20. A failure costs 20 times the distance, in standard deviations, by which the mean exceeds the limit. At the conventional 97.5% the t limit's loss is 0.728, Hall's 1.136 and the fitted family's 0.835. Each at its own best level: the t limit 0.726 at 97.00%, Hall's 0.805 at 90.00%, the family 0.735 at 92.50%.

A failure charged by its size

Charge an upper limit's failure by how far the true mean exceeds it rather than by a fixed amount, and the level a limit should be set at falls further than any other change in this comparison moves it: on fifteen exponential observations, a penalty of twenty per standard deviation of shortfall asks for a t limit at 97% where a penalty of twenty per failure asked for 99.95%. The failures are small whichever limit produces them — about an eighth of a standard deviation — so a charge by size is a small charge per failure. The ranking hardly moves: the t limit stays within about 1% of the fitted family, whose failures are no smaller than its own, and Hall's transformation stays last by 11%.

tails · Student

Named alongside it

The objects these essays reach for when they reach for this one.

Decision theoryExpected lossSample sizeSkewnessStudent's tUpper confidence limitCoverageExpected shortfall

All concepts