Thought Toys · Data & inference · Exhibit 135
You have one noisy reading of each of several things that have nothing to do with each other — the weight of a whale, the price of tea, a batting average. You must guess all of them. The obvious plan is to take each reading at face value. It is beaten, for every possible set of true values, by a rule that nudges every reading toward one arbitrary point you picked before looking. Some of the individual guesses get worse. The total gets better — and that is a guarantee, not a tendency. It starts working at three.
One round: the truths, the readings, and the shrunken guesses ● truth ● reading, at face value ● shrunk toward the point
—
—
Each horizontal line is one quantity. The green mark is its true value, fixed for the whole experiment. On every round you get one noisy reading of each — the grey dot, the truth plus a standard normal error — and must guess the truth from it. The grey dot is the face-value guess. The amber dot is the other rule: every reading pulled toward the dashed line, the point ν you chose before looking, by a fraction the rule reads off the data.
The score is the total squared error across all the lines, and the two bars on the right keep the running average over every round so far. Taking the readings at face value scores exactly p on average — one unit of error per quantity, and the white mark on the grey bar is that number. The amber bar's expected height is below it. Run five hundred rounds and it settles below it; move the point ν somewhere silly, or spread the truths out, or draw a completely new set of truths, and it still settles below it — though the further the truths sit from the point, the thinner the gap, and a short run can wobble above the mark before the average finds it. The truths, by the way, are drawn once, within the spread you set, and stay fixed until you ask for new ones; the theorem does not care how they were chosen.
Now drag How many down to two. The two bars become the same bar: with two things the formula has nothing to shrink by, and with one it would push the reading away from the point, so for one or two the page leaves the reading alone. Push back up to three and the amber bar drops under again. That step, from two to three, is the whole paradox. Nothing about the third quantity is related to the first two. It is simply that from three dimensions up, “move everything a little toward one point” is a bet that pays off more often than it costs.
Why does it not cost more? Look at the arrows. When the readings happen to sit far from the point, S — their total squared distance from it — is large, the fraction (p−2)/S is tiny, and the rule barely moves them: it is nearly face value, and loses nearly nothing. When the readings sit close to the point it moves them more, and that is exactly when moving them is likely to be right. The rule is not guessing that the truths are near ν. It is reading, from the readings themselves, how much it can afford to.
The catch that keeps this honest: it wins on the total. A single quantity on its own can go either way — one whose truth happens to sit near the point is helped, one whose truth sits far from it is hurt, and only the sum is promised. If you only care about the whale, shrink nothing. Stein's paradox is a fact about batches. It also needs the noise to be what it is here: independent from line to line, bell-shaped, and the same size on every line. What is unrelated is the noise; the truths themselves can be anything at all.
The total squared error of the shrunken rule can be written, by an integration-by-parts identity Stein found, as p minus a quantity that is positive whenever p is at least three — (p−2)² times the average of 1/S. The page estimates that quantity live from the same readings and prints it under the verdict: the rule's expected score, which the identity guarantees is below p — not a promise about any one round, and stated for the unclipped rule, which the clipped one can only beat. At one or two the quantity is zero or does not exist, which is why the threshold is exactly where it is.
The rule, exactly. Readings Xi ~ N(θi, 1), i = 1 … p, independent; a fixed point ν;
S = Σ (Xi − ν)². The shrunken guess is
θ̂i = ν + max(0, 1 − (p−2)/S) · (Xi − ν)
(the positive-part James–Stein estimator; for p ≤ 2 the page leaves the reading alone).
Face value has risk E Σ(Xi − θi)² = p.
Stein's identity gives the untruncated rule risk p − (p−2)² E[1/S],
which is below p for every θ when p ≥ 3 and equals exactly 2 when every
θi = ν; the positive-part rule does at least as well. Shrinking toward the readings'
own mean instead of a fixed point uses the factor (p−3) and needs p ≥ 4
(Efron & Morris 1973).
Checked before this page was written (improve/verify/135-stein-paradox.js):
Also in Data & inference: Simpson's paradox →