Thought Toys · Chance & inference · Exhibit 121
A rating is not a measurement of how good you are. It is a guess, revised after every game by one number — how much the newest result is allowed to move you. Turn that number up and your rating is current, and wrong, because most single games are mostly luck. Turn it down and your rating is calm, and wrong, because it still describes the player you were last year. There is a best setting in between, and it depends on something the rating system cannot see.
Forty players, rated by their results ● a player ● a rating that was exactly right
—
—
Forty players each have a real strength, which the rating system never gets to see. They play each other at random. After every game both ratings move by K times the surprise — the gap between what happened and what the two ratings had predicted. Beat someone the system expected you to beat and you gain almost nothing; beat someone it expected to crush you and you gain nearly the whole of K.
The left panel is the scoreboard's honesty. Each dot is one player: across is the strength they really have, up is the number the system has given them. The cyan diagonal is where a perfect rating would put everybody. Nobody sits exactly on it, and the interesting question is not whether the cloud is fat but why.
Drag K down to 1. The cloud goes lopsided — strong players stuck below the line, weak players floating above it — because at one point a game a rating takes hundreds of games to travel anywhere, and it is still on its way. Now drag K up to 80 and the cloud fattens symmetrically. Nobody is lagging now; everybody is being flung around by results that were coin-flips.
The middle-right panel puts a number on that. It runs the whole season again at eleven different values of K and plots the typical error each one leaves. It is a valley. Everything to the left of the bottom is a rating that has not caught up; everything to the right is a rating that is reacting to noise. The dial does not trade speed against accuracy. It trades one kind of wrongness against another, and the best you can do is stand where the two are equally bad.
Now the part worth finding. Drag the second slider — how fast the players themselves are really changing — and watch the valley slide. In a field where nobody improves, the valley sits far to the left: any responsiveness you buy is wasted, because there is nothing to respond to. In a churning field it slides right, because a slow rating is now permanently describing somebody who no longer exists. In this model the best K is not a property of the rating system. It is a property of the world the rating system is pointed at — and nothing in a player's results tells you which world you are in. How far that generalises is a fair question, and the note below says exactly which world was tested.
This is why chess uses several K values at once: 40 for a newcomer with no history, 20 for an established player, 10 once you are a titled master whose strength is genuinely stable. Those are not three levels of respect. They are three different guesses about how much there is left to learn.
The rule, exactly. Two players with ratings Ra and Rb. The system's prediction, and the entire Elo system, is one line:
Ea = 1 / (1 + 10(Rb − Ra)/400) Ra ← Ra + K(Sa − Ea)— where Sa is 1 for a win and 0 for a loss. Whatever one player gains the other loses, so the total of all ratings never changes; a rating is a claim about where you stand relative to everyone else, not an absolute quantity. The 400 is a choice of units: it is defined as the gap at which the stronger player is expected to score 10 out of 11. Here the same curve also decides who actually wins, on the players' hidden real strengths — so the model the system uses and the world it is measuring agree, and every error you see on this page is lag or luck, never a mis-specified model. Skill drift is a random walk re-centred each round, so the population average stays at 1500 where Elo's zero-sum update pins it.
Which world was tested. Forty players on screen and sixty in the checks, a fixed roster with no newcomers, uniformly random pairings, one game per player per round, and outcomes drawn from the same logistic the ratings use. Every figure below is the mean over several seeds at the stated drift, measured after a burn-in of half the run, so none of them is one lucky season. The checks explore K from 0 to 128 — wider than the slider's 1 to 80 — because a claim about the ends of the curve needs points past where you can drag. Change any of those choices and the shape of the argument should survive while the exact best K moves; that robustness sweep is owed and not yet done, so read the numbers as this world's, not as a law.
Verified in node before this page existed
(improve/verify/121-elo-rating.js, 15 checks):
Arpad Elo published the system in 1967 and FIDE adopted it in 1970. The reading of K as the gain of a tracking filter — and therefore as a bias-against-variance choice rather than an administrative one — is the same idea that sits under a Kalman filter's gain and under the learning rate in gradient descent. Modern systems (Glicko, TrueSkill) do the obvious next thing: carry an explicit uncertainty per player and let each player have their own K, large while the system is unsure of you and small once it is not.
Also in Chance & inference: Buffon's needle →