Thought Toys · Strategy & computation · Exhibit 33

The learning-rate cliff

A ball rolls downhill in fixed-size steps, always jumping toward the bottom. Take timid steps and it glides in politely. Take bigger ones and it gets there faster — right up until one exact step size, past which every jump overshoots worse than the last and the ball is gone.

A ball on a bowl-shaped curve, stepping toward the bottom with each click of the clock. A trail of its recent positions fades behind it. Below the bowl, a strip of bars shows the size of the ball's distance from the bottom over its last several steps, shrinking when it is converging and growing when it is diverging.

step 0 · x = 3.200 · η·L = 0.90 (cliff at 2)

your turn — drag the step size past the cliff and watch the ball fly apart

What you're seeing

The curve is a bowl — a stand-in for the "loss" a machine-learning model is trying to shrink, or just a plain function you want the minimum of. Gradient descent — the workhorse optimization algorithm behind nearly every trained model — repeatedly looks at the local slope and takes a step downhill, proportional to that slope. The size of that step is the learning rate, η. Nudge η up from zero: at first the ball takes small, cautious hops and glides straight into the bottom. Push it a little more and the hops get big enough to overshoot the bottom — the ball lands on the far wall, then the near wall, then the far wall again, bouncing back and forth — but each bounce still lands closer than the last, so it still spirals down to rest.

Then comes the cliff. At one exact step size the bounces stop shrinking altogether — the ball lands at exactly the same height on alternating walls, forever, never getting any closer to the bottom. Push η just past that point and each bounce is worse than the one before: the ball climbs a little higher on the wall every step, and within a few dozen steps it's flown off the bowl entirely, hurtling toward infinity. A single extra notch on the dial is the difference between "finds the answer" and "diverges to nonsense" — the exact failure mode behind a training run whose loss suddenly explodes into NaN.

Now drag how steep the bowl is. A steeper bowl has a stronger restoring slope at any given distance from the bottom, so the very same step size that was perfectly safe on a gentle bowl can push the ball past its own cliff. There is no single learning rate that is safe for every problem — only one that is safe for a given curvature, which is exactly why real training runs spend so much effort tuning (or adapting) it.

The rule, exactly. The bowl is f(x) = ½Lx², whose slope is f′(x) = Lx. Gradient descent subtracts the step size times the slope: xk+1 = xk − ηLxk = (1 − ηLxk — a plain geometric sequence with ratio r = 1 − ηL. So xk = x0·rk exactly: it shrinks to zero whenever |r|<1 (that is, 0<ηL<2 — smoothly for ηL<1, bouncing-but-shrinking for 1<ηL<2), and explodes whenever |r|>1 (ηL>2). Verified in node (improve/verify/33-gradient-descent.js): the closed form matches the simulation to machine precision across a grid of step sizes and curvatures; every ηL<2 case converges to under 1e-8 within 400 steps; every ηL>2 case clears 1,000,000 within 3,000 steps, growing every single step. Counter-examples: at exactly ηL=2 the ball bounces between +x0 and −x0 for 2,000 straight steps without ever converging — more steps do not help at the cliff edge; and η=1.2 converges cleanly when L=1 but diverges past 1,000,000 within 60 steps once L=2, using that exact same η — proof that a fixed learning rate is not a property of the algorithm alone, but of the algorithm and the problem's curvature together.

Also in Strategy & computation: A* pathfinding →

All 23 in Strategy & computation
  1. 10The evolution of trust
  2. 23Sorting algorithms
  3. 24PageRank & the random surfer
  4. 25Huffman coding
  5. 26Dijkstra's shortest path
  6. 27Nash equilibria
  7. 33The learning-rate cliff — you are here
  8. 36A* pathfinding
  9. 37Braess's paradox
  10. 44Diffie–Hellman key exchange
  11. 45Preferential attachment
  12. 46Aliasing & the Nyquist limit
  13. 47The secretary problem
  14. 49Freeze too fast, stay stuck
  15. 51Cross one line, and its territory closes
  16. 56Catch one error, miss the next
  17. 57Why more processors stop helping
  18. 58Why a busy line explodes
  19. 59The set that's only sure when it says no
  20. 60The fit that memorizes instead of learns
  21. 61When the wire breaks, pick one
  22. 67Better at both, and still better off trading
  23. 69Everyone was consistent. The vote wasn't.

← the cabinet · Thought Toys — a cabinet of explorable explanations. Exhibit 33.