Thought Toys · Strategy & computation · Exhibit 33
A ball rolls downhill in fixed-size steps, always jumping toward the bottom. Take timid steps and it glides in politely. Take bigger ones and it gets there faster — right up until one exact step size, past which every jump overshoots worse than the last and the ball is gone.
Rolling toward the bottom
step 0 · x = 3.200 · η·L = 0.90 (cliff at 2)
The curve is a bowl — a stand-in for the "loss" a machine-learning model is trying to shrink, or just a plain function you want the minimum of. Gradient descent — the workhorse optimization algorithm behind nearly every trained model — repeatedly looks at the local slope and takes a step downhill, proportional to that slope. The size of that step is the learning rate, η. Nudge η up from zero: at first the ball takes small, cautious hops and glides straight into the bottom. Push it a little more and the hops get big enough to overshoot the bottom — the ball lands on the far wall, then the near wall, then the far wall again, bouncing back and forth — but each bounce still lands closer than the last, so it still spirals down to rest.
Then comes the cliff. At one exact step size the bounces stop shrinking altogether — the ball lands at
exactly the same height on alternating walls, forever, never getting any closer to the bottom. Push η
just past that point and each bounce is worse than the one before: the ball climbs a little higher on
the wall every step, and within a few dozen steps it's flown off the bowl entirely, hurtling toward infinity.
A single extra notch on the dial is the difference between "finds the answer" and "diverges to nonsense" — the
exact failure mode behind a training run whose loss suddenly explodes into NaN.
Now drag how steep the bowl is. A steeper bowl has a stronger restoring slope at any given distance from the bottom, so the very same step size that was perfectly safe on a gentle bowl can push the ball past its own cliff. There is no single learning rate that is safe for every problem — only one that is safe for a given curvature, which is exactly why real training runs spend so much effort tuning (or adapting) it.
improve/verify/33-gradient-descent.js): the closed form matches the simulation to
machine precision across a grid of step sizes and curvatures; every ηL<2 case converges to under
1e-8 within 400 steps; every ηL>2 case clears 1,000,000 within 3,000 steps, growing every single
step. Counter-examples: at exactly ηL=2 the ball bounces between
+x0 and −x0 for 2,000 straight steps without ever converging — more
steps do not help at the cliff edge; and η=1.2 converges cleanly when L=1 but diverges past
1,000,000 within 60 steps once L=2, using that exact same η — proof that a fixed learning rate is
not a property of the algorithm alone, but of the algorithm and the problem's curvature together.
Also in Strategy & computation: A* pathfinding →
← the cabinet · Thought Toys — a cabinet of explorable explanations. Exhibit 33.