The fit that memorizes instead of learns

What you're seeing

The dashed curve is the real pattern generating the data — a plain sine wave. The 18 dots are noisy samples of it, and the solid curve is the best-fit polynomial of degree d: 1 is a straight line, 2 a parabola, and so on up to a curve wiggly enough to have twelve independent bends. Underneath, the second chart tracks two kinds of error as d climbs: training error — how far the fit sits from the 18 dots it was actually given, and true error — how far the fit sits from the real sine curve, checked at hundreds of points it never saw.

Training error can only ever fall as d rises — more flexibility can always hug the given points more closely, and by d=17 it would pass through all 18 exactly. True error tells a completely different story: it falls too, at first, because a low-degree line or parabola genuinely can't bend the way a sine wave does — that's real, honest bias. But past a point, roughly where d starts to exceed what the data can actually support, the two curves split. The fit keeps hugging the dots tighter, but the extra bends between them start tracking that particular batch of noise rather than the sine wave underneath — and true error turns and climbs, sometimes by orders of magnitude. Turn the noise dial down toward zero and try it: with nothing to memorize, the split disappears and more flexibility just keeps helping.