SGD avec momentum

Vanilla SGD steps using only the current gradient. Every update starts from scratch, so a noisy slope makes it wobble and crawl toward the minimum.

Momentum gives SGD memory. It keeps a running average of past gradients and rolls in that direction, like a ball gathering speed downhill.

Momentum tracks a velocity that blends the old velocity with the new gradient. The weights then move by that smoothed velocity each step.

The momentum coefficient beta sets how much past steps count. A common value is 0.9, meaning most of the velocity carries over.

Because momentum averages many gradients, random noise in any single batch mostly cancels. The path to the minimum becomes smoother and steadier.

Built-up speed lets the optimizer coast through tiny dips and flat spots that would stall plain SGD. The ball does not stop at every pebble.

You do not code this by hand. Just pass momentum to the built-in SGD optimizer and PyTorch tracks the velocity for you.

A sharper variant, Nesterov momentum, peeks at where the velocity is heading before measuring the gradient. It often converges a touch faster.

Too much speed can carry you past the valley floor. If loss bounces or diverges, lower the learning rate or trim momentum a little.

Even with fancy optimizers around, SGD with momentum remains a strong baseline and often generalizes beautifully on vision models.

The payoff is real: momentum usually reaches a good minimum in fewer epochs than plain SGD, with less zig-zagging along the way.

Make sure momentum's role is clear.

Momentum gives SGD a velocity that blends past gradients, smoothing noise and coasting past small bumps. Set momentum near 0.9 for a faster, steadier descent. 🏂