SGD with Momentum
Smoothing updates to roll past noise.
SGD with Momentum is a free Deep Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Plain SGD Forgets
Vanilla SGD steps using only the current gradient. Every update starts from scratch, so a noisy slope makes it wobble and crawl toward the minimum.
Borrow Some Inertia
Momentum gives SGD memory. It keeps a running average of past gradients and rolls in that direction, like a ball gathering speed downhill.
The Velocity Vector
Momentum tracks a velocity that blends the old velocity with the new gradient. The weights then move by that smoothed velocity each step.
v = beta * v + grad
w = w - lr * vThe Beta Knob
The momentum coefficient beta sets how much past steps count. A common value is 0.9, meaning most of the velocity carries over.
Smooths Out Noise
Because momentum averages many gradients, random noise in any single batch mostly cancels. The path to the minimum becomes smoother and steadier.
Rolls Past Small Bumps
Built-up speed lets the optimizer coast through tiny dips and flat spots that would stall plain SGD. The ball does not stop at every pebble.
Turn It On in PyTorch
You do not code this by hand. Just pass momentum to the built-in SGD optimizer and PyTorch tracks the velocity for you.
opt = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)Nesterov Looks Ahead
A sharper variant, Nesterov momentum, peeks at where the velocity is heading before measuring the gradient. It often converges a touch faster.
opt = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, nesterov=True)Mind the Overshoot
Too much speed can carry you past the valley floor. If loss bounces or diverges, lower the learning rate or trim momentum a little.
Why It Still Matters
Even with fancy optimizers around, SGD with momentum remains a strong baseline and often generalizes beautifully on vision models.
A Faster Descent
The payoff is real: momentum usually reaches a good minimum in fewer epochs than plain SGD, with less zig-zagging along the way.
Quick Check
Make sure momentum's role is clear.
Recap
Momentum gives SGD a velocity that blends past gradients, smoothing noise and coasting past small bumps. Set momentum near 0.9 for a faster, steadier descent. 🏂
Frequently asked questions
Is the “SGD with Momentum” lesson free?
Yes — the full text of “SGD with Momentum” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “SGD with Momentum”?
Smoothing updates to roll past noise. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “SGD with Momentum” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.