Adam & AdamW Explained
Adaptive rates plus decoupled weight decay.
Adam & AdamW Explained is a free Deep Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
One Rate Per Weight
SGD uses a single learning rate for every parameter. Adam adapts the step size for each weight on its own, based on that weight's gradient history.
Two Moving Averages
Adam tracks two running averages: the mean of gradients and the mean of their squares. Together they form the first and second moments.
First Moment Is Momentum
The first moment is basically momentum, the smoothed average of recent gradients. It decides the overall direction each weight should move.
Second Moment Scales Steps
The second moment estimates each gradient's size. Adam divides by its square root, so noisy weights take smaller steps and quiet ones take larger.
The Betas
Two decay rates, the betas, control those averages, typically 0.9 and 0.999. They balance how much recent versus older gradients matter.
Bias Correction
The averages start at zero, so early steps look too small. Adam applies a bias correction to fix this so updates are sane from step one.
Use It in PyTorch
One line gives you Adam. The default learning rate of 0.001 works well across a huge range of models, which is why it is so popular.
opt = torch.optim.Adam(model.parameters(), lr=1e-3)Adam's Weight Decay Flaw
Classic Adam mixes weight decay into the gradient, where the adaptive scaling distorts it. The regularization ends up weaker than you intended.
AdamW Fixes It
AdamW decouples weight decay from the gradient step and applies it directly to the weights. The decay now works as a clean, predictable shrink.
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)The Modern Default
For transformers and most large models, AdamW is the standard choice. Reach for it first whenever you want fast, reliable convergence.
Adaptive, With Caveats
Adam often trains faster than SGD, yet plain SGD with momentum can generalize better on vision tasks. Try both when accuracy really counts.
Quick Check
Pin down the Adam to AdamW difference.
Recap
Adam adapts a learning rate per weight using gradient mean and variance, while AdamW fixes its weight decay. AdamW is today's go-to optimizer. ⚙️
Frequently asked questions
Is the “Adam & AdamW Explained” lesson free?
Yes — the full text of “Adam & AdamW Explained” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Adam & AdamW Explained”?
Adaptive rates plus decoupled weight decay. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Adam & AdamW Explained” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- SGD with Momentum
- Adam & AdamW Explained
- Weight Decay vs L2 Regularization
- Learning Rate Schedules & Warmup