Weight Decay vs L2 Regularization
The subtle difference that matters.
Weight Decay vs L2 Regularization is a free Deep Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Keep Weights Small
Big weights often mean an overfit model. Both weight decay and L2 regularization push weights toward zero so the network stays simpler.
L2 Adds to the Loss
L2 regularization adds a penalty term, the sum of squared weights, straight into the loss. Minimizing loss then also means shrinking the weights.
loss = data_loss + lam * (w ** 2).sum()Decay Shrinks Directly
Weight decay skips the loss and instead multiplies each weight by a number slightly under one every step. It shrinks weights before the gradient update.
w = w - lr * grad - lr * wd * wSame Thing for SGD
With plain SGD, the two are mathematically identical. The L2 penalty's gradient is exactly the decay term, so it makes no practical difference.
Adam Breaks the Tie
The catch appears with Adam. Its per-weight scaling divides the L2 gradient unevenly, so L2 and true weight decay stop being equal.
Why It Distorts
Adam shrinks weights with large gradients less and small ones more. Folded-in L2 inherits that bias, weakening the regularization where you need it.
Decoupling Is the Fix
AdamW applies decoupled weight decay, shrinking every weight by the same fraction independent of its gradient. That restores honest regularization.
Set It in PyTorch
The weight_decay argument controls the strength. In AdamW it is true decoupled decay, the behavior you usually want.
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)Pick a Strength
Values like 0.01 or 0.0001 are typical. Too much decay underfits; too little lets weights grow and overfit. Tune it like any hyperparameter.
Spare the Biases
It is common to skip decay on bias and norm parameters. Shrinking them rarely helps and can quietly hurt how the model trains.
The Takeaway
With SGD, reach for either name freely. With adaptive optimizers, prefer AdamW so your weight decay actually behaves as designed.
Quick Check
Test when the two truly diverge.
Recap
L2 adds a penalty to the loss; weight decay shrinks weights directly. They match under SGD but split under Adam, which is why AdamW decouples decay. 🪶
Frequently asked questions
Is the “Weight Decay vs L2 Regularization” lesson free?
Yes — the full text of “Weight Decay vs L2 Regularization” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Weight Decay vs L2 Regularization”?
The subtle difference that matters. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Weight Decay vs L2 Regularization” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- SGD with Momentum
- Adam & AdamW Explained
- Weight Decay vs L2 Regularization
- Learning Rate Schedules & Warmup