Deep Learning Academy · 课时

权重衰减与 L2 正则化

理解其中影响实际效果的细微差异

权重衰减与 L2 正则化 是 CoddyKit 上的免费 Deep Learning Academy 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Deep Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Deep Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Keep Weights Small

Big weights often mean an overfit model. Both weight decay and L2 regularization push weights toward zero so the network stays simpler.

L2 Adds to the Loss

L2 regularization adds a penalty term, the sum of squared weights, straight into the loss. Minimizing loss then also means shrinking the weights.

loss = data_loss + lam * (w ** 2).sum()

Decay Shrinks Directly

Weight decay skips the loss and instead multiplies each weight by a number slightly under one every step. It shrinks weights before the gradient update.

w = w - lr * grad - lr * wd * w

Same Thing for SGD

With plain SGD, the two are mathematically identical. The L2 penalty's gradient is exactly the decay term, so it makes no practical difference.

Adam Breaks the Tie

The catch appears with Adam. Its per-weight scaling divides the L2 gradient unevenly, so L2 and true weight decay stop being equal.

Why It Distorts

Adam shrinks weights with large gradients less and small ones more. Folded-in L2 inherits that bias, weakening the regularization where you need it.

Decoupling Is the Fix

AdamW applies decoupled weight decay, shrinking every weight by the same fraction independent of its gradient. That restores honest regularization.

Set It in PyTorch

The weight_decay argument controls the strength. In AdamW it is true decoupled decay, the behavior you usually want.

opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

Pick a Strength

Values like 0.01 or 0.0001 are typical. Too much decay underfits; too little lets weights grow and overfit. Tune it like any hyperparameter.

Spare the Biases

It is common to skip decay on bias and norm parameters. Shrinking them rarely helps and can quietly hurt how the model trains.

The Takeaway

With SGD, reach for either name freely. With adaptive optimizers, prefer AdamW so your weight decay actually behaves as designed.

Quick Check

Test when the two truly diverge.

Recap

L2 adds a penalty to the loss; weight decay shrinks weights directly. They match under SGD but split under Adam, which is why AdamW decouples decay. 🪶

常见问题解答

「权重衰减与 L2 正则化」课时是免费的吗？

是的 — 「权重衰减与 L2 正则化」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Deep Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Deep Learning Academy 课程共包含 4 节课。

「权重衰减与 L2 正则化」这节课中我会学到什么？

理解其中影响实际效果的细微差异你通过在浏览器中直接运行的动手代码来练习 Deep Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Deep Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Deep Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「权重衰减与 L2 正则化」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Deep Learning Academy 课中编写并运行代码吗？

能。每节 Deep Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Deep Learning Academy