การเสื่อมน้ำหนักกับการทำให้เป็นระเบียบแบบ L2

Big weights often mean an overfit model. Both weight decay and L2 regularization push weights toward zero so the network stays simpler.

L2 regularization adds a penalty term, the sum of squared weights, straight into the loss. Minimizing loss then also means shrinking the weights.

Weight decay skips the loss and instead multiplies each weight by a number slightly under one every step. It shrinks weights before the gradient update.

With plain SGD, the two are mathematically identical. The L2 penalty's gradient is exactly the decay term, so it makes no practical difference.

The catch appears with Adam. Its per-weight scaling divides the L2 gradient unevenly, so L2 and true weight decay stop being equal.

Adam shrinks weights with large gradients less and small ones more. Folded-in L2 inherits that bias, weakening the regularization where you need it.

AdamW applies decoupled weight decay, shrinking every weight by the same fraction independent of its gradient. That restores honest regularization.

The weight_decay argument controls the strength. In AdamW it is true decoupled decay, the behavior you usually want.

Values like 0.01 or 0.0001 are typical. Too much decay underfits; too little lets weights grow and overfit. Tune it like any hyperparameter.

It is common to skip decay on bias and norm parameters. Shrinking them rarely helps and can quietly hurt how the model trains.

With SGD, reach for either name freely. With adaptive optimizers, prefer AdamW so your weight decay actually behaves as designed.

Test when the two truly diverge.

L2 adds a penalty to the loss; weight decay shrinks weights directly. They match under SGD but split under Adam, which is why AdamW decouples decay. 🪶