Adam과 AdamW 이해하기

SGD uses a single learning rate for every parameter. Adam adapts the step size for each weight on its own, based on that weight's gradient history.

Adam tracks two running averages: the mean of gradients and the mean of their squares. Together they form the first and second moments.

The first moment is basically momentum, the smoothed average of recent gradients. It decides the overall direction each weight should move.

The second moment estimates each gradient's size. Adam divides by its square root, so noisy weights take smaller steps and quiet ones take larger.

Two decay rates, the betas, control those averages, typically 0.9 and 0.999. They balance how much recent versus older gradients matter.

The averages start at zero, so early steps look too small. Adam applies a bias correction to fix this so updates are sane from step one.

One line gives you Adam. The default learning rate of 0.001 works well across a huge range of models, which is why it is so popular.

Classic Adam mixes weight decay into the gradient, where the adaptive scaling distorts it. The regularization ends up weaker than you intended.

AdamW decouples weight decay from the gradient step and applies it directly to the weights. The decay now works as a clean, predictable shrink.

For transformers and most large models, AdamW is the standard choice. Reach for it first whenever you want fast, reliable convergence.

Adam often trains faster than SGD, yet plain SGD with momentum can generalize better on vision tasks. Try both when accuracy really counts.

Adam adapts a learning rate per weight using gradient mean and variance, while AdamW fixes its weight decay. AdamW is today's go-to optimizer. ⚙️