학습률 일정과 워밍업

One fixed learning rate is rarely ideal for the whole run. A schedule changes it over time, usually large early and small near the end.

Big early steps cover ground fast. Shrinking the rate later lets the model settle precisely into a minimum instead of bouncing around it.

The simplest schedule is step decay: drop the rate by a fixed factor every set number of epochs, like halving it every thirty epochs.

Cosine schedules glide the rate down a smooth curve toward zero. The gentle, gradual decay is a favorite for training modern networks.

Fresh weights are fragile. A full-size rate on step one can blow up the loss, especially with large batches or deep transformers.

Warmup ramps the rate up from near zero over the first few hundred steps. This gentle start keeps early training stable before full speed.

The classic recipe is warmup followed by decay: climb to the peak rate, then ride a cosine curve down. It is the go-to for big models.

A scheduler does nothing until you call step() on it, usually once per epoch right after the optimizer updates the weights.

Log the live learning rate while you train. Seeing it warm up and decay confirms the schedule fires when expected and helps you debug.

Prefer reacting to results? ReduceLROnPlateau drops the rate only when validation loss stops improving, no fixed timetable needed.

A good schedule often boosts final accuracy with zero extra data. Warmup plus cosine decay is a strong, safe default to start from.

A schedule shrinks the learning rate over time so the model settles cleanly, while warmup ramps it up first for stable starts. Warmup plus cosine is a great default. 📉