Learning Rate Schedules & Warmup
Step, cosine, and warmup strategies.
Learning Rate Schedules & Warmup is a free Deep Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
A Rate That Changes
One fixed learning rate is rarely ideal for the whole run. A schedule changes it over time, usually large early and small near the end.
Why Decay Helps
Big early steps cover ground fast. Shrinking the rate later lets the model settle precisely into a minimum instead of bouncing around it.
Step Decay
The simplest schedule is step decay: drop the rate by a fixed factor every set number of epochs, like halving it every thirty epochs.
sched = torch.optim.lr_scheduler.StepLR(opt, step_size=30, gamma=0.5)Cosine Annealing
Cosine schedules glide the rate down a smooth curve toward zero. The gentle, gradual decay is a favorite for training modern networks.
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=100)The Cold-Start Problem
Fresh weights are fragile. A full-size rate on step one can blow up the loss, especially with large batches or deep transformers.
Warmup Eases In
Warmup ramps the rate up from near zero over the first few hundred steps. This gentle start keeps early training stable before full speed.
Warmup Then Decay
The classic recipe is warmup followed by decay: climb to the peak rate, then ride a cosine curve down. It is the go-to for big models.
Step the Scheduler
A scheduler does nothing until you call step() on it, usually once per epoch right after the optimizer updates the weights.
opt.step()
sched.step()Watch the Current Rate
Log the live learning rate while you train. Seeing it warm up and decay confirms the schedule fires when expected and helps you debug.
print(sched.get_last_lr())Plateau-Based Decay
Prefer reacting to results? ReduceLROnPlateau drops the rate only when validation loss stops improving, no fixed timetable needed.
sched = torch.optim.lr_scheduler.ReduceLROnPlateau(opt)Schedules Are Free Wins
A good schedule often boosts final accuracy with zero extra data. Warmup plus cosine decay is a strong, safe default to start from.
Quick Check
Confirm what warmup is for.
Recap
A schedule shrinks the learning rate over time so the model settles cleanly, while warmup ramps it up first for stable starts. Warmup plus cosine is a great default. 📉
Frequently asked questions
Is the “Learning Rate Schedules & Warmup” lesson free?
Yes — the full text of “Learning Rate Schedules & Warmup” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Learning Rate Schedules & Warmup”?
Step, cosine, and warmup strategies. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Learning Rate Schedules & Warmup” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- SGD with Momentum
- Adam & AdamW Explained
- Weight Decay vs L2 Regularization
- Learning Rate Schedules & Warmup