Fine-Tune with a Lower Learning Rate
Gently update pretrained weights.
Fine-Tune with a Lower Learning Rate is a free Deep Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Beyond Just the Head
Freezing the backbone is fast, but sometimes you want the whole model to adapt. Fine-tuning unfreezes it and trains everything together. 🔧
Unfreeze the Backbone
To fine-tune, you flip requires_grad back to True so the pretrained weights can move again during training.
for p in model.parameters():
p.requires_grad = TrueThe Danger of a Big Step
Those pretrained weights are already excellent. A large learning rate would overwrite them with noisy updates and destroy what they learned.
Go Gentle
The fix is a lower learning rate. Small steps nudge the pretrained weights toward your task without erasing their knowledge.
opt = torch.optim.Adam(model.parameters(), lr=1e-5)How Much Lower?
A common rule of thumb: use a learning rate ten to a hundred times smaller than you would for training from scratch.
Warm Up the Head First
A popular trick: train only the head for a few epochs, then unfreeze and fine-tune. This avoids a shock to the backbone early on.
Watch Validation Loss
Fine-tuning can quickly overfit a small dataset. Keep an eye on validation loss and stop as soon as it starts climbing.
Fewer Epochs Needed
Because the model already understands the data, fine-tuning usually converges in just a few epochs, not dozens.
Pair It With a Scheduler
A scheduler can shrink the learning rate further over time, letting the model settle gently into a good solution.
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=5)Full Fine-Tune vs Frozen
Full fine-tuning needs more data and compute but reaches higher accuracy. Frozen training is cheaper. The tradeoff depends on your dataset size.
Keep the Best Checkpoint
Since fine-tuning can drift, save the model whenever validation accuracy improves so you never lose your best version.
Quick Check
You unfroze a pretrained model to fine-tune it. What learning rate should you use?
Recap
You unfroze the backbone, fine-tuned with a low learning rate, warmed up the head first, and watched validation to stop overfitting. 🎯
Frequently asked questions
Is the “Fine-Tune with a Lower Learning Rate” lesson free?
Yes — the full text of “Fine-Tune with a Lower Learning Rate” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Fine-Tune with a Lower Learning Rate”?
Gently update pretrained weights. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Fine-Tune with a Lower Learning Rate” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Freeze the Backbone, Train the Head
- Fine-Tune with a Lower Learning Rate
- Discriminative Layer-Wise Rates
- Fine-Tune a Hugging Face Model