Discriminative Layer-Wise Rates
Train deep and shallow layers differently.
Discriminative Layer-Wise Rates is a free Deep Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
One Rate for All?
So far every layer shared a single learning rate. But early and late layers learn very different things. Maybe they deserve different rates. 🎚️
Early Layers Are General
The early layers of a pretrained net detect edges, colors, and textures. These features transfer to almost any task, so they barely need changing.
Late Layers Are Specific
The later layers capture task-specific patterns. These are the ones that should adapt most to your new dataset.
The Big Idea
Discriminative fine-tuning gives deep layers a tiny learning rate and shallow output layers a larger one. Each part learns at the right speed.
Group the Parameters
To use different rates, you split the model into parameter groups, each with its own settings.
backbone_params = model.features.parameters()
head_params = model.classifier.parameters()Per-Group Learning Rates
PyTorch optimizers accept a list of dicts, one per group, each carrying its own lr. The backbone gets a small one, the head a bigger one.
opt = torch.optim.Adam([
{'params': backbone_params, 'lr': 1e-5},
{'params': head_params, 'lr': 1e-3}])Why It Helps
This protects the precious general features while letting the head adapt quickly. You often get higher accuracy than a single shared rate.
Slanted Across Many Layers
You can scale rates smoothly: each deeper block gets a slightly smaller rate. This gradient of speeds is sometimes called a slanted schedule.
Gradual Unfreezing
A partner technique is gradual unfreezing: unfreeze the top layer first, train, then unfreeze the next, working downward over epochs.
Inspect Your Groups
You can read back each group's settings from the optimizer to confirm every param_group got the rate you intended.
for g in opt.param_groups:
print(g['lr'])When to Reach for It
Discriminative rates pay off most on deep networks where the gap between general and specific layers is large.
Quick Check
In discriminative fine-tuning, which layers should get the smallest learning rate?
Recap
You gave deep layers small rates and the head a larger one using parameter groups, optionally with gradual unfreezing, for sharper transfer learning. 🎯
Frequently asked questions
Is the “Discriminative Layer-Wise Rates” lesson free?
Yes — the full text of “Discriminative Layer-Wise Rates” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Discriminative Layer-Wise Rates”?
Train deep and shallow layers differently. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Discriminative Layer-Wise Rates” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Freeze the Backbone, Train the Head
- Fine-Tune with a Lower Learning Rate
- Discriminative Layer-Wise Rates
- Fine-Tune a Hugging Face Model