Fine-Tuning: Unfreezing and Low Learning Rates

Feature extraction adapts only the classification head, leaving the backbone frozen. Fine-tuning goes further by also updating some or all backbone layers, allowing the model to adapt its representations to your specific data distribution.

Fine-tuning is most beneficial when your domain differs from ImageNet — for example, satellite images, medical scans, or industrial defect photos. The ImageNet features partially transfer, but adapting them yields measurably higher accuracy. The risk is catastrophic forgetting: if you fine-tune with a high learning rate, the new data overwrites the carefully pre-learned features, causing performance to collapse.

The standard fine-tuning recipe has two phases. Phase 1: Freeze the backbone completely and train only the new classification head for several epochs until it converges. This ensures the head starts from a reasonable state before backbone gradients mix in.

Phase 2: Unfreeze all or some backbone layers and continue training with a very low learning rate (typically 10–100× smaller than phase 1). This gently nudges the pre-learned features toward your domain without destroying them. Skipping phase 1 and fine-tuning from the start with a high LR is the most common mistake that causes poor results.

Rather than unfreezing the entire backbone at once, you can unfreeze it layer by layer starting from the top (closest to the output). This is because later layers contain the most task-specific features, while early layers learn universal features (edges, textures) that rarely need updating.

For ResNet-50, the typical selective unfreezing order is: (1) fc (already trainable), (2) layer4, (3) layer3, and finally (4) layer1 and layer2 if needed. Each step requires evaluating whether the additional unfreezing improves validation accuracy, since more trainable parameters increase overfitting risk on small datasets.

Differential learning rates assign different learning rates to different parts of the network. The new head trains with the highest rate (1e-3), recently unfrozen backbone layers with a moderate rate (1e-4), and early backbone layers with the lowest rate (1e-5 or frozen entirely).

PyTorch's optimiser accepts a list of parameter groups, each with its own lr. This is the standard technique in transfer learning literature and is used in fast.ai's discriminative learning rates. The intuition: parts of the network with more general features need fewer adjustments than task-specific layers.

Catastrophic forgetting occurs when fine-tuning with a large learning rate overwrites the pre-learned features, causing validation accuracy to drop below even the feature-extraction baseline. You will see it as a sharp spike in training loss early in phase 2, followed by slow recovery.

To prevent it: (1) always start phase 2 with a very low learning rate, (2) monitor validation accuracy every epoch and stop immediately if it drops below the phase 1 best, (3) use a learning rate scheduler that gradually warms up the backbone LR, and (4) save the best model checkpoint during phase 1 so you can restore it if phase 2 fails.

A fixed learning rate is rarely optimal throughout fine-tuning. Learning rate schedulers adjust the LR during training to improve convergence. For fine-tuning, two schedulers work especially well: CosineAnnealingLR smoothly decays the LR from initial value to near zero following a cosine curve, and ReduceLROnPlateau reduces the LR whenever validation metric stops improving.

ReduceLROnPlateau with patience=2 is a safe default: if validation loss does not improve for 2 consecutive epochs, the LR is multiplied by factor=0.1. This often recovers progress when training plateaus.

Fine-tuning with a larger batch size reduces gradient noise, which is beneficial when making small adjustments to pre-learned features. However, very large batches can degrade generalisation for fine-tuning (the 'sharp minima' problem). A batch size of 32-64 is a good starting point.

Add weight decay (L2 regularisation) to the optimiser: optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-4). Weight decay prevents any single weight from growing too large, which is especially important for fine-tuning where the pre-trained weights already have sensible magnitudes that we do not want to perturb too aggressively.

Data augmentation applies random transformations to training images at each epoch, effectively multiplying the dataset size. This is especially important during fine-tuning when you have few images per class. Common augmentations for natural images: random horizontal flip, random rotation (±15°), random crop, colour jitter (brightness, contrast, saturation).

Apply augmentation only during training, not during validation or inference. Use a separate transform for the validation set that only applies deterministic preprocessing (resize, centre crop, normalise). PyTorch's transforms.Compose makes this easy to manage.

How much improvement does fine-tuning give over feature extraction? On a typical custom dataset with ~1000 images per class, feature extraction might yield 88% accuracy while fine-tuning reaches 92-95%. The improvement depends heavily on how different your data is from ImageNet.

Track these four numbers to understand what is happening: phase 1 val accuracy (feature extraction baseline), phase 2 initial val accuracy (should not drop below phase 1 if LR is correct), phase 2 best val accuracy (fine-tuning benefit), and test accuracy (final, reported only once at the very end to avoid test set leakage).

Fine-tuning ViT (Vision Transformer) models requires some additional care compared to CNNs. ViT models contain Layer Normalisation rather than Batch Normalisation, so the model.eval() / model.train() modes primarily affect dropout rather than normalisation statistics — this makes fine-tuning slightly simpler.

ViT models also benefit from a lower peak learning rate during fine-tuning (1e-5 to 5e-5 vs 1e-4 for CNNs) because the attention weights are sensitive to large updates. The AdamW optimiser is strongly preferred for ViT fine-tuning. With careful fine-tuning, ViT-B/16 can surpass CNN baselines on many tasks.

Test your understanding of fine-tuning strategies from this lesson.

In this lesson you learned: two-phase fine-tuning trains the head first (frozen backbone), then unfreezes the backbone with a very low learning rate to avoid catastrophic forgetting, differential learning rates assign higher rates to the head and lower rates to deeper backbone layers, and selective unfreezing from the top layers down gives the best accuracy-efficiency trade-off. Next up we apply these techniques to a real medical imaging challenge with scarce labels.