Domain Adaptation: Medical Imaging with Scarce Labels

Medical imaging presents unique transfer learning challenges. Unlike natural photographs, chest X-rays, MRI scans, and histology slides look nothing like ImageNet images: they are grayscale or have domain-specific colour patterns, the features that matter (lesions, nodules, calcifications) are subtle and domain-specific, and labelled data requires expert radiologists — making large labelled datasets expensive and rare.

Despite these challenges, ImageNet pre-trained models consistently outperform training from scratch on medical imaging tasks, even when the visual appearance differs substantially. Universal low-level features (edge detectors, texture filters) transfer across domains, providing a strong initialisation that speeds convergence and improves generalisation with scarce labels.

CheXpert is a benchmark chest X-ray dataset with 224,316 images and 14 labels (Cardiomegaly, Pleural Effusion, Pneumonia, Atelectasis, etc.). In our scarce-label scenario, we simulate using only a small fraction — say 1% (about 2,243 images) — to mimic real-world clinical settings where annotation budget is limited.

This is a multi-label classification problem: each image can have multiple pathologies simultaneously, unlike single-label classification. The target is a vector of 14 binary values, and we use Binary Cross-Entropy with Logits (BCE) applied element-wise. Evaluation uses AUC-ROC per pathology, averaged across all 14 labels.

Most medical images (X-rays, CT scans) are grayscale (1 channel), but ImageNet pre-trained models expect 3-channel RGB inputs. The simplest solution is Image.convert('RGB') which replicates the single channel three times, or transforms.Grayscale(num_output_channels=3) in the transform pipeline.

This is slightly wasteful — the three channels are identical — but in practice it works well because the model simply learns to weight all three channels equally. An alternative is to replace the first convolutional layer with a new Conv2d(1, 64, kernel_size=7, ...) and initialise it by averaging the three input channel weights. This is more principled but adds training complexity.

For multi-label classification, each label is an independent binary prediction. We use nn.BCEWithLogitsLoss() applied to a vector of 14 logits. The loss is computed element-wise and averaged across both the 14 classes and the batch size.

Imbalanced labels are a major challenge in medical imaging: only 5-10% of images have Pneumonia or Consolidation, while 40-60% have Pleural Effusion. Pass pos_weight to BCEWithLogitsLoss to upweight the rare positive class: a pos_weight of 10 makes the model pay 10× more attention to positive examples of that pathology.

Replace the ResNet-50 classification head with a linear layer that outputs 14 logits (one per pathology), rather than the default 1000 for ImageNet. We do not apply sigmoid in the forward pass — the BCEWithLogitsLoss applies it internally for numerical stability. At inference time, apply sigmoid manually to get probabilities.

Adding a dropout layer before the final linear layer is especially important with scarce data, as regularisation prevents the small head from overfitting. A dropout probability of 0.3-0.5 is typical for medical imaging fine-tuning.

Accuracy is useless for imbalanced medical datasets. If only 5% of patients have Pneumonia, a model that always predicts 'no pneumonia' achieves 95% accuracy while being completely useless clinically. AUC-ROC (Area Under the ROC Curve) measures discrimination ability across all thresholds.

We compute AUC-ROC for each of the 14 pathologies separately and report the mean AUC across all labels. AUC of 0.5 is random chance; 0.7 is acceptable; 0.85+ is clinical grade; 0.9+ often approaches radiologist-level performance. scikit-learn's roc_auc_score computes this efficiently.

When you have only hundreds or a few thousand labelled medical images, several techniques help maximally exploit the available data. Strong data augmentation is the most important: random flips, rotations, contrast and brightness jitter all help, while keeping augmentations clinically plausible (a chest X-ray should not be vertically flipped — that would never occur in clinical practice).

Progressive resizing (training initially at lower resolution, then increasing) is another effective technique. Start at 128×128 to quickly iterate, then fine-tune at 224×224 or even 320×320 for final accuracy. This is much faster than always training at full resolution and often matches full-resolution accuracy.

When ImageNet features do not transfer well enough, self-supervised pre-training on unlabelled medical images is a powerful alternative. Methods like SimCLR, MoCo, and DINO learn representations by training the model to recognise that two augmented versions of the same image are similar, without any labels.

The workflow: (1) pre-train on large unlabelled medical datasets (CheXpert has 224K unlabelled images), (2) fine-tune with the small labelled set. This consistently outperforms ImageNet transfer on medical tasks because the model learns features specific to the medical domain rather than general object recognition features.

In medical decision support, knowing how confident a model is is as important as the prediction itself. A model that says 'Pneumonia: 95% probability' should be more trusted than one saying '52% probability'. Standard softmax probabilities are often overconfident and do not represent true uncertainty.

Monte Carlo Dropout (MC Dropout) approximates Bayesian uncertainty by keeping dropout active at inference time and running forward pass multiple times. The variance of predictions across runs estimates uncertainty. Predictions with high variance should be flagged for human review rather than acted upon automatically.

Deploying ML models in medical settings carries serious ethical responsibilities. Dataset bias is a critical concern: models trained predominantly on images from one hospital's scanners, patient demographics, or scanner manufacturers may fail on images from different settings — a known problem in radiology AI.

Before deployment, always evaluate model performance across demographic subgroups (age, sex, race) and across different scanning equipment. Regulatory frameworks like the FDA's AI/ML-based Software as a Medical Device (SaMD) guidelines in the US and the EU Medical Device Regulation (MDR) require rigorous clinical validation before deployment. Models should assist radiologists, not replace them — especially on high-stakes pathologies where errors have life-or-death consequences.

Test your understanding of domain adaptation for medical imaging from this lesson.

In this lesson you learned: medical imaging transfer learning applies ImageNet pre-trained models to domains with grayscale images and scarce labels by converting channels and replacing the classification head, multi-label BCE loss with pos_weight handles the extreme class imbalance typical of pathology datasets, and AUC-ROC is the correct evaluation metric that measures discrimination ability independent of class balance. With these transfer learning skills mastered, you are ready to explore NLP with BERT in the next course.