Dropout Regularisation to Prevent Overfitting

Dropout, introduced by Srivastava et al. in 2014, is a regularisation technique that randomly sets a fraction of neuron activations to zero during each training step. By forcing the network to operate with a random subset of neurons, it prevents any single neuron from becoming too specialised and forces the network to learn redundant representations. This reduces overfitting significantly in large fully connected networks.

PyTorch implements inverted dropout: during training, surviving activations are scaled up by 1/(1-p) to compensate for the zeroed neurons. This means the expected sum of activations stays the same regardless of the dropout rate. The advantage is that at inference you disable dropout and use the network as-is — no need to scale outputs. This is why Dropout in eval mode is simply an identity function.

Dropout layers are typically placed after activation functions in fully connected networks, or after pooling in CNNs. Common dropout probabilities are p=0.5 for large hidden layers and p=0.2-0.3 for smaller layers or CNNs. Dropout is usually not applied to the input layer or the final output layer. The example below shows a deep MLP with Dropout between hidden layers.

Forgetting to switch between training and evaluation mode is one of the most common and hard-to-debug mistakes. In training mode (model.train()), Dropout randomly zeros activations. In eval mode (model.eval()), Dropout is completely disabled and all neurons are active. Failing to call model.eval() before evaluation means the validation loss will be artificially higher and non-deterministic, making it impossible to reliably compare runs.

To see Dropout's regularisation effect, compare training and validation loss curves with and without it. Without Dropout, a large network will typically show the classic overfitting pattern: training loss near zero while validation loss remains high. With Dropout, both curves track each other more closely. The gap between training and validation accuracy (the generalisation gap) is the key metric Dropout reduces.

The dropout probability p is a hyperparameter you need to tune. p=0.5 is the original default from the Dropout paper and works well for very large layers. For convolutional layers, p=0.1 to 0.25 is typical because spatial features are more structured. For Transformers, p=0.1 is standard. If your model is not overfitting, reduce p or remove Dropout. If it is strongly overfitting, increase p incrementally and monitor validation performance.

An advanced use of Dropout is MC Dropout for uncertainty estimation. Instead of disabling Dropout at inference, keep it enabled and run the model N times on the same input. The mean of the N predictions is the estimate; the variance across predictions measures model uncertainty. This approximates Bayesian inference without a full Bayesian neural network. It is used in safety-critical applications like medical diagnosis where knowing 'I don't know' is valuable.

nn.Dropout2d (Spatial Dropout) zeros entire channels rather than individual activations in convolutional feature maps. This is more effective for CNNs because adjacent pixels in the same channel are highly correlated — dropping individual pixels has little effect. By dropping entire channels, the network cannot rely on any single filter and must distribute information across many channels. Use Dropout2d after Conv2d layers in place of regular Dropout.

Dropout and weight decay (L2 regularisation) are complementary regularisers — use them together. Dropout prevents co-adaptation between neurons; weight decay penalises large weights, shrinking all parameters towards zero. The combination is stronger than either alone. In AdamW, weight decay is applied correctly as a separate term, not mixed into the gradient update like in Adam. Most production models use both techniques.

Use the training vs validation loss plot to diagnose whether you need more regularisation. The validation loss increasing while training loss decreases is the signature of overfitting. Systematically increase dropout probability or add Dropout layers where missing. Other signals of severe overfitting include training accuracy 98%+ while validation accuracy stagnates at 70-80%, or very high variance in validation metrics across different random seeds.

To use Dropout correctly: add nn.Dropout(p) after hidden layer activations; call model.train() during training to enable it and model.eval() during evaluation to disable it; start with p=0.5 for large FC layers and reduce if the model underfits. Pair with weight decay via AdamW for best results. Always track both training and validation metrics to confirm Dropout is actually helping — sometimes it hurts convergence speed without improving generalisation when the model is already well-regularised.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: Dropout randomly zeros neuron activations during training to prevent co-adaptation and reduce overfitting, inverted dropout scales surviving values by 1/(1-p) so inference requires no adjustment, and model.eval() disables Dropout for deterministic and correct inference. Next up we explore weight initialisation strategies that prevent vanishing and exploding gradients in deep networks.