0Pricing
Machine Learning Academy · Lesson

Batch Normalisation: Stable and Faster Training

Learners will insert nn.BatchNorm1d between layers, observe faster convergence on a deep network, and understand how BatchNorm normalises activations within each mini-batch.

Batch Normalisation: Stable and Faster Training is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Problem Batch Norm Solves

Deep neural networks suffer from internal covariate shift — the distribution of each layer's inputs changes during training as the previous layer's weights update. This forces each layer to continuously adapt to a shifting input distribution, slowing training. Batch Normalisation (Batch Norm), introduced by Ioffe and Szegedy in 2015, addresses this by normalising layer inputs within each mini-batch, dramatically accelerating training and reducing sensitivity to weight initialisation.

# Without batch norm: deep networks train slowly and
# require very careful weight init and LR tuning.

# With batch norm: can use higher learning rates,
# less sensitive to initialisation, acts as regulariser.

# Batch norm normalises each feature to:
# mean=0, std=1 within the batch, then
# applies learnable scale (gamma) and shift (beta).
print('Batch Norm: normalize -> scale -> shift')

How Batch Norm Works Mathematically

For each feature dimension, Batch Norm computes the mean and variance across the current mini-batch, then normalises each value. After normalisation it applies two learnable parameters: gamma (scale) and beta (shift). This allows the network to undo the normalisation if needed — the identity transform is recoverable. A small constant epsilon is added to the variance to prevent division by zero.

import torch

def batch_norm_manual(x, gamma, beta, eps=1e-5):
    # x shape: (batch_size, features)
    mu = x.mean(dim=0)           # mean per feature
    var = x.var(dim=0, unbiased=False)  # var per feature
    x_norm = (x - mu) / (var + eps).sqrt()
    return gamma * x_norm + beta  # scale and shift

x = torch.randn(32, 8)  # batch=32, 8 features
gamma = torch.ones(8)
beta = torch.zeros(8)

out = batch_norm_manual(x, gamma, beta)
print('Mean near 0:', out.mean(dim=0).abs().max().item() < 0.01)
print('Std near 1:', (out.std(dim=0) - 1).abs().max().item() < 0.01)

nn.BatchNorm1d for Fully Connected Layers

nn.BatchNorm1d is used after linear layers in feedforward networks. It takes num_features (the size of the previous layer's output) as its argument. Batch Norm is typically placed after the linear layer but before the activation function, though there is ongoing debate about whether before or after activation is better. The module maintains running mean and variance for use during inference.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(16, 64),
    nn.BatchNorm1d(64),   # after linear, before activation
    nn.ReLU(),
    nn.Linear(64, 32),
    nn.BatchNorm1d(32),
    nn.ReLU(),
    nn.Linear(32, 10)
)

x = torch.randn(32, 16)   # batch of 32
out = model(x)
print(out.shape)           # torch.Size([32, 10])

Training vs Inference Behaviour

Batch Norm behaves differently in training and inference. During training it uses the mini-batch statistics (mean and variance of the current batch). During inference it uses running statistics (exponential moving average accumulated during training) so that predictions are deterministic and independent of batch size. This is why you must call model.eval() during inference — it switches BatchNorm to use the running statistics.

import torch
import torch.nn as nn

bn = nn.BatchNorm1d(4)

# Training mode: uses batch statistics, updates running stats
bn.train()
x = torch.randn(8, 4)
out_train = bn(x)

# After training, running_mean and running_var are populated
print('Running mean:', bn.running_mean)

# Eval mode: uses running statistics (deterministic)
bn.eval()
x_new = torch.randn(1, 4)   # single sample -- works!
out_eval = bn(x_new)
print(out_eval.shape)         # torch.Size([1, 4])

Learnable Parameters: gamma and beta

Batch Norm has two learnable parameters per feature: weight (gamma, initialised to 1) and bias (beta, initialised to 0). These allow the network to learn to rescale and reshift the normalised values if that improves the task. They are updated by the optimizer just like regular weight matrices. If you want to freeze Batch Norm layers during fine-tuning, set requires_grad=False on these parameters.

import torch.nn as nn

bn = nn.BatchNorm1d(8)
print('gamma (weight):', bn.weight.data)   # all 1s
print('beta (bias):', bn.bias.data)         # all 0s
print('gamma requires_grad:', bn.weight.requires_grad)  # True
print('beta requires_grad:', bn.bias.requires_grad)     # True

# Total trainable params in this BN layer:
# 2 * 8 = 16 (gamma and beta for 8 features)
params = sum(p.numel() for p in bn.parameters())
print('Params:', params)   # 16

Observing Faster Convergence

One of the clearest benefits of Batch Norm is faster convergence. Networks with Batch Norm typically reach the same validation accuracy in fewer epochs and can use larger learning rates without instability. The normalisation keeps activations in a healthy range throughout training, preventing the saturation that slows learning in networks using sigmoid or tanh activations. The effect is most pronounced in deep networks with many layers.

import torch
import torch.nn as nn
import torch.optim as optim

def make_model(use_bn):
    layers = [nn.Linear(16, 64)]
    if use_bn: layers.append(nn.BatchNorm1d(64))
    layers.append(nn.ReLU())
    layers.append(nn.Linear(64, 2))
    return nn.Sequential(*layers)

X = torch.randn(200, 16)
y = torch.randint(0, 2, (200,))

for use_bn in [False, True]:
    model = make_model(use_bn)
    opt = optim.SGD(model.parameters(), lr=0.1)
    crit = nn.CrossEntropyLoss()
    for _ in range(20):
        opt.zero_grad(); loss = crit(model(X), y)
        loss.backward(); opt.step()
    print(f'BN={use_bn}: final_loss={loss.item():.4f}')

Batch Norm as a Regulariser

Batch Norm acts as a mild regulariser because each training sample is normalised with respect to the other samples in the mini-batch — introducing stochasticity similar to Dropout. This means networks with Batch Norm often need less Dropout. The regularisation effect diminishes with larger batch sizes because the batch statistics become more deterministic, approaching the true population statistics and removing the stochastic element.

# Key insight: batch norm introduces noise proportional to
# 1/sqrt(batch_size) because batch statistics are noisy
# estimates of population statistics.

# Small batch (e.g., 8): high noise -> more regularisation
# Large batch (e.g., 512): low noise -> less regularisation

# Common pattern: use batch norm AND a small dropout
# for strong regularisation in deep networks
model = __import__('torch').nn.Sequential(
    __import__('torch').nn.Linear(32, 128),
    __import__('torch').nn.BatchNorm1d(128),
    __import__('torch').nn.ReLU(),
    __import__('torch').nn.Dropout(0.2)  # mild dropout
)
print('BN + light Dropout: balanced regularisation')

nn.BatchNorm2d for Convolutional Networks

In convolutional networks, nn.BatchNorm2d normalises across the batch and spatial dimensions for each channel independently. It takes num_channels as its argument (matching the number of output channels from the preceding Conv2d layer). The standard pattern is Conv2d -> BatchNorm2d -> ReLU, which is used in nearly every modern CNN architecture including ResNet, VGG, and EfficientNet.

import torch
import torch.nn as nn

# Standard CNN block: Conv -> BN -> ReLU
conv_block = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3, padding=1),
    nn.BatchNorm2d(64),    # 64 = number of output channels
    nn.ReLU(inplace=True)
)

# Input: batch of 8 RGB images, 32x32 pixels
x = torch.randn(8, 3, 32, 32)
out = conv_block(x)
print(out.shape)   # torch.Size([8, 64, 32, 32])

Layer Norm vs Batch Norm

Layer Normalisation (used in Transformers) normalises across features within a single sample, rather than across the batch. This makes it independent of batch size, which is essential for variable-length sequences and small batches. Batch Norm normalises across the batch for each feature — ideal for CNNs and large-batch training. The wrong choice can hurt: using Batch Norm in a Transformer or Layer Norm in a CNN is a common architecture mistake.

import torch
import torch.nn as nn

x = torch.randn(4, 8)  # batch=4, features=8

# Batch Norm: normalise across batch for each feature
bn = nn.BatchNorm1d(8)
bn_out = bn(x)   # statistics computed over 4 samples

# Layer Norm: normalise across features for each sample
ln = nn.LayerNorm(8)
ln_out = ln(x)   # statistics computed over 8 features

print('BN output shape:', bn_out.shape)   # (4, 8)
print('LN output shape:', ln_out.shape)   # (4, 8)
# Same shape, different normalisation axes

Freezing Batch Norm During Fine-Tuning

When fine-tuning a pre-trained model on a small dataset, the running statistics in Batch Norm layers were estimated on the original large dataset. Allowing them to update on a small fine-tuning batch can corrupt them and hurt performance. A common strategy is to freeze Batch Norm layers by putting them permanently in eval mode. In PyTorch this is done by calling model.apply with a custom function that freezes each BN layer.

import torch.nn as nn

def freeze_bn(module):
    '''Keep BN in eval mode during fine-tuning.'''
    if isinstance(module, (nn.BatchNorm1d,
                           nn.BatchNorm2d,
                           nn.BatchNorm3d)):
        module.eval()  # use running stats, not batch stats
        module.weight.requires_grad_(False)
        module.bias.requires_grad_(False)

model = nn.Sequential(
    nn.Linear(4, 8),
    nn.BatchNorm1d(8),
    nn.ReLU()
)
model.apply(freeze_bn)
print('BN frozen for fine-tuning')

Batch Norm Limitations and Alternatives

Batch Norm has known limitations: it requires a minimum batch size (usually 16+) for reliable statistics; it is ineffective or harmful on very small batches; it introduces data dependency between samples in a batch, complicating parallelisation. Alternatives include Group Norm (divide channels into groups), Instance Norm (normalise per sample and channel, used in style transfer), and Layer Norm (used in Transformers). Choosing the right normalisation is architecture-dependent.

import torch
import torch.nn as nn

x = torch.randn(4, 16, 10)  # (batch, channels, seq_len)

# GroupNorm: 4 groups of 4 channels each
gn = nn.GroupNorm(num_groups=4, num_channels=16)
print('GroupNorm:', gn(x).shape)

# InstanceNorm: normalise each sample+channel independently
ins = nn.InstanceNorm1d(16)
print('InstanceNorm:', ins(x).shape)

# LayerNorm: normalise across last N dimensions
ln = nn.LayerNorm([16, 10])
print('LayerNorm:', ln(x).shape)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Batch Normalisation normalises layer inputs within each mini-batch to stabilise and accelerate training, nn.BatchNorm1d and nn.BatchNorm2d are used in fully connected and convolutional layers respectively, and model.eval() switches BN to use running statistics accumulated during training for deterministic inference. Next up we add Dropout regularisation to prevent overfitting.

Frequently asked questions

Is the “Batch Normalisation: Stable and Faster Training” lesson free?

Yes — the full text of “Batch Normalisation: Stable and Faster Training” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Batch Normalisation: Stable and Faster Training”?

Learners will insert nn.BatchNorm1d between layers, observe faster convergence on a deep network, and understand how BatchNorm normalises activations within each mini-batch. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Batch Normalisation: Stable and Faster Training” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Learning Rate: The Most Important Hyperparameter
  2. Batch Normalisation: Stable and Faster Training
  3. Dropout Regularisation to Prevent Overfitting
  4. Weight Initialisation: Xavier and He Initialisation
← Back to Machine Learning Academy