PricingLoginStart free
Machine Learning Academy · Leçon

Normalisation par lots : un entraînement stable et plus rapide

Vous insérerez nn.BatchNorm1d entre les couches, observerez une convergence plus rapide sur un réseau profond et comprendrez comment BatchNorm normalise les activations au sein de chaque mini-lot.

Leçon 2 sur 413 étapes

Normalisation par lots : un entraînement stable et plus rapide est une leçon Machine Learning Academy gratuite sur CoddyKit. Ceci est la leçon 2 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage Machine Learning Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours Machine Learning Academy comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

The Problem Batch Norm Solves

Deep neural networks suffer from internal covariate shift — the distribution of each layer's inputs changes during training as the previous layer's weights update. This forces each layer to continuously adapt to a shifting input distribution, slowing training. Batch Normalisation (Batch Norm), introduced by Ioffe and Szegedy in 2015, addresses this by normalising layer inputs within each mini-batch, dramatically accelerating training and reducing sensitivity to weight initialisation.

# Without batch norm: deep networks train slowly and
# require very careful weight init and LR tuning.

# With batch norm: can use higher learning rates,
# less sensitive to initialisation, acts as regulariser.

# Batch norm normalises each feature to:
# mean=0, std=1 within the batch, then
# applies learnable scale (gamma) and shift (beta).
print('Batch Norm: normalize -> scale -> shift')

How Batch Norm Works Mathematically

For each feature dimension, Batch Norm computes the mean and variance across the current mini-batch, then normalises each value. After normalisation it applies two learnable parameters: gamma (scale) and beta (shift). This allows the network to undo the normalisation if needed — the identity transform is recoverable. A small constant epsilon is added to the variance to prevent division by zero.

import torch

def batch_norm_manual(x, gamma, beta, eps=1e-5):
    # x shape: (batch_size, features)
    mu = x.mean(dim=0)           # mean per feature
    var = x.var(dim=0, unbiased=False)  # var per feature
    x_norm = (x - mu) / (var + eps).sqrt()
    return gamma * x_norm + beta  # scale and shift

x = torch.randn(32, 8)  # batch=32, 8 features
gamma = torch.ones(8)
beta = torch.zeros(8)

out = batch_norm_manual(x, gamma, beta)
print('Mean near 0:', out.mean(dim=0).abs().max().item() < 0.01)
print('Std near 1:', (out.std(dim=0) - 1).abs().max().item() < 0.01)

nn.BatchNorm1d for Fully Connected Layers

nn.BatchNorm1d is used after linear layers in feedforward networks. It takes num_features (the size of the previous layer's output) as its argument. Batch Norm is typically placed after the linear layer but before the activation function, though there is ongoing debate about whether before or after activation is better. The module maintains running mean and variance for use during inference.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(16, 64),
    nn.BatchNorm1d(64),   # after linear, before activation
    nn.ReLU(),
    nn.Linear(64, 32),
    nn.BatchNorm1d(32),
    nn.ReLU(),
    nn.Linear(32, 10)
)

x = torch.randn(32, 16)   # batch of 32
out = model(x)
print(out.shape)           # torch.Size([32, 10])

Training vs Inference Behaviour

Batch Norm behaves differently in training and inference. During training it uses the mini-batch statistics (mean and variance of the current batch). During inference it uses running statistics (exponential moving average accumulated during training) so that predictions are deterministic and independent of batch size. This is why you must call model.eval() during inference — it switches BatchNorm to use the running statistics.

import torch
import torch.nn as nn

bn = nn.BatchNorm1d(4)

# Training mode: uses batch statistics, updates running stats
bn.train()
x = torch.randn(8, 4)
out_train = bn(x)

# After training, running_mean and running_var are populated
print('Running mean:', bn.running_mean)

# Eval mode: uses running statistics (deterministic)
bn.eval()
x_new = torch.randn(1, 4)   # single sample -- works!
out_eval = bn(x_new)
print(out_eval.shape)         # torch.Size([1, 4])

Learnable Parameters: gamma and beta

Batch Norm has two learnable parameters per feature: weight (gamma, initialised to 1) and bias (beta, initialised to 0). These allow the network to learn to rescale and reshift the normalised values if that improves the task. They are updated by the optimizer just like regular weight matrices. If you want to freeze Batch Norm layers during fine-tuning, set requires_grad=False on these parameters.

import torch.nn as nn

bn = nn.BatchNorm1d(8)
print('gamma (weight):', bn.weight.data)   # all 1s
print('beta (bias):', bn.bias.data)         # all 0s
print('gamma requires_grad:', bn.weight.requires_grad)  # True
print('beta requires_grad:', bn.bias.requires_grad)     # True

# Total trainable params in this BN layer:
# 2 * 8 = 16 (gamma and beta for 8 features)
params = sum(p.numel() for p in bn.parameters())
print('Params:', params)   # 16

Observing Faster Convergence

One of the clearest benefits of Batch Norm is faster convergence. Networks with Batch Norm typically reach the same validation accuracy in fewer epochs and can use larger learning rates without instability. The normalisation keeps activations in a healthy range throughout training, preventing the saturation that slows learning in networks using sigmoid or tanh activations. The effect is most pronounced in deep networks with many layers.

import torch
import torch.nn as nn
import torch.optim as optim

def make_model(use_bn):
    layers = [nn.Linear(16, 64)]
    if use_bn: layers.append(nn.BatchNorm1d(64))
    layers.append(nn.ReLU())
    layers.append(nn.Linear(64, 2))
    return nn.Sequential(*layers)

X = torch.randn(200, 16)
y = torch.randint(0, 2, (200,))

for use_bn in [False, True]:
    model = make_model(use_bn)
    opt = optim.SGD(model.parameters(), lr=0.1)
    crit = nn.CrossEntropyLoss()
    for _ in range(20):
        opt.zero_grad(); loss = crit(model(X), y)
        loss.backward(); opt.step()
    print(f'BN={use_bn}: final_loss={loss.item():.4f}')

Batch Norm as a Regulariser

Batch Norm acts as a mild regulariser because each training sample is normalised with respect to the other samples in the mini-batch — introducing stochasticity similar to Dropout. This means networks with Batch Norm often need less Dropout. The regularisation effect diminishes with larger batch sizes because the batch statistics become more deterministic, approaching the true population statistics and removing the stochastic element.

# Key insight: batch norm introduces noise proportional to
# 1/sqrt(batch_size) because batch statistics are noisy
# estimates of population statistics.

# Small batch (e.g., 8): high noise -> more regularisation
# Large batch (e.g., 512): low noise -> less regularisation

# Common pattern: use batch norm AND a small dropout
# for strong regularisation in deep networks
model = __import__('torch').nn.Sequential(
    __import__('torch').nn.Linear(32, 128),
    __import__('torch').nn.BatchNorm1d(128),
    __import__('torch').nn.ReLU(),
    __import__('torch').nn.Dropout(0.2)  # mild dropout
)
print('BN + light Dropout: balanced regularisation')

nn.BatchNorm2d for Convolutional Networks

In convolutional networks, nn.BatchNorm2d normalises across the batch and spatial dimensions for each channel independently. It takes num_channels as its argument (matching the number of output channels from the preceding Conv2d layer). The standard pattern is Conv2d -> BatchNorm2d -> ReLU, which is used in nearly every modern CNN architecture including ResNet, VGG, and EfficientNet.

import torch
import torch.nn as nn

# Standard CNN block: Conv -> BN -> ReLU
conv_block = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3, padding=1),
    nn.BatchNorm2d(64),    # 64 = number of output channels
    nn.ReLU(inplace=True)
)

# Input: batch of 8 RGB images, 32x32 pixels
x = torch.randn(8, 3, 32, 32)
out = conv_block(x)
print(out.shape)   # torch.Size([8, 64, 32, 32])

Layer Norm vs Batch Norm

Layer Normalisation (used in Transformers) normalises across features within a single sample, rather than across the batch. This makes it independent of batch size, which is essential for variable-length sequences and small batches. Batch Norm normalises across the batch for each feature — ideal for CNNs and large-batch training. The wrong choice can hurt: using Batch Norm in a Transformer or Layer Norm in a CNN is a common architecture mistake.

import torch
import torch.nn as nn

x = torch.randn(4, 8)  # batch=4, features=8

# Batch Norm: normalise across batch for each feature
bn = nn.BatchNorm1d(8)
bn_out = bn(x)   # statistics computed over 4 samples

# Layer Norm: normalise across features for each sample
ln = nn.LayerNorm(8)
ln_out = ln(x)   # statistics computed over 8 features

print('BN output shape:', bn_out.shape)   # (4, 8)
print('LN output shape:', ln_out.shape)   # (4, 8)
# Same shape, different normalisation axes

Freezing Batch Norm During Fine-Tuning

When fine-tuning a pre-trained model on a small dataset, the running statistics in Batch Norm layers were estimated on the original large dataset. Allowing them to update on a small fine-tuning batch can corrupt them and hurt performance. A common strategy is to freeze Batch Norm layers by putting them permanently in eval mode. In PyTorch this is done by calling model.apply with a custom function that freezes each BN layer.

import torch.nn as nn

def freeze_bn(module):
    '''Keep BN in eval mode during fine-tuning.'''
    if isinstance(module, (nn.BatchNorm1d,
                           nn.BatchNorm2d,
                           nn.BatchNorm3d)):
        module.eval()  # use running stats, not batch stats
        module.weight.requires_grad_(False)
        module.bias.requires_grad_(False)

model = nn.Sequential(
    nn.Linear(4, 8),
    nn.BatchNorm1d(8),
    nn.ReLU()
)
model.apply(freeze_bn)
print('BN frozen for fine-tuning')

Batch Norm Limitations and Alternatives

Batch Norm has known limitations: it requires a minimum batch size (usually 16+) for reliable statistics; it is ineffective or harmful on very small batches; it introduces data dependency between samples in a batch, complicating parallelisation. Alternatives include Group Norm (divide channels into groups), Instance Norm (normalise per sample and channel, used in style transfer), and Layer Norm (used in Transformers). Choosing the right normalisation is architecture-dependent.

import torch
import torch.nn as nn

x = torch.randn(4, 16, 10)  # (batch, channels, seq_len)

# GroupNorm: 4 groups of 4 channels each
gn = nn.GroupNorm(num_groups=4, num_channels=16)
print('GroupNorm:', gn(x).shape)

# InstanceNorm: normalise each sample+channel independently
ins = nn.InstanceNorm1d(16)
print('InstanceNorm:', ins(x).shape)

# LayerNorm: normalise across last N dimensions
ln = nn.LayerNorm([16, 10])
print('LayerNorm:', ln(x).shape)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Réponds-y dans la leçon interactive

Lesson Recap

In this lesson you learned: Batch Normalisation normalises layer inputs within each mini-batch to stabilise and accelerate training, nn.BatchNorm1d and nn.BatchNorm2d are used in fully connected and convolutional layers respectively, and model.eval() switches BN to use running statistics accumulated during training for deterministic inference. Next up we add Dropout regularisation to prevent overfitting.

Gratuit pour commencer

Apprends Python avec un tuteur IA — gratuit

Écris et exécute du vrai code dans ton navigateur, obtiens de l'aide instantanée d'un tuteur IA disponible 24h/24, et reprends là où tu t'es arrêté sur le web ou dans l'app.

Cours
30
Leçons
120
Commencer gratuitementVoir le cours complet

Questions Fréquemment Posées

La leçon « Normalisation par lots : un entraînement stable et plus rapide » est-elle gratuite ?

Oui — le texte complet de « Normalisation par lots : un entraînement stable et plus rapide » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours Machine Learning Academy, passe à CoddyKit PRO. Le cours Machine Learning Academy comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Normalisation par lots : un entraînement stable et plus rapide » ?

Vous insérerez nn.BatchNorm1d entre les couches, observerez une convergence plus rapide sur un réseau profond et comprendrez comment BatchNorm normalise les activations au sein de chaque mini-lot. Tu pratiques Machine Learning Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer Machine Learning Academy ?

Aucune expérience préalable n'est requise. Machine Learning Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 2 sur 4.

Combien de temps prend la leçon « Normalisation par lots : un entraînement stable et plus rapide » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon Machine Learning Academy ?

Oui. Chaque leçon Machine Learning Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

  1. Taux d’apprentissage : l’hyperparamètre le plus important
  2. Normalisation par lots : un entraînement stable et plus rapide
  3. Régularisation par abandon pour éviter le surapprentissage
  4. Initialisation des poids : initialisations de Xavier et de He
← Retour à Machine Learning Academy