0Pricing
Machine Learning Academy · Lekcja

Normalizacja wsadowa: stabilniejsze i szybsze trenowanie

Uczą się Państwo wstawiać nn.BatchNorm1d między warstwy, obserwować szybszą zbieżność głębokiej sieci oraz rozumieć, jak BatchNorm normalizuje aktywacje w każdym mini-wsadowym zbiorze.

Normalizacja wsadowa: stabilniejsze i szybsze trenowanie to bezpłatna lekcja Machine Learning Academy na CoddyKit. To lekcja 2 z 4. Możesz przeczytać całą lekcję poniżej za darmo — a potem ćwiczyć ją interaktywnie w przeglądarce z wbudowanym edytorem kodu i tutorem AI dostępnym 24/7. To część ścieżki edukacyjnej Machine Learning Academy, a Twój postęp synchronizuje się między webem a aplikacją CoddyKit. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.

Części tej lekcji nie zostały jeszcze przetłumaczone i są wyświetlane po angielsku.

The Problem Batch Norm Solves

Deep neural networks suffer from internal covariate shift — the distribution of each layer's inputs changes during training as the previous layer's weights update. This forces each layer to continuously adapt to a shifting input distribution, slowing training. Batch Normalisation (Batch Norm), introduced by Ioffe and Szegedy in 2015, addresses this by normalising layer inputs within each mini-batch, dramatically accelerating training and reducing sensitivity to weight initialisation.

# Without batch norm: deep networks train slowly and
# require very careful weight init and LR tuning.

# With batch norm: can use higher learning rates,
# less sensitive to initialisation, acts as regulariser.

# Batch norm normalises each feature to:
# mean=0, std=1 within the batch, then
# applies learnable scale (gamma) and shift (beta).
print('Batch Norm: normalize -> scale -> shift')

How Batch Norm Works Mathematically

For each feature dimension, Batch Norm computes the mean and variance across the current mini-batch, then normalises each value. After normalisation it applies two learnable parameters: gamma (scale) and beta (shift). This allows the network to undo the normalisation if needed — the identity transform is recoverable. A small constant epsilon is added to the variance to prevent division by zero.

import torch

def batch_norm_manual(x, gamma, beta, eps=1e-5):
    # x shape: (batch_size, features)
    mu = x.mean(dim=0)           # mean per feature
    var = x.var(dim=0, unbiased=False)  # var per feature
    x_norm = (x - mu) / (var + eps).sqrt()
    return gamma * x_norm + beta  # scale and shift

x = torch.randn(32, 8)  # batch=32, 8 features
gamma = torch.ones(8)
beta = torch.zeros(8)

out = batch_norm_manual(x, gamma, beta)
print('Mean near 0:', out.mean(dim=0).abs().max().item() < 0.01)
print('Std near 1:', (out.std(dim=0) - 1).abs().max().item() < 0.01)

nn.BatchNorm1d for Fully Connected Layers

nn.BatchNorm1d is used after linear layers in feedforward networks. It takes num_features (the size of the previous layer's output) as its argument. Batch Norm is typically placed after the linear layer but before the activation function, though there is ongoing debate about whether before or after activation is better. The module maintains running mean and variance for use during inference.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(16, 64),
    nn.BatchNorm1d(64),   # after linear, before activation
    nn.ReLU(),
    nn.Linear(64, 32),
    nn.BatchNorm1d(32),
    nn.ReLU(),
    nn.Linear(32, 10)
)

x = torch.randn(32, 16)   # batch of 32
out = model(x)
print(out.shape)           # torch.Size([32, 10])

Training vs Inference Behaviour

Batch Norm behaves differently in training and inference. During training it uses the mini-batch statistics (mean and variance of the current batch). During inference it uses running statistics (exponential moving average accumulated during training) so that predictions are deterministic and independent of batch size. This is why you must call model.eval() during inference — it switches BatchNorm to use the running statistics.

import torch
import torch.nn as nn

bn = nn.BatchNorm1d(4)

# Training mode: uses batch statistics, updates running stats
bn.train()
x = torch.randn(8, 4)
out_train = bn(x)

# After training, running_mean and running_var are populated
print('Running mean:', bn.running_mean)

# Eval mode: uses running statistics (deterministic)
bn.eval()
x_new = torch.randn(1, 4)   # single sample -- works!
out_eval = bn(x_new)
print(out_eval.shape)         # torch.Size([1, 4])

Learnable Parameters: gamma and beta

Batch Norm has two learnable parameters per feature: weight (gamma, initialised to 1) and bias (beta, initialised to 0). These allow the network to learn to rescale and reshift the normalised values if that improves the task. They are updated by the optimizer just like regular weight matrices. If you want to freeze Batch Norm layers during fine-tuning, set requires_grad=False on these parameters.

import torch.nn as nn

bn = nn.BatchNorm1d(8)
print('gamma (weight):', bn.weight.data)   # all 1s
print('beta (bias):', bn.bias.data)         # all 0s
print('gamma requires_grad:', bn.weight.requires_grad)  # True
print('beta requires_grad:', bn.bias.requires_grad)     # True

# Total trainable params in this BN layer:
# 2 * 8 = 16 (gamma and beta for 8 features)
params = sum(p.numel() for p in bn.parameters())
print('Params:', params)   # 16

Observing Faster Convergence

One of the clearest benefits of Batch Norm is faster convergence. Networks with Batch Norm typically reach the same validation accuracy in fewer epochs and can use larger learning rates without instability. The normalisation keeps activations in a healthy range throughout training, preventing the saturation that slows learning in networks using sigmoid or tanh activations. The effect is most pronounced in deep networks with many layers.

import torch
import torch.nn as nn
import torch.optim as optim

def make_model(use_bn):
    layers = [nn.Linear(16, 64)]
    if use_bn: layers.append(nn.BatchNorm1d(64))
    layers.append(nn.ReLU())
    layers.append(nn.Linear(64, 2))
    return nn.Sequential(*layers)

X = torch.randn(200, 16)
y = torch.randint(0, 2, (200,))

for use_bn in [False, True]:
    model = make_model(use_bn)
    opt = optim.SGD(model.parameters(), lr=0.1)
    crit = nn.CrossEntropyLoss()
    for _ in range(20):
        opt.zero_grad(); loss = crit(model(X), y)
        loss.backward(); opt.step()
    print(f'BN={use_bn}: final_loss={loss.item():.4f}')

Batch Norm as a Regulariser

Batch Norm acts as a mild regulariser because each training sample is normalised with respect to the other samples in the mini-batch — introducing stochasticity similar to Dropout. This means networks with Batch Norm often need less Dropout. The regularisation effect diminishes with larger batch sizes because the batch statistics become more deterministic, approaching the true population statistics and removing the stochastic element.

# Key insight: batch norm introduces noise proportional to
# 1/sqrt(batch_size) because batch statistics are noisy
# estimates of population statistics.

# Small batch (e.g., 8): high noise -> more regularisation
# Large batch (e.g., 512): low noise -> less regularisation

# Common pattern: use batch norm AND a small dropout
# for strong regularisation in deep networks
model = __import__('torch').nn.Sequential(
    __import__('torch').nn.Linear(32, 128),
    __import__('torch').nn.BatchNorm1d(128),
    __import__('torch').nn.ReLU(),
    __import__('torch').nn.Dropout(0.2)  # mild dropout
)
print('BN + light Dropout: balanced regularisation')

nn.BatchNorm2d for Convolutional Networks

In convolutional networks, nn.BatchNorm2d normalises across the batch and spatial dimensions for each channel independently. It takes num_channels as its argument (matching the number of output channels from the preceding Conv2d layer). The standard pattern is Conv2d -> BatchNorm2d -> ReLU, which is used in nearly every modern CNN architecture including ResNet, VGG, and EfficientNet.

import torch
import torch.nn as nn

# Standard CNN block: Conv -> BN -> ReLU
conv_block = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3, padding=1),
    nn.BatchNorm2d(64),    # 64 = number of output channels
    nn.ReLU(inplace=True)
)

# Input: batch of 8 RGB images, 32x32 pixels
x = torch.randn(8, 3, 32, 32)
out = conv_block(x)
print(out.shape)   # torch.Size([8, 64, 32, 32])

Layer Norm vs Batch Norm

Layer Normalisation (used in Transformers) normalises across features within a single sample, rather than across the batch. This makes it independent of batch size, which is essential for variable-length sequences and small batches. Batch Norm normalises across the batch for each feature — ideal for CNNs and large-batch training. The wrong choice can hurt: using Batch Norm in a Transformer or Layer Norm in a CNN is a common architecture mistake.

import torch
import torch.nn as nn

x = torch.randn(4, 8)  # batch=4, features=8

# Batch Norm: normalise across batch for each feature
bn = nn.BatchNorm1d(8)
bn_out = bn(x)   # statistics computed over 4 samples

# Layer Norm: normalise across features for each sample
ln = nn.LayerNorm(8)
ln_out = ln(x)   # statistics computed over 8 features

print('BN output shape:', bn_out.shape)   # (4, 8)
print('LN output shape:', ln_out.shape)   # (4, 8)
# Same shape, different normalisation axes

Freezing Batch Norm During Fine-Tuning

When fine-tuning a pre-trained model on a small dataset, the running statistics in Batch Norm layers were estimated on the original large dataset. Allowing them to update on a small fine-tuning batch can corrupt them and hurt performance. A common strategy is to freeze Batch Norm layers by putting them permanently in eval mode. In PyTorch this is done by calling model.apply with a custom function that freezes each BN layer.

import torch.nn as nn

def freeze_bn(module):
    '''Keep BN in eval mode during fine-tuning.'''
    if isinstance(module, (nn.BatchNorm1d,
                           nn.BatchNorm2d,
                           nn.BatchNorm3d)):
        module.eval()  # use running stats, not batch stats
        module.weight.requires_grad_(False)
        module.bias.requires_grad_(False)

model = nn.Sequential(
    nn.Linear(4, 8),
    nn.BatchNorm1d(8),
    nn.ReLU()
)
model.apply(freeze_bn)
print('BN frozen for fine-tuning')

Batch Norm Limitations and Alternatives

Batch Norm has known limitations: it requires a minimum batch size (usually 16+) for reliable statistics; it is ineffective or harmful on very small batches; it introduces data dependency between samples in a batch, complicating parallelisation. Alternatives include Group Norm (divide channels into groups), Instance Norm (normalise per sample and channel, used in style transfer), and Layer Norm (used in Transformers). Choosing the right normalisation is architecture-dependent.

import torch
import torch.nn as nn

x = torch.randn(4, 16, 10)  # (batch, channels, seq_len)

# GroupNorm: 4 groups of 4 channels each
gn = nn.GroupNorm(num_groups=4, num_channels=16)
print('GroupNorm:', gn(x).shape)

# InstanceNorm: normalise each sample+channel independently
ins = nn.InstanceNorm1d(16)
print('InstanceNorm:', ins(x).shape)

# LayerNorm: normalise across last N dimensions
ln = nn.LayerNorm([16, 10])
print('LayerNorm:', ln(x).shape)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Batch Normalisation normalises layer inputs within each mini-batch to stabilise and accelerate training, nn.BatchNorm1d and nn.BatchNorm2d are used in fully connected and convolutional layers respectively, and model.eval() switches BN to use running statistics accumulated during training for deterministic inference. Next up we add Dropout regularisation to prevent overfitting.

Często zadawane pytania

Czy lekcja „Normalizacja wsadowa: stabilniejsze i szybsze trenowanie” jest bezpłatna?

Tak — pełny tekst „Normalizacja wsadowa: stabilniejsze i szybsze trenowanie” jest dostępny za darmo tutaj w sieci. Aby ćwiczyć ją interaktywnie (wbudowany edytor kodu i tutor AI dostępny 24/7) i odblokować resztę kursu Machine Learning Academy, przejdź na CoddyKit PRO. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.

Co nauczysz się w „Normalizacja wsadowa: stabilniejsze i szybsze trenowanie”?

Uczą się Państwo wstawiać nn.BatchNorm1d między warstwy, obserwować szybszą zbieżność głębokiej sieci oraz rozumieć, jak BatchNorm normalizuje aktywacje w każdym mini-wsadowym zbiorze. Ćwiczysz Machine Learning Academy z praktycznym kodem, który uruchamiasz bezpośrednio w przeglądarce, a tutor AI dostępny 24/7 odpowiada na Twoje pytania podczas pracy nad lekcją.

Czy potrzebuję doświadczenia, aby zacząć Machine Learning Academy?

Nie wymagamy żadnego doświadczenia. Machine Learning Academy w CoddyKit jest strukturyzowany dla początkujących i zaawansowanych użytkowników, więc możesz zacząć tutaj lub od początku i uczyć się w swoim tempie. To lekcja 2 z 4.

Ile czasu zajmuje lekcja „Normalizacja wsadowa: stabilniejsze i szybsze trenowanie”?

Większość lekcji CoddyKit trwa około 5–10 minut. Każda lekcja to mały, interaktywny krok, dzięki czemu robisz systematyczne postępy i zawsze wracasz dokładnie do tego samego miejsca — na webie i w aplikacji.

Czy mogę pisać i uruchamiać kod w tej lekcji Machine Learning Academy?

Tak. Każda lekcja Machine Learning Academy zawiera wbudowany edytor kodu, więc piszesz i uruchamiasz prawdziwy kod bezpośrednio w przeglądarce i od razu otrzymujesz sprzężenie zwrotne od AI — bez konfiguracji na komputerze.

Wszystkie lekcje w tym kursie

  1. Współczynnik uczenia: najważniejszy hiperparametr
  2. Normalizacja wsadowa: stabilniejsze i szybsze trenowanie
  3. Regularyzacja Dropout zapobiegająca przeuczeniu
  4. Inicjalizacja wag: inicjalizacja Xavier i He
← Powrót do Machine Learning Academy