0Pricing
Machine Learning Academy · 课时

批归一化：更稳定、更快速的训练

您将在各层之间插入 nn.BatchNorm1d，观察深层网络更快的收敛速度，并理解 BatchNorm 如何在每个小批次内对激活值进行归一化。

批归一化：更稳定、更快速的训练 是 CoddyKit 上的免费 Machine Learning Academy 课时。 这是第 2 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

The Problem Batch Norm Solves

Deep neural networks suffer from internal covariate shift — the distribution of each layer's inputs changes during training as the previous layer's weights update. This forces each layer to continuously adapt to a shifting input distribution, slowing training. Batch Normalisation (Batch Norm), introduced by Ioffe and Szegedy in 2015, addresses this by normalising layer inputs within each mini-batch, dramatically accelerating training and reducing sensitivity to weight initialisation.

# Without batch norm: deep networks train slowly and
# require very careful weight init and LR tuning.

# With batch norm: can use higher learning rates,
# less sensitive to initialisation, acts as regulariser.

# Batch norm normalises each feature to:
# mean=0, std=1 within the batch, then
# applies learnable scale (gamma) and shift (beta).
print('Batch Norm: normalize -> scale -> shift')

How Batch Norm Works Mathematically

For each feature dimension, Batch Norm computes the mean and variance across the current mini-batch, then normalises each value. After normalisation it applies two learnable parameters: gamma (scale) and beta (shift). This allows the network to undo the normalisation if needed — the identity transform is recoverable. A small constant epsilon is added to the variance to prevent division by zero.

import torch

def batch_norm_manual(x, gamma, beta, eps=1e-5):
    # x shape: (batch_size, features)
    mu = x.mean(dim=0)           # mean per feature
    var = x.var(dim=0, unbiased=False)  # var per feature
    x_norm = (x - mu) / (var + eps).sqrt()
    return gamma * x_norm + beta  # scale and shift

x = torch.randn(32, 8)  # batch=32, 8 features
gamma = torch.ones(8)
beta = torch.zeros(8)

out = batch_norm_manual(x, gamma, beta)
print('Mean near 0:', out.mean(dim=0).abs().max().item() < 0.01)
print('Std near 1:', (out.std(dim=0) - 1).abs().max().item() < 0.01)

nn.BatchNorm1d for Fully Connected Layers

nn.BatchNorm1d is used after linear layers in feedforward networks. It takes num_features (the size of the previous layer's output) as its argument. Batch Norm is typically placed after the linear layer but before the activation function, though there is ongoing debate about whether before or after activation is better. The module maintains running mean and variance for use during inference.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(16, 64),
    nn.BatchNorm1d(64),   # after linear, before activation
    nn.ReLU(),
    nn.Linear(64, 32),
    nn.BatchNorm1d(32),
    nn.ReLU(),
    nn.Linear(32, 10)
)

x = torch.randn(32, 16)   # batch of 32
out = model(x)
print(out.shape)           # torch.Size([32, 10])

Training vs Inference Behaviour

Batch Norm behaves differently in training and inference. During training it uses the mini-batch statistics (mean and variance of the current batch). During inference it uses running statistics (exponential moving average accumulated during training) so that predictions are deterministic and independent of batch size. This is why you must call model.eval() during inference — it switches BatchNorm to use the running statistics.

import torch
import torch.nn as nn

bn = nn.BatchNorm1d(4)

# Training mode: uses batch statistics, updates running stats
bn.train()
x = torch.randn(8, 4)
out_train = bn(x)

# After training, running_mean and running_var are populated
print('Running mean:', bn.running_mean)

# Eval mode: uses running statistics (deterministic)
bn.eval()
x_new = torch.randn(1, 4)   # single sample -- works!
out_eval = bn(x_new)
print(out_eval.shape)         # torch.Size([1, 4])

Learnable Parameters: gamma and beta

Batch Norm has two learnable parameters per feature: weight (gamma, initialised to 1) and bias (beta, initialised to 0). These allow the network to learn to rescale and reshift the normalised values if that improves the task. They are updated by the optimizer just like regular weight matrices. If you want to freeze Batch Norm layers during fine-tuning, set requires_grad=False on these parameters.

import torch.nn as nn

bn = nn.BatchNorm1d(8)
print('gamma (weight):', bn.weight.data)   # all 1s
print('beta (bias):', bn.bias.data)         # all 0s
print('gamma requires_grad:', bn.weight.requires_grad)  # True
print('beta requires_grad:', bn.bias.requires_grad)     # True

# Total trainable params in this BN layer:
# 2 * 8 = 16 (gamma and beta for 8 features)
params = sum(p.numel() for p in bn.parameters())
print('Params:', params)   # 16

Observing Faster Convergence

One of the clearest benefits of Batch Norm is faster convergence. Networks with Batch Norm typically reach the same validation accuracy in fewer epochs and can use larger learning rates without instability. The normalisation keeps activations in a healthy range throughout training, preventing the saturation that slows learning in networks using sigmoid or tanh activations. The effect is most pronounced in deep networks with many layers.

import torch
import torch.nn as nn
import torch.optim as optim

def make_model(use_bn):
    layers = [nn.Linear(16, 64)]
    if use_bn: layers.append(nn.BatchNorm1d(64))
    layers.append(nn.ReLU())
    layers.append(nn.Linear(64, 2))
    return nn.Sequential(*layers)

X = torch.randn(200, 16)
y = torch.randint(0, 2, (200,))

for use_bn in [False, True]:
    model = make_model(use_bn)
    opt = optim.SGD(model.parameters(), lr=0.1)
    crit = nn.CrossEntropyLoss()
    for _ in range(20):
        opt.zero_grad(); loss = crit(model(X), y)
        loss.backward(); opt.step()
    print(f'BN={use_bn}: final_loss={loss.item():.4f}')

Batch Norm as a Regulariser

Batch Norm acts as a mild regulariser because each training sample is normalised with respect to the other samples in the mini-batch — introducing stochasticity similar to Dropout. This means networks with Batch Norm often need less Dropout. The regularisation effect diminishes with larger batch sizes because the batch statistics become more deterministic, approaching the true population statistics and removing the stochastic element.

# Key insight: batch norm introduces noise proportional to
# 1/sqrt(batch_size) because batch statistics are noisy
# estimates of population statistics.

# Small batch (e.g., 8): high noise -> more regularisation
# Large batch (e.g., 512): low noise -> less regularisation

# Common pattern: use batch norm AND a small dropout
# for strong regularisation in deep networks
model = __import__('torch').nn.Sequential(
    __import__('torch').nn.Linear(32, 128),
    __import__('torch').nn.BatchNorm1d(128),
    __import__('torch').nn.ReLU(),
    __import__('torch').nn.Dropout(0.2)  # mild dropout
)
print('BN + light Dropout: balanced regularisation')

nn.BatchNorm2d for Convolutional Networks

In convolutional networks, nn.BatchNorm2d normalises across the batch and spatial dimensions for each channel independently. It takes num_channels as its argument (matching the number of output channels from the preceding Conv2d layer). The standard pattern is Conv2d -> BatchNorm2d -> ReLU, which is used in nearly every modern CNN architecture including ResNet, VGG, and EfficientNet.

import torch
import torch.nn as nn

# Standard CNN block: Conv -> BN -> ReLU
conv_block = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3, padding=1),
    nn.BatchNorm2d(64),    # 64 = number of output channels
    nn.ReLU(inplace=True)
)

# Input: batch of 8 RGB images, 32x32 pixels
x = torch.randn(8, 3, 32, 32)
out = conv_block(x)
print(out.shape)   # torch.Size([8, 64, 32, 32])

Layer Norm vs Batch Norm

Layer Normalisation (used in Transformers) normalises across features within a single sample, rather than across the batch. This makes it independent of batch size, which is essential for variable-length sequences and small batches. Batch Norm normalises across the batch for each feature — ideal for CNNs and large-batch training. The wrong choice can hurt: using Batch Norm in a Transformer or Layer Norm in a CNN is a common architecture mistake.

import torch
import torch.nn as nn

x = torch.randn(4, 8)  # batch=4, features=8

# Batch Norm: normalise across batch for each feature
bn = nn.BatchNorm1d(8)
bn_out = bn(x)   # statistics computed over 4 samples

# Layer Norm: normalise across features for each sample
ln = nn.LayerNorm(8)
ln_out = ln(x)   # statistics computed over 8 features

print('BN output shape:', bn_out.shape)   # (4, 8)
print('LN output shape:', ln_out.shape)   # (4, 8)
# Same shape, different normalisation axes

Freezing Batch Norm During Fine-Tuning

When fine-tuning a pre-trained model on a small dataset, the running statistics in Batch Norm layers were estimated on the original large dataset. Allowing them to update on a small fine-tuning batch can corrupt them and hurt performance. A common strategy is to freeze Batch Norm layers by putting them permanently in eval mode. In PyTorch this is done by calling model.apply with a custom function that freezes each BN layer.

import torch.nn as nn

def freeze_bn(module):
    '''Keep BN in eval mode during fine-tuning.'''
    if isinstance(module, (nn.BatchNorm1d,
                           nn.BatchNorm2d,
                           nn.BatchNorm3d)):
        module.eval()  # use running stats, not batch stats
        module.weight.requires_grad_(False)
        module.bias.requires_grad_(False)

model = nn.Sequential(
    nn.Linear(4, 8),
    nn.BatchNorm1d(8),
    nn.ReLU()
)
model.apply(freeze_bn)
print('BN frozen for fine-tuning')

Batch Norm Limitations and Alternatives

Batch Norm has known limitations: it requires a minimum batch size (usually 16+) for reliable statistics; it is ineffective or harmful on very small batches; it introduces data dependency between samples in a batch, complicating parallelisation. Alternatives include Group Norm (divide channels into groups), Instance Norm (normalise per sample and channel, used in style transfer), and Layer Norm (used in Transformers). Choosing the right normalisation is architecture-dependent.

import torch
import torch.nn as nn

x = torch.randn(4, 16, 10)  # (batch, channels, seq_len)

# GroupNorm: 4 groups of 4 channels each
gn = nn.GroupNorm(num_groups=4, num_channels=16)
print('GroupNorm:', gn(x).shape)

# InstanceNorm: normalise each sample+channel independently
ins = nn.InstanceNorm1d(16)
print('InstanceNorm:', ins(x).shape)

# LayerNorm: normalise across last N dimensions
ln = nn.LayerNorm([16, 10])
print('LayerNorm:', ln(x).shape)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Batch Normalisation normalises layer inputs within each mini-batch to stabilise and accelerate training, nn.BatchNorm1d and nn.BatchNorm2d are used in fully connected and convolutional layers respectively, and model.eval() switches BN to use running statistics accumulated during training for deterministic inference. Next up we add Dropout regularisation to prevent overfitting.

常见问题解答

「批归一化：更稳定、更快速的训练」课时是免费的吗？

是的 — 「批归一化：更稳定、更快速的训练」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。

「批归一化：更稳定、更快速的训练」这节课中我会学到什么？

您将在各层之间插入 nn.BatchNorm1d，观察深层网络更快的收敛速度，并理解 BatchNorm 如何在每个小批次内对激活值进行归一化。 你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Machine Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 2 节课，共 4 节。

「批归一化：更稳定、更快速的训练」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Machine Learning Academy 课中编写并运行代码吗？

能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 学习率：最重要的超参数
  2. 批归一化：更稳定、更快速的训练
  3. 使用 Dropout 正则化防止过拟合
  4. 权重初始化：Xavier 与 He 初始化
← 返回 Machine Learning Academy