0Pricing
Machine Learning Academy · Aula

Normalização em lotes: treinamento estável e mais rápido

Os alunos inserirã o nn.BatchNorm1d entre as camadas, observarão uma convergência mais rápida em uma rede profunda e entenderão como o BatchNorm normaliza as ativações em cada minilote.

Normalização em lotes: treinamento estável e mais rápido é uma aula grátis de Machine Learning Academy no CoddyKit. Esta é a aula 2 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Machine Learning Academy, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Machine Learning Academy inclui 4 aulas no total.

Partes desta aula ainda não foram traduzidas e aparecem em inglês.

The Problem Batch Norm Solves

Deep neural networks suffer from internal covariate shift — the distribution of each layer's inputs changes during training as the previous layer's weights update. This forces each layer to continuously adapt to a shifting input distribution, slowing training. Batch Normalisation (Batch Norm), introduced by Ioffe and Szegedy in 2015, addresses this by normalising layer inputs within each mini-batch, dramatically accelerating training and reducing sensitivity to weight initialisation.

# Without batch norm: deep networks train slowly and
# require very careful weight init and LR tuning.

# With batch norm: can use higher learning rates,
# less sensitive to initialisation, acts as regulariser.

# Batch norm normalises each feature to:
# mean=0, std=1 within the batch, then
# applies learnable scale (gamma) and shift (beta).
print('Batch Norm: normalize -> scale -> shift')

How Batch Norm Works Mathematically

For each feature dimension, Batch Norm computes the mean and variance across the current mini-batch, then normalises each value. After normalisation it applies two learnable parameters: gamma (scale) and beta (shift). This allows the network to undo the normalisation if needed — the identity transform is recoverable. A small constant epsilon is added to the variance to prevent division by zero.

import torch

def batch_norm_manual(x, gamma, beta, eps=1e-5):
    # x shape: (batch_size, features)
    mu = x.mean(dim=0)           # mean per feature
    var = x.var(dim=0, unbiased=False)  # var per feature
    x_norm = (x - mu) / (var + eps).sqrt()
    return gamma * x_norm + beta  # scale and shift

x = torch.randn(32, 8)  # batch=32, 8 features
gamma = torch.ones(8)
beta = torch.zeros(8)

out = batch_norm_manual(x, gamma, beta)
print('Mean near 0:', out.mean(dim=0).abs().max().item() < 0.01)
print('Std near 1:', (out.std(dim=0) - 1).abs().max().item() < 0.01)

nn.BatchNorm1d for Fully Connected Layers

nn.BatchNorm1d is used after linear layers in feedforward networks. It takes num_features (the size of the previous layer's output) as its argument. Batch Norm is typically placed after the linear layer but before the activation function, though there is ongoing debate about whether before or after activation is better. The module maintains running mean and variance for use during inference.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(16, 64),
    nn.BatchNorm1d(64),   # after linear, before activation
    nn.ReLU(),
    nn.Linear(64, 32),
    nn.BatchNorm1d(32),
    nn.ReLU(),
    nn.Linear(32, 10)
)

x = torch.randn(32, 16)   # batch of 32
out = model(x)
print(out.shape)           # torch.Size([32, 10])

Training vs Inference Behaviour

Batch Norm behaves differently in training and inference. During training it uses the mini-batch statistics (mean and variance of the current batch). During inference it uses running statistics (exponential moving average accumulated during training) so that predictions are deterministic and independent of batch size. This is why you must call model.eval() during inference — it switches BatchNorm to use the running statistics.

import torch
import torch.nn as nn

bn = nn.BatchNorm1d(4)

# Training mode: uses batch statistics, updates running stats
bn.train()
x = torch.randn(8, 4)
out_train = bn(x)

# After training, running_mean and running_var are populated
print('Running mean:', bn.running_mean)

# Eval mode: uses running statistics (deterministic)
bn.eval()
x_new = torch.randn(1, 4)   # single sample -- works!
out_eval = bn(x_new)
print(out_eval.shape)         # torch.Size([1, 4])

Learnable Parameters: gamma and beta

Batch Norm has two learnable parameters per feature: weight (gamma, initialised to 1) and bias (beta, initialised to 0). These allow the network to learn to rescale and reshift the normalised values if that improves the task. They are updated by the optimizer just like regular weight matrices. If you want to freeze Batch Norm layers during fine-tuning, set requires_grad=False on these parameters.

import torch.nn as nn

bn = nn.BatchNorm1d(8)
print('gamma (weight):', bn.weight.data)   # all 1s
print('beta (bias):', bn.bias.data)         # all 0s
print('gamma requires_grad:', bn.weight.requires_grad)  # True
print('beta requires_grad:', bn.bias.requires_grad)     # True

# Total trainable params in this BN layer:
# 2 * 8 = 16 (gamma and beta for 8 features)
params = sum(p.numel() for p in bn.parameters())
print('Params:', params)   # 16

Observing Faster Convergence

One of the clearest benefits of Batch Norm is faster convergence. Networks with Batch Norm typically reach the same validation accuracy in fewer epochs and can use larger learning rates without instability. The normalisation keeps activations in a healthy range throughout training, preventing the saturation that slows learning in networks using sigmoid or tanh activations. The effect is most pronounced in deep networks with many layers.

import torch
import torch.nn as nn
import torch.optim as optim

def make_model(use_bn):
    layers = [nn.Linear(16, 64)]
    if use_bn: layers.append(nn.BatchNorm1d(64))
    layers.append(nn.ReLU())
    layers.append(nn.Linear(64, 2))
    return nn.Sequential(*layers)

X = torch.randn(200, 16)
y = torch.randint(0, 2, (200,))

for use_bn in [False, True]:
    model = make_model(use_bn)
    opt = optim.SGD(model.parameters(), lr=0.1)
    crit = nn.CrossEntropyLoss()
    for _ in range(20):
        opt.zero_grad(); loss = crit(model(X), y)
        loss.backward(); opt.step()
    print(f'BN={use_bn}: final_loss={loss.item():.4f}')

Batch Norm as a Regulariser

Batch Norm acts as a mild regulariser because each training sample is normalised with respect to the other samples in the mini-batch — introducing stochasticity similar to Dropout. This means networks with Batch Norm often need less Dropout. The regularisation effect diminishes with larger batch sizes because the batch statistics become more deterministic, approaching the true population statistics and removing the stochastic element.

# Key insight: batch norm introduces noise proportional to
# 1/sqrt(batch_size) because batch statistics are noisy
# estimates of population statistics.

# Small batch (e.g., 8): high noise -> more regularisation
# Large batch (e.g., 512): low noise -> less regularisation

# Common pattern: use batch norm AND a small dropout
# for strong regularisation in deep networks
model = __import__('torch').nn.Sequential(
    __import__('torch').nn.Linear(32, 128),
    __import__('torch').nn.BatchNorm1d(128),
    __import__('torch').nn.ReLU(),
    __import__('torch').nn.Dropout(0.2)  # mild dropout
)
print('BN + light Dropout: balanced regularisation')

nn.BatchNorm2d for Convolutional Networks

In convolutional networks, nn.BatchNorm2d normalises across the batch and spatial dimensions for each channel independently. It takes num_channels as its argument (matching the number of output channels from the preceding Conv2d layer). The standard pattern is Conv2d -> BatchNorm2d -> ReLU, which is used in nearly every modern CNN architecture including ResNet, VGG, and EfficientNet.

import torch
import torch.nn as nn

# Standard CNN block: Conv -> BN -> ReLU
conv_block = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3, padding=1),
    nn.BatchNorm2d(64),    # 64 = number of output channels
    nn.ReLU(inplace=True)
)

# Input: batch of 8 RGB images, 32x32 pixels
x = torch.randn(8, 3, 32, 32)
out = conv_block(x)
print(out.shape)   # torch.Size([8, 64, 32, 32])

Layer Norm vs Batch Norm

Layer Normalisation (used in Transformers) normalises across features within a single sample, rather than across the batch. This makes it independent of batch size, which is essential for variable-length sequences and small batches. Batch Norm normalises across the batch for each feature — ideal for CNNs and large-batch training. The wrong choice can hurt: using Batch Norm in a Transformer or Layer Norm in a CNN is a common architecture mistake.

import torch
import torch.nn as nn

x = torch.randn(4, 8)  # batch=4, features=8

# Batch Norm: normalise across batch for each feature
bn = nn.BatchNorm1d(8)
bn_out = bn(x)   # statistics computed over 4 samples

# Layer Norm: normalise across features for each sample
ln = nn.LayerNorm(8)
ln_out = ln(x)   # statistics computed over 8 features

print('BN output shape:', bn_out.shape)   # (4, 8)
print('LN output shape:', ln_out.shape)   # (4, 8)
# Same shape, different normalisation axes

Freezing Batch Norm During Fine-Tuning

When fine-tuning a pre-trained model on a small dataset, the running statistics in Batch Norm layers were estimated on the original large dataset. Allowing them to update on a small fine-tuning batch can corrupt them and hurt performance. A common strategy is to freeze Batch Norm layers by putting them permanently in eval mode. In PyTorch this is done by calling model.apply with a custom function that freezes each BN layer.

import torch.nn as nn

def freeze_bn(module):
    '''Keep BN in eval mode during fine-tuning.'''
    if isinstance(module, (nn.BatchNorm1d,
                           nn.BatchNorm2d,
                           nn.BatchNorm3d)):
        module.eval()  # use running stats, not batch stats
        module.weight.requires_grad_(False)
        module.bias.requires_grad_(False)

model = nn.Sequential(
    nn.Linear(4, 8),
    nn.BatchNorm1d(8),
    nn.ReLU()
)
model.apply(freeze_bn)
print('BN frozen for fine-tuning')

Batch Norm Limitations and Alternatives

Batch Norm has known limitations: it requires a minimum batch size (usually 16+) for reliable statistics; it is ineffective or harmful on very small batches; it introduces data dependency between samples in a batch, complicating parallelisation. Alternatives include Group Norm (divide channels into groups), Instance Norm (normalise per sample and channel, used in style transfer), and Layer Norm (used in Transformers). Choosing the right normalisation is architecture-dependent.

import torch
import torch.nn as nn

x = torch.randn(4, 16, 10)  # (batch, channels, seq_len)

# GroupNorm: 4 groups of 4 channels each
gn = nn.GroupNorm(num_groups=4, num_channels=16)
print('GroupNorm:', gn(x).shape)

# InstanceNorm: normalise each sample+channel independently
ins = nn.InstanceNorm1d(16)
print('InstanceNorm:', ins(x).shape)

# LayerNorm: normalise across last N dimensions
ln = nn.LayerNorm([16, 10])
print('LayerNorm:', ln(x).shape)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Batch Normalisation normalises layer inputs within each mini-batch to stabilise and accelerate training, nn.BatchNorm1d and nn.BatchNorm2d are used in fully connected and convolutional layers respectively, and model.eval() switches BN to use running statistics accumulated during training for deterministic inference. Next up we add Dropout regularisation to prevent overfitting.

Perguntas Frequentes

A aula “Normalização em lotes: treinamento estável e mais rápido” é grátis?

Sim — o texto completo de “Normalização em lotes: treinamento estável e mais rápido” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Machine Learning Academy, atualize para CoddyKit PRO. O curso de Machine Learning Academy inclui 4 aulas no total.

O que vou aprender em “Normalização em lotes: treinamento estável e mais rápido”?

Os alunos inserirã o nn.BatchNorm1d entre as camadas, observarão uma convergência mais rápida em uma rede profunda e entenderão como o BatchNorm normaliza as ativações em cada minilote. Você pratica Machine Learning Academy com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.

Preciso ter experiência prévia para começar Machine Learning Academy?

Nenhuma experiência prévia é necessária. Machine Learning Academy no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 2 de 4.

Quanto tempo leva a aula “Normalização em lotes: treinamento estável e mais rápido”?

A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.

Posso escrever e executar código nesta aula de Machine Learning Academy?

Sim. Cada aula de Machine Learning Academy inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.

Todas as aulas deste curso

  1. Taxa de aprendizado: o hiperparâmetro mais importante
  2. Normalização em lotes: treinamento estável e mais rápido
  3. Regularização com dropout para evitar sobreajuste
  4. Inicialização de pesos: inicialização Xavier e He
← Voltar para Machine Learning Academy