0Pricing
Machine Learning Academy · Aula

Taxa de aprendizado: o hiperparâmetro mais importante

Os alunos executarão um teste de faixa da taxa de aprendizado, representarão a perda em relação à LR, identificarão a faixa ideal e aplicarão um agendamento CosineAnnealingLR para evitar platôs.

Taxa de aprendizado: o hiperparâmetro mais importante é uma aula grátis de Machine Learning Academy no CoddyKit. Esta é a aula 1 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Machine Learning Academy, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Machine Learning Academy inclui 4 aulas no total.

Partes desta aula ainda não foram traduzidas e aparecem em inglês.

Why Learning Rate Matters Most

The learning rate (LR) is the single hyperparameter that most affects whether a neural network trains successfully. It controls how large a step the optimizer takes in the direction of the negative gradient. Too large and the model diverges; too small and training takes forever or gets stuck. Unlike architecture choices, LR must be tuned almost every time you change the dataset, model size, or batch size.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(1, 1)

# Too large: diverges
optimizer_big = optim.SGD(model.parameters(), lr=10.0)

# Too small: barely moves
optimizer_small = optim.SGD(model.parameters(), lr=1e-6)

# Good: converges steadily
optimizer_good = optim.SGD(model.parameters(), lr=0.01)

print('LR comparison: 10.0, 1e-6, 0.01')

Effect of LR on Loss Curves

Different learning rates produce recognisable patterns in the loss curve. Too high: loss oscillates wildly or increases after a few steps. Too low: loss decreases extremely slowly, nearly flat. Just right: loss decreases smoothly and consistently. Plotting loss vs batch/epoch for a few representative LR values (e.g., 1e-4, 1e-3, 1e-2, 1e-1) before committing to a long training run is standard practice.

import torch
import torch.nn as nn
import torch.optim as optim

def train_one_lr(lr, steps=50):
    model = nn.Linear(1, 1)
    opt = optim.SGD(model.parameters(), lr=lr)
    X = torch.randn(100, 1)
    y = 2 * X + 1
    losses = []
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.MSELoss()(model(X), y)
        loss.backward(); opt.step()
        losses.append(loss.item())
    return losses[-1]

for lr in [1e-4, 1e-2, 0.1, 1.0]:
    final = train_one_lr(lr)
    print(f'LR={lr:.4f}: final_loss={final:.4f}')

The Learning Rate Range Test

The LR range test (popularised by Leslie Smith) finds a good LR automatically. Start with a very small LR and increase it exponentially over many mini-batches while recording the loss. The loss first decreases, then rises steeply when LR is too large. The optimal LR is roughly 10x smaller than where the loss starts rising. This test takes only a few minutes and eliminates the need for an expensive grid search over LR.

import torch
import torch.nn as nn
import torch.optim as optim
import math

def lr_range_test(model, loader, criterion, start_lr=1e-7, end_lr=10, num_iter=100):
    optimizer = optim.SGD(model.parameters(), lr=start_lr)
    lrs, losses = [], []
    mult = (end_lr / start_lr) ** (1 / num_iter)
    lr = start_lr
    for i, (X, y) in enumerate(loader):
        if i >= num_iter: break
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward(); optimizer.step()
        lrs.append(lr)
        losses.append(loss.item())
        lr *= mult
        for pg in optimizer.param_groups:
            pg['lr'] = lr
    return lrs, losses

Warm-Up: Starting Small and Growing

Learning rate warm-up starts training with a very small LR and gradually increases it to the target LR over the first few hundred steps or epochs. This prevents large unstable updates at the very beginning when weights are randomly initialised and gradients are noisy. Warm-up is especially important for Transformers and large batch training, where large initial steps can send the model to a poor region of loss landscape that is hard to escape.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

def warmup_lambda(current_step, warmup_steps=100):
    if current_step < warmup_steps:
        return current_step / warmup_steps
    return 1.0

scheduler = optim.lr_scheduler.LambdaLR(
    optimizer, lr_lambda=warmup_lambda
)

for step in range(5):
    scheduler.step()
    print(f'Step {step}: LR={optimizer.param_groups[0]["lr"]:.6f}')

Step Decay Scheduling with StepLR

StepLR reduces the learning rate by a multiplicative factor gamma every step_size epochs. For example, halving the LR every 10 epochs (gamma=0.5, step_size=10) is a common schedule for image classification. Step decay is simple to reason about and works well when you know roughly how many epochs the model needs to settle. The scheduler must be called after the optimizer step each epoch.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(
    optimizer, step_size=3, gamma=0.5
)

for epoch in range(9):
    # (training happens here)
    scheduler.step()
    print(f'After epoch {epoch}: LR={optimizer.param_groups[0]["lr"]:.4f}')
# 0.1 -> 0.1 -> 0.1 -> 0.05 -> 0.05 -> 0.05 -> 0.025 ...

Cosine Annealing: Smooth Decay to Zero

CosineAnnealingLR decays the LR following a cosine curve from the initial LR to a minimum (eta_min, default 0) over T_max steps. The cosine shape gives fast initial decrease with a soft landing near zero. CosineAnnealingWarmRestarts adds periodic restarts that escape local minima, creating a distinctive sawtooth LR pattern. Cosine schedules are among the most widely used in modern deep learning.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10, eta_min=1e-4
)

for epoch in range(10):
    scheduler.step()
    lr = optimizer.param_groups[0]['lr']
    print(f'Epoch {epoch}: LR={lr:.5f}')
# Smoothly decays from 0.1 to 0.0001 over 10 epochs

ReduceLROnPlateau: Adaptive Scheduling

ReduceLROnPlateau monitors a metric (usually validation loss) and reduces LR by a factor when the metric stops improving for a specified number of epochs (patience). This is the most adaptive scheduler because it responds to actual training dynamics rather than a predetermined schedule. It is especially effective when you are unsure how many epochs training will take or when training progress is uneven.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode='min',       # reduce when metric stops going down
    factor=0.5,       # multiply LR by 0.5
    patience=3,       # wait 3 epochs before reducing
    min_lr=1e-6
)

# Each epoch, pass the validation loss
for epoch in range(10):
    val_loss = 1.0 / (epoch + 1)   # simulated decreasing loss
    scheduler.step(val_loss)
    print(f'Epoch {epoch}: LR={optimizer.param_groups[0]["lr"]}')

Batch Size and Its Relationship to LR

Batch size and learning rate are tightly coupled. When you double the batch size, gradients are averaged over twice as many samples — they are less noisy. A commonly used heuristic is the linear scaling rule: multiply the LR by the same factor as the batch size increase. For example, doubling batch size from 64 to 128 suggests doubling LR as well. This rule works well in the moderate regime but breaks down for very large batches.

# Linear scaling rule: if base LR=0.01 with batch_size=64
# and you change to batch_size=256 (4x larger):

base_lr = 0.01
base_batch = 64
new_batch = 256

scaled_lr = base_lr * (new_batch / base_batch)
print(f'Scaled LR: {scaled_lr}')   # 0.04

# But use warm-up when scaling to very large batches
# to avoid instability at the start of training

Gradient Clipping: Preventing Exploding Gradients

When the LR is slightly too high or gradients are naturally large (common in RNNs), gradient clipping prevents parameter updates from being catastrophically large. torch.nn.utils.clip_grad_norm_ rescales the gradient vector to have a maximum L2 norm. The clipping happens after calling .backward() but before optimizer.step(). A max norm of 1.0 is a common default for recurrent networks.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.LSTM(4, 8, batch_first=True)
optimizer = optim.Adam(model.parameters(), lr=0.001)

x = torch.randn(16, 10, 4)   # batch=16, seq=10, features=4
out, _ = model(x)
loss = out.sum()
loss.backward()

# Clip gradients before optimizer step
total_norm = nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0
)
print(f'Gradient norm before clip: {total_norm:.4f}')
optimizer.step()

Finding LR with PyTorch Lightning or Manual Loop

Many practitioners use PyTorch Lightning's built-in LR finder, which automates the LR range test. If you are using raw PyTorch, implement the range test manually by exponentially increasing LR over 100 mini-batches and plotting loss vs LR on a log scale. The sweet spot is where loss decreases most steeply. Tools like torch-lr-finder package wrap this into a single function call for convenience.

# Manual LR finder sketch (pseudocode)
import math

# 1. Save initial model state
# torch.save(model.state_dict(), 'init.pt')

# 2. Sweep LR exponentially from 1e-7 to 1
start, end, steps = 1e-7, 1.0, 100
mult = (end / start) ** (1 / steps)
lr = start
for i, (X, y) in enumerate(train_loader):
    if i >= steps: break
    # train one step with current lr...
    lr *= mult

# 3. Plot lrs vs losses on log-linear scale
# 4. Pick LR where loss drops fastest
# 5. Restore initial model state

LR Summary and Practical Rules

The most important practical rules for learning rate: start with 1e-3 for Adam and 0.01 for SGD as default values. Always run a quick LR range test when using a new dataset or architecture. Use cosine annealing or ReduceLROnPlateau rather than a fixed LR for best convergence. Scale LR linearly with batch size. Always use gradient clipping for RNNs. The effort spent tuning LR pays off more than almost any other optimisation.

# Quick reference for default starting points
defaults = {
    'SGD':   {'lr': 0.01, 'momentum': 0.9},
    'Adam':  {'lr': 1e-3, 'betas': (0.9, 0.999)},
    'AdamW': {'lr': 1e-3, 'weight_decay': 0.01},
    'RMSprop': {'lr': 1e-4}
}

print('Default LR starting points:')
for opt, params in defaults.items():
    print(f'  {opt}: {params}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the learning rate is the most critical hyperparameter controlling convergence speed and stability, the LR range test finds a good LR quickly by sweeping from small to large and looking for the steepest loss decrease, and schedulers like CosineAnnealingLR and ReduceLROnPlateau automatically adjust LR during training for better final performance. Next up we explore batch normalisation for faster and more stable training.

Perguntas Frequentes

A aula “Taxa de aprendizado: o hiperparâmetro mais importante” é grátis?

Sim — o texto completo de “Taxa de aprendizado: o hiperparâmetro mais importante” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Machine Learning Academy, atualize para CoddyKit PRO. O curso de Machine Learning Academy inclui 4 aulas no total.

O que vou aprender em “Taxa de aprendizado: o hiperparâmetro mais importante”?

Os alunos executarão um teste de faixa da taxa de aprendizado, representarão a perda em relação à LR, identificarão a faixa ideal e aplicarão um agendamento CosineAnnealingLR para evitar platôs. Você pratica Machine Learning Academy com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.

Preciso ter experiência prévia para começar Machine Learning Academy?

Nenhuma experiência prévia é necessária. Machine Learning Academy no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 1 de 4.

Quanto tempo leva a aula “Taxa de aprendizado: o hiperparâmetro mais importante”?

A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.

Posso escrever e executar código nesta aula de Machine Learning Academy?

Sim. Cada aula de Machine Learning Academy inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.

Todas as aulas deste curso

  1. Taxa de aprendizado: o hiperparâmetro mais importante
  2. Normalização em lotes: treinamento estável e mais rápido
  3. Regularização com dropout para evitar sobreajuste
  4. Inicialização de pesos: inicialização Xavier e He
← Voltar para Machine Learning Academy