0Pricing
Machine Learning Academy · 课时

学习率：最重要的超参数

您将运行学习率范围测试，绘制损失与 LR 的关系图，找出最佳范围，并应用 CosineAnnealingLR 调度器以避免训练停滞。

学习率：最重要的超参数 是 CoddyKit 上的免费 Machine Learning Academy 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Learning Rate Matters Most

The learning rate (LR) is the single hyperparameter that most affects whether a neural network trains successfully. It controls how large a step the optimizer takes in the direction of the negative gradient. Too large and the model diverges; too small and training takes forever or gets stuck. Unlike architecture choices, LR must be tuned almost every time you change the dataset, model size, or batch size.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(1, 1)

# Too large: diverges
optimizer_big = optim.SGD(model.parameters(), lr=10.0)

# Too small: barely moves
optimizer_small = optim.SGD(model.parameters(), lr=1e-6)

# Good: converges steadily
optimizer_good = optim.SGD(model.parameters(), lr=0.01)

print('LR comparison: 10.0, 1e-6, 0.01')

Effect of LR on Loss Curves

Different learning rates produce recognisable patterns in the loss curve. Too high: loss oscillates wildly or increases after a few steps. Too low: loss decreases extremely slowly, nearly flat. Just right: loss decreases smoothly and consistently. Plotting loss vs batch/epoch for a few representative LR values (e.g., 1e-4, 1e-3, 1e-2, 1e-1) before committing to a long training run is standard practice.

import torch
import torch.nn as nn
import torch.optim as optim

def train_one_lr(lr, steps=50):
    model = nn.Linear(1, 1)
    opt = optim.SGD(model.parameters(), lr=lr)
    X = torch.randn(100, 1)
    y = 2 * X + 1
    losses = []
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.MSELoss()(model(X), y)
        loss.backward(); opt.step()
        losses.append(loss.item())
    return losses[-1]

for lr in [1e-4, 1e-2, 0.1, 1.0]:
    final = train_one_lr(lr)
    print(f'LR={lr:.4f}: final_loss={final:.4f}')

The Learning Rate Range Test

The LR range test (popularised by Leslie Smith) finds a good LR automatically. Start with a very small LR and increase it exponentially over many mini-batches while recording the loss. The loss first decreases, then rises steeply when LR is too large. The optimal LR is roughly 10x smaller than where the loss starts rising. This test takes only a few minutes and eliminates the need for an expensive grid search over LR.

import torch
import torch.nn as nn
import torch.optim as optim
import math

def lr_range_test(model, loader, criterion, start_lr=1e-7, end_lr=10, num_iter=100):
    optimizer = optim.SGD(model.parameters(), lr=start_lr)
    lrs, losses = [], []
    mult = (end_lr / start_lr) ** (1 / num_iter)
    lr = start_lr
    for i, (X, y) in enumerate(loader):
        if i >= num_iter: break
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward(); optimizer.step()
        lrs.append(lr)
        losses.append(loss.item())
        lr *= mult
        for pg in optimizer.param_groups:
            pg['lr'] = lr
    return lrs, losses

Warm-Up: Starting Small and Growing

Learning rate warm-up starts training with a very small LR and gradually increases it to the target LR over the first few hundred steps or epochs. This prevents large unstable updates at the very beginning when weights are randomly initialised and gradients are noisy. Warm-up is especially important for Transformers and large batch training, where large initial steps can send the model to a poor region of loss landscape that is hard to escape.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

def warmup_lambda(current_step, warmup_steps=100):
    if current_step < warmup_steps:
        return current_step / warmup_steps
    return 1.0

scheduler = optim.lr_scheduler.LambdaLR(
    optimizer, lr_lambda=warmup_lambda
)

for step in range(5):
    scheduler.step()
    print(f'Step {step}: LR={optimizer.param_groups[0]["lr"]:.6f}')

Step Decay Scheduling with StepLR

StepLR reduces the learning rate by a multiplicative factor gamma every step_size epochs. For example, halving the LR every 10 epochs (gamma=0.5, step_size=10) is a common schedule for image classification. Step decay is simple to reason about and works well when you know roughly how many epochs the model needs to settle. The scheduler must be called after the optimizer step each epoch.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(
    optimizer, step_size=3, gamma=0.5
)

for epoch in range(9):
    # (training happens here)
    scheduler.step()
    print(f'After epoch {epoch}: LR={optimizer.param_groups[0]["lr"]:.4f}')
# 0.1 -> 0.1 -> 0.1 -> 0.05 -> 0.05 -> 0.05 -> 0.025 ...

Cosine Annealing: Smooth Decay to Zero

CosineAnnealingLR decays the LR following a cosine curve from the initial LR to a minimum (eta_min, default 0) over T_max steps. The cosine shape gives fast initial decrease with a soft landing near zero. CosineAnnealingWarmRestarts adds periodic restarts that escape local minima, creating a distinctive sawtooth LR pattern. Cosine schedules are among the most widely used in modern deep learning.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10, eta_min=1e-4
)

for epoch in range(10):
    scheduler.step()
    lr = optimizer.param_groups[0]['lr']
    print(f'Epoch {epoch}: LR={lr:.5f}')
# Smoothly decays from 0.1 to 0.0001 over 10 epochs

ReduceLROnPlateau: Adaptive Scheduling

ReduceLROnPlateau monitors a metric (usually validation loss) and reduces LR by a factor when the metric stops improving for a specified number of epochs (patience). This is the most adaptive scheduler because it responds to actual training dynamics rather than a predetermined schedule. It is especially effective when you are unsure how many epochs training will take or when training progress is uneven.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode='min',       # reduce when metric stops going down
    factor=0.5,       # multiply LR by 0.5
    patience=3,       # wait 3 epochs before reducing
    min_lr=1e-6
)

# Each epoch, pass the validation loss
for epoch in range(10):
    val_loss = 1.0 / (epoch + 1)   # simulated decreasing loss
    scheduler.step(val_loss)
    print(f'Epoch {epoch}: LR={optimizer.param_groups[0]["lr"]}')

Batch Size and Its Relationship to LR

Batch size and learning rate are tightly coupled. When you double the batch size, gradients are averaged over twice as many samples — they are less noisy. A commonly used heuristic is the linear scaling rule: multiply the LR by the same factor as the batch size increase. For example, doubling batch size from 64 to 128 suggests doubling LR as well. This rule works well in the moderate regime but breaks down for very large batches.

# Linear scaling rule: if base LR=0.01 with batch_size=64
# and you change to batch_size=256 (4x larger):

base_lr = 0.01
base_batch = 64
new_batch = 256

scaled_lr = base_lr * (new_batch / base_batch)
print(f'Scaled LR: {scaled_lr}')   # 0.04

# But use warm-up when scaling to very large batches
# to avoid instability at the start of training

Gradient Clipping: Preventing Exploding Gradients

When the LR is slightly too high or gradients are naturally large (common in RNNs), gradient clipping prevents parameter updates from being catastrophically large. torch.nn.utils.clip_grad_norm_ rescales the gradient vector to have a maximum L2 norm. The clipping happens after calling .backward() but before optimizer.step(). A max norm of 1.0 is a common default for recurrent networks.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.LSTM(4, 8, batch_first=True)
optimizer = optim.Adam(model.parameters(), lr=0.001)

x = torch.randn(16, 10, 4)   # batch=16, seq=10, features=4
out, _ = model(x)
loss = out.sum()
loss.backward()

# Clip gradients before optimizer step
total_norm = nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0
)
print(f'Gradient norm before clip: {total_norm:.4f}')
optimizer.step()

Finding LR with PyTorch Lightning or Manual Loop

Many practitioners use PyTorch Lightning's built-in LR finder, which automates the LR range test. If you are using raw PyTorch, implement the range test manually by exponentially increasing LR over 100 mini-batches and plotting loss vs LR on a log scale. The sweet spot is where loss decreases most steeply. Tools like torch-lr-finder package wrap this into a single function call for convenience.

# Manual LR finder sketch (pseudocode)
import math

# 1. Save initial model state
# torch.save(model.state_dict(), 'init.pt')

# 2. Sweep LR exponentially from 1e-7 to 1
start, end, steps = 1e-7, 1.0, 100
mult = (end / start) ** (1 / steps)
lr = start
for i, (X, y) in enumerate(train_loader):
    if i >= steps: break
    # train one step with current lr...
    lr *= mult

# 3. Plot lrs vs losses on log-linear scale
# 4. Pick LR where loss drops fastest
# 5. Restore initial model state

LR Summary and Practical Rules

The most important practical rules for learning rate: start with 1e-3 for Adam and 0.01 for SGD as default values. Always run a quick LR range test when using a new dataset or architecture. Use cosine annealing or ReduceLROnPlateau rather than a fixed LR for best convergence. Scale LR linearly with batch size. Always use gradient clipping for RNNs. The effort spent tuning LR pays off more than almost any other optimisation.

# Quick reference for default starting points
defaults = {
    'SGD':   {'lr': 0.01, 'momentum': 0.9},
    'Adam':  {'lr': 1e-3, 'betas': (0.9, 0.999)},
    'AdamW': {'lr': 1e-3, 'weight_decay': 0.01},
    'RMSprop': {'lr': 1e-4}
}

print('Default LR starting points:')
for opt, params in defaults.items():
    print(f'  {opt}: {params}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the learning rate is the most critical hyperparameter controlling convergence speed and stability, the LR range test finds a good LR quickly by sweeping from small to large and looking for the steepest loss decrease, and schedulers like CosineAnnealingLR and ReduceLROnPlateau automatically adjust LR during training for better final performance. Next up we explore batch normalisation for faster and more stable training.

常见问题解答

「学习率：最重要的超参数」课时是免费的吗？

是的 — 「学习率：最重要的超参数」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。

「学习率：最重要的超参数」这节课中我会学到什么？

您将运行学习率范围测试，绘制损失与 LR 的关系图，找出最佳范围，并应用 CosineAnnealingLR 调度器以避免训练停滞。 你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Machine Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。

「学习率：最重要的超参数」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Machine Learning Academy 课中编写并运行代码吗？

能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 学习率：最重要的超参数
  2. 批归一化：更稳定、更快速的训练
  3. 使用 Dropout 正则化防止过拟合
  4. 权重初始化：Xavier 与 He 初始化
← 返回 Machine Learning Academy