0Pricing
Machine Learning Academy · Lesson

Learning Rate: The Most Important Hyperparameter

Learners will run a learning-rate range test, plot loss vs LR, identify the optimal range, and apply a CosineAnnealingLR schedule to avoid plateaus.

Learning Rate: The Most Important Hyperparameter is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Learning Rate Matters Most

The learning rate (LR) is the single hyperparameter that most affects whether a neural network trains successfully. It controls how large a step the optimizer takes in the direction of the negative gradient. Too large and the model diverges; too small and training takes forever or gets stuck. Unlike architecture choices, LR must be tuned almost every time you change the dataset, model size, or batch size.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(1, 1)

# Too large: diverges
optimizer_big = optim.SGD(model.parameters(), lr=10.0)

# Too small: barely moves
optimizer_small = optim.SGD(model.parameters(), lr=1e-6)

# Good: converges steadily
optimizer_good = optim.SGD(model.parameters(), lr=0.01)

print('LR comparison: 10.0, 1e-6, 0.01')

Effect of LR on Loss Curves

Different learning rates produce recognisable patterns in the loss curve. Too high: loss oscillates wildly or increases after a few steps. Too low: loss decreases extremely slowly, nearly flat. Just right: loss decreases smoothly and consistently. Plotting loss vs batch/epoch for a few representative LR values (e.g., 1e-4, 1e-3, 1e-2, 1e-1) before committing to a long training run is standard practice.

import torch
import torch.nn as nn
import torch.optim as optim

def train_one_lr(lr, steps=50):
    model = nn.Linear(1, 1)
    opt = optim.SGD(model.parameters(), lr=lr)
    X = torch.randn(100, 1)
    y = 2 * X + 1
    losses = []
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.MSELoss()(model(X), y)
        loss.backward(); opt.step()
        losses.append(loss.item())
    return losses[-1]

for lr in [1e-4, 1e-2, 0.1, 1.0]:
    final = train_one_lr(lr)
    print(f'LR={lr:.4f}: final_loss={final:.4f}')

The Learning Rate Range Test

The LR range test (popularised by Leslie Smith) finds a good LR automatically. Start with a very small LR and increase it exponentially over many mini-batches while recording the loss. The loss first decreases, then rises steeply when LR is too large. The optimal LR is roughly 10x smaller than where the loss starts rising. This test takes only a few minutes and eliminates the need for an expensive grid search over LR.

import torch
import torch.nn as nn
import torch.optim as optim
import math

def lr_range_test(model, loader, criterion, start_lr=1e-7, end_lr=10, num_iter=100):
    optimizer = optim.SGD(model.parameters(), lr=start_lr)
    lrs, losses = [], []
    mult = (end_lr / start_lr) ** (1 / num_iter)
    lr = start_lr
    for i, (X, y) in enumerate(loader):
        if i >= num_iter: break
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward(); optimizer.step()
        lrs.append(lr)
        losses.append(loss.item())
        lr *= mult
        for pg in optimizer.param_groups:
            pg['lr'] = lr
    return lrs, losses

Warm-Up: Starting Small and Growing

Learning rate warm-up starts training with a very small LR and gradually increases it to the target LR over the first few hundred steps or epochs. This prevents large unstable updates at the very beginning when weights are randomly initialised and gradients are noisy. Warm-up is especially important for Transformers and large batch training, where large initial steps can send the model to a poor region of loss landscape that is hard to escape.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

def warmup_lambda(current_step, warmup_steps=100):
    if current_step < warmup_steps:
        return current_step / warmup_steps
    return 1.0

scheduler = optim.lr_scheduler.LambdaLR(
    optimizer, lr_lambda=warmup_lambda
)

for step in range(5):
    scheduler.step()
    print(f'Step {step}: LR={optimizer.param_groups[0]["lr"]:.6f}')

Step Decay Scheduling with StepLR

StepLR reduces the learning rate by a multiplicative factor gamma every step_size epochs. For example, halving the LR every 10 epochs (gamma=0.5, step_size=10) is a common schedule for image classification. Step decay is simple to reason about and works well when you know roughly how many epochs the model needs to settle. The scheduler must be called after the optimizer step each epoch.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(
    optimizer, step_size=3, gamma=0.5
)

for epoch in range(9):
    # (training happens here)
    scheduler.step()
    print(f'After epoch {epoch}: LR={optimizer.param_groups[0]["lr"]:.4f}')
# 0.1 -> 0.1 -> 0.1 -> 0.05 -> 0.05 -> 0.05 -> 0.025 ...

Cosine Annealing: Smooth Decay to Zero

CosineAnnealingLR decays the LR following a cosine curve from the initial LR to a minimum (eta_min, default 0) over T_max steps. The cosine shape gives fast initial decrease with a soft landing near zero. CosineAnnealingWarmRestarts adds periodic restarts that escape local minima, creating a distinctive sawtooth LR pattern. Cosine schedules are among the most widely used in modern deep learning.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10, eta_min=1e-4
)

for epoch in range(10):
    scheduler.step()
    lr = optimizer.param_groups[0]['lr']
    print(f'Epoch {epoch}: LR={lr:.5f}')
# Smoothly decays from 0.1 to 0.0001 over 10 epochs

ReduceLROnPlateau: Adaptive Scheduling

ReduceLROnPlateau monitors a metric (usually validation loss) and reduces LR by a factor when the metric stops improving for a specified number of epochs (patience). This is the most adaptive scheduler because it responds to actual training dynamics rather than a predetermined schedule. It is especially effective when you are unsure how many epochs training will take or when training progress is uneven.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode='min',       # reduce when metric stops going down
    factor=0.5,       # multiply LR by 0.5
    patience=3,       # wait 3 epochs before reducing
    min_lr=1e-6
)

# Each epoch, pass the validation loss
for epoch in range(10):
    val_loss = 1.0 / (epoch + 1)   # simulated decreasing loss
    scheduler.step(val_loss)
    print(f'Epoch {epoch}: LR={optimizer.param_groups[0]["lr"]}')

Batch Size and Its Relationship to LR

Batch size and learning rate are tightly coupled. When you double the batch size, gradients are averaged over twice as many samples — they are less noisy. A commonly used heuristic is the linear scaling rule: multiply the LR by the same factor as the batch size increase. For example, doubling batch size from 64 to 128 suggests doubling LR as well. This rule works well in the moderate regime but breaks down for very large batches.

# Linear scaling rule: if base LR=0.01 with batch_size=64
# and you change to batch_size=256 (4x larger):

base_lr = 0.01
base_batch = 64
new_batch = 256

scaled_lr = base_lr * (new_batch / base_batch)
print(f'Scaled LR: {scaled_lr}')   # 0.04

# But use warm-up when scaling to very large batches
# to avoid instability at the start of training

Gradient Clipping: Preventing Exploding Gradients

When the LR is slightly too high or gradients are naturally large (common in RNNs), gradient clipping prevents parameter updates from being catastrophically large. torch.nn.utils.clip_grad_norm_ rescales the gradient vector to have a maximum L2 norm. The clipping happens after calling .backward() but before optimizer.step(). A max norm of 1.0 is a common default for recurrent networks.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.LSTM(4, 8, batch_first=True)
optimizer = optim.Adam(model.parameters(), lr=0.001)

x = torch.randn(16, 10, 4)   # batch=16, seq=10, features=4
out, _ = model(x)
loss = out.sum()
loss.backward()

# Clip gradients before optimizer step
total_norm = nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0
)
print(f'Gradient norm before clip: {total_norm:.4f}')
optimizer.step()

Finding LR with PyTorch Lightning or Manual Loop

Many practitioners use PyTorch Lightning's built-in LR finder, which automates the LR range test. If you are using raw PyTorch, implement the range test manually by exponentially increasing LR over 100 mini-batches and plotting loss vs LR on a log scale. The sweet spot is where loss decreases most steeply. Tools like torch-lr-finder package wrap this into a single function call for convenience.

# Manual LR finder sketch (pseudocode)
import math

# 1. Save initial model state
# torch.save(model.state_dict(), 'init.pt')

# 2. Sweep LR exponentially from 1e-7 to 1
start, end, steps = 1e-7, 1.0, 100
mult = (end / start) ** (1 / steps)
lr = start
for i, (X, y) in enumerate(train_loader):
    if i >= steps: break
    # train one step with current lr...
    lr *= mult

# 3. Plot lrs vs losses on log-linear scale
# 4. Pick LR where loss drops fastest
# 5. Restore initial model state

LR Summary and Practical Rules

The most important practical rules for learning rate: start with 1e-3 for Adam and 0.01 for SGD as default values. Always run a quick LR range test when using a new dataset or architecture. Use cosine annealing or ReduceLROnPlateau rather than a fixed LR for best convergence. Scale LR linearly with batch size. Always use gradient clipping for RNNs. The effort spent tuning LR pays off more than almost any other optimisation.

# Quick reference for default starting points
defaults = {
    'SGD':   {'lr': 0.01, 'momentum': 0.9},
    'Adam':  {'lr': 1e-3, 'betas': (0.9, 0.999)},
    'AdamW': {'lr': 1e-3, 'weight_decay': 0.01},
    'RMSprop': {'lr': 1e-4}
}

print('Default LR starting points:')
for opt, params in defaults.items():
    print(f'  {opt}: {params}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the learning rate is the most critical hyperparameter controlling convergence speed and stability, the LR range test finds a good LR quickly by sweeping from small to large and looking for the steepest loss decrease, and schedulers like CosineAnnealingLR and ReduceLROnPlateau automatically adjust LR during training for better final performance. Next up we explore batch normalisation for faster and more stable training.

Frequently asked questions

Is the “Learning Rate: The Most Important Hyperparameter” lesson free?

Yes — the full text of “Learning Rate: The Most Important Hyperparameter” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Learning Rate: The Most Important Hyperparameter”?

Learners will run a learning-rate range test, plot loss vs LR, identify the optimal range, and apply a CosineAnnealingLR schedule to avoid plateaus. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Learning Rate: The Most Important Hyperparameter” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Learning Rate: The Most Important Hyperparameter
  2. Batch Normalisation: Stable and Faster Training
  3. Dropout Regularisation to Prevent Overfitting
  4. Weight Initialisation: Xavier and He Initialisation
← Back to Machine Learning Academy