0Pricing
Machine Learning Academy · Pelajaran

Laju Pembelajaran: Hyperparameter Terpenting

Peserta akan menjalankan pengujian rentang laju pembelajaran, memplot loss terhadap LR, mengidentifikasi rentang optimal, dan menerapkan jadwal CosineAnnealingLR untuk menghindari plateau.

Laju Pembelajaran: Hyperparameter Terpenting adalah pelajaran Machine Learning Academy gratis di CoddyKit. Ini adalah pelajaran 1 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar Machine Learning Academy, dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.

Why Learning Rate Matters Most

The learning rate (LR) is the single hyperparameter that most affects whether a neural network trains successfully. It controls how large a step the optimizer takes in the direction of the negative gradient. Too large and the model diverges; too small and training takes forever or gets stuck. Unlike architecture choices, LR must be tuned almost every time you change the dataset, model size, or batch size.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(1, 1)

# Too large: diverges
optimizer_big = optim.SGD(model.parameters(), lr=10.0)

# Too small: barely moves
optimizer_small = optim.SGD(model.parameters(), lr=1e-6)

# Good: converges steadily
optimizer_good = optim.SGD(model.parameters(), lr=0.01)

print('LR comparison: 10.0, 1e-6, 0.01')

Effect of LR on Loss Curves

Different learning rates produce recognisable patterns in the loss curve. Too high: loss oscillates wildly or increases after a few steps. Too low: loss decreases extremely slowly, nearly flat. Just right: loss decreases smoothly and consistently. Plotting loss vs batch/epoch for a few representative LR values (e.g., 1e-4, 1e-3, 1e-2, 1e-1) before committing to a long training run is standard practice.

import torch
import torch.nn as nn
import torch.optim as optim

def train_one_lr(lr, steps=50):
    model = nn.Linear(1, 1)
    opt = optim.SGD(model.parameters(), lr=lr)
    X = torch.randn(100, 1)
    y = 2 * X + 1
    losses = []
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.MSELoss()(model(X), y)
        loss.backward(); opt.step()
        losses.append(loss.item())
    return losses[-1]

for lr in [1e-4, 1e-2, 0.1, 1.0]:
    final = train_one_lr(lr)
    print(f'LR={lr:.4f}: final_loss={final:.4f}')

The Learning Rate Range Test

The LR range test (popularised by Leslie Smith) finds a good LR automatically. Start with a very small LR and increase it exponentially over many mini-batches while recording the loss. The loss first decreases, then rises steeply when LR is too large. The optimal LR is roughly 10x smaller than where the loss starts rising. This test takes only a few minutes and eliminates the need for an expensive grid search over LR.

import torch
import torch.nn as nn
import torch.optim as optim
import math

def lr_range_test(model, loader, criterion, start_lr=1e-7, end_lr=10, num_iter=100):
    optimizer = optim.SGD(model.parameters(), lr=start_lr)
    lrs, losses = [], []
    mult = (end_lr / start_lr) ** (1 / num_iter)
    lr = start_lr
    for i, (X, y) in enumerate(loader):
        if i >= num_iter: break
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward(); optimizer.step()
        lrs.append(lr)
        losses.append(loss.item())
        lr *= mult
        for pg in optimizer.param_groups:
            pg['lr'] = lr
    return lrs, losses

Warm-Up: Starting Small and Growing

Learning rate warm-up starts training with a very small LR and gradually increases it to the target LR over the first few hundred steps or epochs. This prevents large unstable updates at the very beginning when weights are randomly initialised and gradients are noisy. Warm-up is especially important for Transformers and large batch training, where large initial steps can send the model to a poor region of loss landscape that is hard to escape.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

def warmup_lambda(current_step, warmup_steps=100):
    if current_step < warmup_steps:
        return current_step / warmup_steps
    return 1.0

scheduler = optim.lr_scheduler.LambdaLR(
    optimizer, lr_lambda=warmup_lambda
)

for step in range(5):
    scheduler.step()
    print(f'Step {step}: LR={optimizer.param_groups[0]["lr"]:.6f}')

Step Decay Scheduling with StepLR

StepLR reduces the learning rate by a multiplicative factor gamma every step_size epochs. For example, halving the LR every 10 epochs (gamma=0.5, step_size=10) is a common schedule for image classification. Step decay is simple to reason about and works well when you know roughly how many epochs the model needs to settle. The scheduler must be called after the optimizer step each epoch.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(
    optimizer, step_size=3, gamma=0.5
)

for epoch in range(9):
    # (training happens here)
    scheduler.step()
    print(f'After epoch {epoch}: LR={optimizer.param_groups[0]["lr"]:.4f}')
# 0.1 -> 0.1 -> 0.1 -> 0.05 -> 0.05 -> 0.05 -> 0.025 ...

Cosine Annealing: Smooth Decay to Zero

CosineAnnealingLR decays the LR following a cosine curve from the initial LR to a minimum (eta_min, default 0) over T_max steps. The cosine shape gives fast initial decrease with a soft landing near zero. CosineAnnealingWarmRestarts adds periodic restarts that escape local minima, creating a distinctive sawtooth LR pattern. Cosine schedules are among the most widely used in modern deep learning.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10, eta_min=1e-4
)

for epoch in range(10):
    scheduler.step()
    lr = optimizer.param_groups[0]['lr']
    print(f'Epoch {epoch}: LR={lr:.5f}')
# Smoothly decays from 0.1 to 0.0001 over 10 epochs

ReduceLROnPlateau: Adaptive Scheduling

ReduceLROnPlateau monitors a metric (usually validation loss) and reduces LR by a factor when the metric stops improving for a specified number of epochs (patience). This is the most adaptive scheduler because it responds to actual training dynamics rather than a predetermined schedule. It is especially effective when you are unsure how many epochs training will take or when training progress is uneven.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode='min',       # reduce when metric stops going down
    factor=0.5,       # multiply LR by 0.5
    patience=3,       # wait 3 epochs before reducing
    min_lr=1e-6
)

# Each epoch, pass the validation loss
for epoch in range(10):
    val_loss = 1.0 / (epoch + 1)   # simulated decreasing loss
    scheduler.step(val_loss)
    print(f'Epoch {epoch}: LR={optimizer.param_groups[0]["lr"]}')

Batch Size and Its Relationship to LR

Batch size and learning rate are tightly coupled. When you double the batch size, gradients are averaged over twice as many samples — they are less noisy. A commonly used heuristic is the linear scaling rule: multiply the LR by the same factor as the batch size increase. For example, doubling batch size from 64 to 128 suggests doubling LR as well. This rule works well in the moderate regime but breaks down for very large batches.

# Linear scaling rule: if base LR=0.01 with batch_size=64
# and you change to batch_size=256 (4x larger):

base_lr = 0.01
base_batch = 64
new_batch = 256

scaled_lr = base_lr * (new_batch / base_batch)
print(f'Scaled LR: {scaled_lr}')   # 0.04

# But use warm-up when scaling to very large batches
# to avoid instability at the start of training

Gradient Clipping: Preventing Exploding Gradients

When the LR is slightly too high or gradients are naturally large (common in RNNs), gradient clipping prevents parameter updates from being catastrophically large. torch.nn.utils.clip_grad_norm_ rescales the gradient vector to have a maximum L2 norm. The clipping happens after calling .backward() but before optimizer.step(). A max norm of 1.0 is a common default for recurrent networks.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.LSTM(4, 8, batch_first=True)
optimizer = optim.Adam(model.parameters(), lr=0.001)

x = torch.randn(16, 10, 4)   # batch=16, seq=10, features=4
out, _ = model(x)
loss = out.sum()
loss.backward()

# Clip gradients before optimizer step
total_norm = nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0
)
print(f'Gradient norm before clip: {total_norm:.4f}')
optimizer.step()

Finding LR with PyTorch Lightning or Manual Loop

Many practitioners use PyTorch Lightning's built-in LR finder, which automates the LR range test. If you are using raw PyTorch, implement the range test manually by exponentially increasing LR over 100 mini-batches and plotting loss vs LR on a log scale. The sweet spot is where loss decreases most steeply. Tools like torch-lr-finder package wrap this into a single function call for convenience.

# Manual LR finder sketch (pseudocode)
import math

# 1. Save initial model state
# torch.save(model.state_dict(), 'init.pt')

# 2. Sweep LR exponentially from 1e-7 to 1
start, end, steps = 1e-7, 1.0, 100
mult = (end / start) ** (1 / steps)
lr = start
for i, (X, y) in enumerate(train_loader):
    if i >= steps: break
    # train one step with current lr...
    lr *= mult

# 3. Plot lrs vs losses on log-linear scale
# 4. Pick LR where loss drops fastest
# 5. Restore initial model state

LR Summary and Practical Rules

The most important practical rules for learning rate: start with 1e-3 for Adam and 0.01 for SGD as default values. Always run a quick LR range test when using a new dataset or architecture. Use cosine annealing or ReduceLROnPlateau rather than a fixed LR for best convergence. Scale LR linearly with batch size. Always use gradient clipping for RNNs. The effort spent tuning LR pays off more than almost any other optimisation.

# Quick reference for default starting points
defaults = {
    'SGD':   {'lr': 0.01, 'momentum': 0.9},
    'Adam':  {'lr': 1e-3, 'betas': (0.9, 0.999)},
    'AdamW': {'lr': 1e-3, 'weight_decay': 0.01},
    'RMSprop': {'lr': 1e-4}
}

print('Default LR starting points:')
for opt, params in defaults.items():
    print(f'  {opt}: {params}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the learning rate is the most critical hyperparameter controlling convergence speed and stability, the LR range test finds a good LR quickly by sweeping from small to large and looking for the steepest loss decrease, and schedulers like CosineAnnealingLR and ReduceLROnPlateau automatically adjust LR during training for better final performance. Next up we explore batch normalisation for faster and more stable training.

Pertanyaan yang Sering Diajukan

Apakah pelajaran “Laju Pembelajaran: Hyperparameter Terpenting” gratis?

Ya — teks lengkap “Laju Pembelajaran: Hyperparameter Terpenting” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus Machine Learning Academy, upgrade ke CoddyKit PRO. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Apa yang akan aku pelajari di “Laju Pembelajaran: Hyperparameter Terpenting”?

Peserta akan menjalankan pengujian rentang laju pembelajaran, memplot loss terhadap LR, mengidentifikasi rentang optimal, dan menerapkan jadwal CosineAnnealingLR untuk menghindari plateau. Kamu berlatih Machine Learning Academy dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.

Apakah aku perlu pengalaman untuk memulai Machine Learning Academy?

Tidak diperlukan pengalaman sebelumnya. Machine Learning Academy di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 1 dari 4.

Berapa lama pelajaran “Laju Pembelajaran: Hyperparameter Terpenting” memakan waktu?

Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.

Bisakah aku menulis dan menjalankan kode dalam pelajaran Machine Learning Academy ini?

Ya. Setiap pelajaran Machine Learning Academy menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.

Semua pelajaran dalam kursus ini

  1. Laju Pembelajaran: Hyperparameter Terpenting
  2. Normalisasi Batch: Pelatihan yang Stabil dan Lebih Cepat
  3. Regularisasi Dropout untuk Mencegah Overfitting
  4. Inisialisasi Bobot: Inisialisasi Xavier dan He
← Kembali ke Machine Learning Academy