0Pricing
Machine Learning Academy · Lección

Tasa de aprendizaje: el hiperparámetro más importante

Ejecutará una prueba de rango de tasas de aprendizaje, representará la pérdida frente a la tasa de aprendizaje, identificará el intervalo óptimo y aplicará una planificación CosineAnnealingLR para evitar estancamientos.

Tasa de aprendizaje: el hiperparámetro más importante es una lección gratuita de Machine Learning Academy en CoddyKit. Esta es la lección 1 de 4. Puedes leer la lección completa abajo gratuitamente — luego la practicas en el navegador con un editor de código integrado y un tutor de IA 24/7. Forma parte de la ruta de aprendizaje de Machine Learning Academy, y tu progreso se sincroniza en la web y la app de CoddyKit. El curso de Machine Learning Academy incluye 4 lecciones en total.

Partes de esta lección aún no han sido traducidas y se muestran en inglés.

Why Learning Rate Matters Most

The learning rate (LR) is the single hyperparameter that most affects whether a neural network trains successfully. It controls how large a step the optimizer takes in the direction of the negative gradient. Too large and the model diverges; too small and training takes forever or gets stuck. Unlike architecture choices, LR must be tuned almost every time you change the dataset, model size, or batch size.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(1, 1)

# Too large: diverges
optimizer_big = optim.SGD(model.parameters(), lr=10.0)

# Too small: barely moves
optimizer_small = optim.SGD(model.parameters(), lr=1e-6)

# Good: converges steadily
optimizer_good = optim.SGD(model.parameters(), lr=0.01)

print('LR comparison: 10.0, 1e-6, 0.01')

Effect of LR on Loss Curves

Different learning rates produce recognisable patterns in the loss curve. Too high: loss oscillates wildly or increases after a few steps. Too low: loss decreases extremely slowly, nearly flat. Just right: loss decreases smoothly and consistently. Plotting loss vs batch/epoch for a few representative LR values (e.g., 1e-4, 1e-3, 1e-2, 1e-1) before committing to a long training run is standard practice.

import torch
import torch.nn as nn
import torch.optim as optim

def train_one_lr(lr, steps=50):
    model = nn.Linear(1, 1)
    opt = optim.SGD(model.parameters(), lr=lr)
    X = torch.randn(100, 1)
    y = 2 * X + 1
    losses = []
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.MSELoss()(model(X), y)
        loss.backward(); opt.step()
        losses.append(loss.item())
    return losses[-1]

for lr in [1e-4, 1e-2, 0.1, 1.0]:
    final = train_one_lr(lr)
    print(f'LR={lr:.4f}: final_loss={final:.4f}')

The Learning Rate Range Test

The LR range test (popularised by Leslie Smith) finds a good LR automatically. Start with a very small LR and increase it exponentially over many mini-batches while recording the loss. The loss first decreases, then rises steeply when LR is too large. The optimal LR is roughly 10x smaller than where the loss starts rising. This test takes only a few minutes and eliminates the need for an expensive grid search over LR.

import torch
import torch.nn as nn
import torch.optim as optim
import math

def lr_range_test(model, loader, criterion, start_lr=1e-7, end_lr=10, num_iter=100):
    optimizer = optim.SGD(model.parameters(), lr=start_lr)
    lrs, losses = [], []
    mult = (end_lr / start_lr) ** (1 / num_iter)
    lr = start_lr
    for i, (X, y) in enumerate(loader):
        if i >= num_iter: break
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward(); optimizer.step()
        lrs.append(lr)
        losses.append(loss.item())
        lr *= mult
        for pg in optimizer.param_groups:
            pg['lr'] = lr
    return lrs, losses

Warm-Up: Starting Small and Growing

Learning rate warm-up starts training with a very small LR and gradually increases it to the target LR over the first few hundred steps or epochs. This prevents large unstable updates at the very beginning when weights are randomly initialised and gradients are noisy. Warm-up is especially important for Transformers and large batch training, where large initial steps can send the model to a poor region of loss landscape that is hard to escape.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

def warmup_lambda(current_step, warmup_steps=100):
    if current_step < warmup_steps:
        return current_step / warmup_steps
    return 1.0

scheduler = optim.lr_scheduler.LambdaLR(
    optimizer, lr_lambda=warmup_lambda
)

for step in range(5):
    scheduler.step()
    print(f'Step {step}: LR={optimizer.param_groups[0]["lr"]:.6f}')

Step Decay Scheduling with StepLR

StepLR reduces the learning rate by a multiplicative factor gamma every step_size epochs. For example, halving the LR every 10 epochs (gamma=0.5, step_size=10) is a common schedule for image classification. Step decay is simple to reason about and works well when you know roughly how many epochs the model needs to settle. The scheduler must be called after the optimizer step each epoch.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(
    optimizer, step_size=3, gamma=0.5
)

for epoch in range(9):
    # (training happens here)
    scheduler.step()
    print(f'After epoch {epoch}: LR={optimizer.param_groups[0]["lr"]:.4f}')
# 0.1 -> 0.1 -> 0.1 -> 0.05 -> 0.05 -> 0.05 -> 0.025 ...

Cosine Annealing: Smooth Decay to Zero

CosineAnnealingLR decays the LR following a cosine curve from the initial LR to a minimum (eta_min, default 0) over T_max steps. The cosine shape gives fast initial decrease with a soft landing near zero. CosineAnnealingWarmRestarts adds periodic restarts that escape local minima, creating a distinctive sawtooth LR pattern. Cosine schedules are among the most widely used in modern deep learning.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10, eta_min=1e-4
)

for epoch in range(10):
    scheduler.step()
    lr = optimizer.param_groups[0]['lr']
    print(f'Epoch {epoch}: LR={lr:.5f}')
# Smoothly decays from 0.1 to 0.0001 over 10 epochs

ReduceLROnPlateau: Adaptive Scheduling

ReduceLROnPlateau monitors a metric (usually validation loss) and reduces LR by a factor when the metric stops improving for a specified number of epochs (patience). This is the most adaptive scheduler because it responds to actual training dynamics rather than a predetermined schedule. It is especially effective when you are unsure how many epochs training will take or when training progress is uneven.

import torch.optim as optim
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode='min',       # reduce when metric stops going down
    factor=0.5,       # multiply LR by 0.5
    patience=3,       # wait 3 epochs before reducing
    min_lr=1e-6
)

# Each epoch, pass the validation loss
for epoch in range(10):
    val_loss = 1.0 / (epoch + 1)   # simulated decreasing loss
    scheduler.step(val_loss)
    print(f'Epoch {epoch}: LR={optimizer.param_groups[0]["lr"]}')

Batch Size and Its Relationship to LR

Batch size and learning rate are tightly coupled. When you double the batch size, gradients are averaged over twice as many samples — they are less noisy. A commonly used heuristic is the linear scaling rule: multiply the LR by the same factor as the batch size increase. For example, doubling batch size from 64 to 128 suggests doubling LR as well. This rule works well in the moderate regime but breaks down for very large batches.

# Linear scaling rule: if base LR=0.01 with batch_size=64
# and you change to batch_size=256 (4x larger):

base_lr = 0.01
base_batch = 64
new_batch = 256

scaled_lr = base_lr * (new_batch / base_batch)
print(f'Scaled LR: {scaled_lr}')   # 0.04

# But use warm-up when scaling to very large batches
# to avoid instability at the start of training

Gradient Clipping: Preventing Exploding Gradients

When the LR is slightly too high or gradients are naturally large (common in RNNs), gradient clipping prevents parameter updates from being catastrophically large. torch.nn.utils.clip_grad_norm_ rescales the gradient vector to have a maximum L2 norm. The clipping happens after calling .backward() but before optimizer.step(). A max norm of 1.0 is a common default for recurrent networks.

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.LSTM(4, 8, batch_first=True)
optimizer = optim.Adam(model.parameters(), lr=0.001)

x = torch.randn(16, 10, 4)   # batch=16, seq=10, features=4
out, _ = model(x)
loss = out.sum()
loss.backward()

# Clip gradients before optimizer step
total_norm = nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0
)
print(f'Gradient norm before clip: {total_norm:.4f}')
optimizer.step()

Finding LR with PyTorch Lightning or Manual Loop

Many practitioners use PyTorch Lightning's built-in LR finder, which automates the LR range test. If you are using raw PyTorch, implement the range test manually by exponentially increasing LR over 100 mini-batches and plotting loss vs LR on a log scale. The sweet spot is where loss decreases most steeply. Tools like torch-lr-finder package wrap this into a single function call for convenience.

# Manual LR finder sketch (pseudocode)
import math

# 1. Save initial model state
# torch.save(model.state_dict(), 'init.pt')

# 2. Sweep LR exponentially from 1e-7 to 1
start, end, steps = 1e-7, 1.0, 100
mult = (end / start) ** (1 / steps)
lr = start
for i, (X, y) in enumerate(train_loader):
    if i >= steps: break
    # train one step with current lr...
    lr *= mult

# 3. Plot lrs vs losses on log-linear scale
# 4. Pick LR where loss drops fastest
# 5. Restore initial model state

LR Summary and Practical Rules

The most important practical rules for learning rate: start with 1e-3 for Adam and 0.01 for SGD as default values. Always run a quick LR range test when using a new dataset or architecture. Use cosine annealing or ReduceLROnPlateau rather than a fixed LR for best convergence. Scale LR linearly with batch size. Always use gradient clipping for RNNs. The effort spent tuning LR pays off more than almost any other optimisation.

# Quick reference for default starting points
defaults = {
    'SGD':   {'lr': 0.01, 'momentum': 0.9},
    'Adam':  {'lr': 1e-3, 'betas': (0.9, 0.999)},
    'AdamW': {'lr': 1e-3, 'weight_decay': 0.01},
    'RMSprop': {'lr': 1e-4}
}

print('Default LR starting points:')
for opt, params in defaults.items():
    print(f'  {opt}: {params}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: the learning rate is the most critical hyperparameter controlling convergence speed and stability, the LR range test finds a good LR quickly by sweeping from small to large and looking for the steepest loss decrease, and schedulers like CosineAnnealingLR and ReduceLROnPlateau automatically adjust LR during training for better final performance. Next up we explore batch normalisation for faster and more stable training.

Preguntas frecuentes

¿La lección «Tasa de aprendizaje: el hiperparámetro más importante» es gratis?

Sí — el texto completo de «Tasa de aprendizaje: el hiperparámetro más importante» es gratis para leer aquí en la web. Para practicarla de forma interactiva (editor de código integrado y tutor de IA 24/7) y desbloquear el resto del curso de Machine Learning Academy, actualiza a CoddyKit PRO. El curso de Machine Learning Academy incluye 4 lecciones en total.

¿Qué aprenderé en «Tasa de aprendizaje: el hiperparámetro más importante»?

Ejecutará una prueba de rango de tasas de aprendizaje, representará la pérdida frente a la tasa de aprendizaje, identificará el intervalo óptimo y aplicará una planificación CosineAnnealingLR para ev… Practicas Machine Learning Academy con código real que ejecutas directamente en el navegador, y un tutor de IA 24/7 responde tus preguntas mientras trabajas en la lección.

¿Necesito experiencia previa para empezar Machine Learning Academy?

No se requiere experiencia previa. Machine Learning Academy en CoddyKit está estructurado para principiantes hasta estudiantes avanzados, así que puedes empezar aquí o desde el inicio y avanzar a tu ritmo. Esta es la lección 1 de 4.

¿Cuánto tiempo toma la lección «Tasa de aprendizaje: el hiperparámetro más importante»?

La mayoría de las lecciones de CoddyKit toman alrededor de 5–10 minutos. Cada una es compacta e interactiva, así que avanzas constantemente y retomas exactamente por donde dejaste en la web y la app.

¿Puedo escribir y ejecutar código en esta lección de Machine Learning Academy?

Sí. Cada lección de Machine Learning Academy incluye un editor de código integrado, así que escribes y ejecutas código real directamente en tu navegador y obtienes retroalimentación instantánea de IA — sin configuración local necesaria.

Todas las lecciones de este curso

  1. Tasa de aprendizaje: el hiperparámetro más importante
  2. Normalización por lotes: entrenamiento estable y más rápido
  3. Regularización con dropout para evitar el sobreajuste
  4. Inicialización de pesos: inicialización de Xavier y He
← Volver a Machine Learning Academy