Training Loop: Loss, Optimizer, and Epochs
Learners will write the PyTorch training loop: zero_grad, forward, compute CrossEntropyLoss, backward, optimizer.step, and track loss and accuracy over epochs.
Training Loop: Loss, Optimizer, and Epochs is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Four Steps of Every Training Loop
The PyTorch training loop has four mandatory steps that repeat for every batch: (1) zero gradients, (2) forward pass, (3) backward pass, and (4) optimizer step. Skipping or reordering these steps produces wrong results silently — gradients accumulate, parameters update incorrectly, or memory leaks. Internalising this pattern is the most important habit for training neural networks with PyTorch.
import torch
import torch.nn as nn
import torch.optim as optim
model = nn.Linear(2, 1)
optimizer = optim.SGD(model.parameters(), lr=0.01)
criterion = nn.MSELoss()
X = torch.randn(20, 2)
y = torch.randn(20, 1)
for step in range(1):
optimizer.zero_grad() # (1) zero grads
y_pred = model(X) # (2) forward
loss = criterion(y_pred, y) # (2) loss
loss.backward() # (3) backward
optimizer.step() # (4) update
print('Loss:', loss.item())Loss Functions: Choosing the Right Criterion
The loss function measures how wrong the model's predictions are. PyTorch's nn module provides many: nn.MSELoss for regression (mean squared error), nn.CrossEntropyLoss for multi-class classification (combines log-softmax and NLL), and nn.BCEWithLogitsLoss for binary classification (combines sigmoid and binary cross-entropy). Using the wrong loss for your task is a common beginner mistake that prevents learning.
import torch
import torch.nn as nn
# Regression
mse = nn.MSELoss()
y_pred = torch.tensor([2.5, 3.0])
y_true = torch.tensor([2.0, 3.5])
print('MSE:', mse(y_pred, y_true).item())
# Multi-class: logits (raw scores), not softmax
ce = nn.CrossEntropyLoss()
logits = torch.tensor([[2.0, 0.5, 1.0]])
labels = torch.tensor([0])
print('CE:', ce(logits, labels).item())Optimizers: SGD, Adam, and AdamW
An optimizer uses computed gradients to update model parameters. SGD (Stochastic Gradient Descent) is the classic optimizer; adding momentum accelerates convergence. Adam adapts the learning rate per parameter using first and second moment estimates, converging faster in practice. AdamW adds proper weight decay (L2 regularisation) and is the default choice for transformer models. All optimizers are in torch.optim.
import torch.optim as optim
import torch.nn as nn
model = nn.Linear(4, 2)
# Stochastic Gradient Descent with momentum
sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# Adam: adaptive learning rate
adam = optim.Adam(model.parameters(), lr=0.001,
betas=(0.9, 0.999))
# AdamW: Adam + proper weight decay
adamw = optim.AdamW(model.parameters(), lr=0.001,
weight_decay=0.01)Iterating Over Epochs and Batches
Training typically runs for many epochs — complete passes through the training dataset. Within each epoch you iterate over batches (subsets of the data). Using a DataLoader handles shuffling and batching automatically. Tracking average loss per epoch lets you monitor whether the model is converging. Printing every epoch (or every N batches) gives you visibility into training progress.
import torch
from torch.utils.data import TensorDataset, DataLoader
import torch.nn as nn
import torch.optim as optim
X = torch.randn(200, 4)
y = torch.randn(200, 1)
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=32, shuffle=True)
model = nn.Linear(4, 1)
optimizer = optim.Adam(model.parameters())
criterion = nn.MSELoss()
for epoch in range(3):
total_loss = 0
for X_batch, y_batch in loader:
optimizer.zero_grad()
pred = model(X_batch)
loss = criterion(pred, y_batch)
loss.backward()
optimizer.step()
total_loss += loss.item()
print(f'Epoch {epoch}: avg_loss={total_loss/len(loader):.4f}')DataLoader: Batching and Shuffling Data
DataLoader wraps a Dataset and provides an iterator that yields batches. Key parameters: batch_size controls how many samples per gradient update; shuffle=True randomises order each epoch (crucial for training); num_workers enables parallel data loading. For custom datasets, subclass torch.utils.data.Dataset and implement __len__ and __getitem__.
import torch
from torch.utils.data import Dataset, DataLoader
class MyDataset(Dataset):
def __init__(self, X, y):
self.X = X
self.y = y
def __len__(self):
return len(self.X)
def __getitem__(self, idx):
return self.X[idx], self.y[idx]
X = torch.randn(100, 5)
y = torch.randint(0, 3, (100,))
dataset = MyDataset(X, y)
loader = DataLoader(dataset, batch_size=16, shuffle=True)
X_batch, y_batch = next(iter(loader))
print(X_batch.shape) # torch.Size([16, 5])CrossEntropyLoss: Logits and Class Indices
nn.CrossEntropyLoss expects raw logits (unnormalised scores), not softmax probabilities. It internally applies log-softmax and computes negative log-likelihood. Target labels should be integer class indices (not one-hot vectors). This is the correct loss for multi-class classification and is more numerically stable than manually applying softmax then NLLLoss. The model's final layer should NOT have a softmax activation when using this loss.
import torch
import torch.nn as nn
criterion = nn.CrossEntropyLoss()
# Batch of 4 samples, 3 classes
logits = torch.tensor([
[2.1, 0.5, 0.3],
[0.1, 3.0, 0.2],
[0.8, 0.9, 2.0],
[1.5, 0.1, 0.4]
])
# True labels as class indices (not one-hot)
labels = torch.tensor([0, 1, 2, 0])
loss = criterion(logits, labels)
print('Loss:', loss.item())Tracking Accuracy During Training
Loss decreasing confirms the model is learning, but accuracy tells you whether predictions are actually correct. For multi-class classification, convert logits to predicted class with torch.argmax along the class dimension, then compare with true labels. Tracking both training loss and training accuracy, and optionally validation accuracy, gives a complete picture of learning dynamics.
import torch
def compute_accuracy(logits, labels):
preds = torch.argmax(logits, dim=1)
correct = (preds == labels).sum().item()
return correct / len(labels)
logits = torch.tensor([
[2.0, 0.1, 0.3],
[0.1, 0.2, 3.5],
[1.0, 0.5, 0.2]
])
labels = torch.tensor([0, 2, 0])
print('Accuracy:', compute_accuracy(logits, labels))
# 1.0 (all three correct)Validation Loop: Evaluating Without Updating
After each training epoch, run a validation loop to assess performance on unseen data. Wrap it with torch.no_grad() to disable gradient computation and call model.eval() to deactivate Dropout and BatchNorm stochasticity. Comparing training loss vs validation loss is the primary tool for detecting overfitting — when validation loss rises while training loss continues to fall, the model has overfit the training set.
import torch
def validate(model, val_loader, criterion):
model.eval()
total_loss = 0
with torch.no_grad():
for X_batch, y_batch in val_loader:
logits = model(X_batch)
loss = criterion(logits, y_batch)
total_loss += loss.item()
model.train() # restore training mode
return total_loss / len(val_loader)
# Usage inside training loop:
# val_loss = validate(model, val_loader, criterion)
# print(f'Val loss: {val_loss:.4f}')Learning Rate Scheduling
A fixed learning rate often leads to slow convergence or oscillation near the minimum. Learning rate schedulers automatically adjust the LR during training. StepLR reduces LR by a factor every N epochs; CosineAnnealingLR smoothly decays to near zero following a cosine curve; ReduceLROnPlateau decreases LR when validation loss stops improving. Schedulers are stepped after the optimizer, typically once per epoch.
import torch.optim as optim
import torch.nn as nn
model = nn.Linear(4, 2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
# Decay LR by 0.5 every 2 epochs
scheduler = optim.lr_scheduler.StepLR(
optimizer, step_size=2, gamma=0.5
)
for epoch in range(6):
# ... training code ...
scheduler.step()
print(f'Epoch {epoch}: LR={scheduler.get_last_lr()[0]}')
# LR: 0.01, 0.01, 0.005, 0.005, 0.0025, 0.0025Saving Checkpoints During Training
Training can take hours or days — saving checkpoints periodically protects against crashes and lets you resume from the best point. A good checkpoint stores the model state dict, optimizer state dict (contains momentum terms), epoch number, and best validation metric. Restoring the optimizer state allows training to resume exactly where it left off, including momentum buffers that affect subsequent updates.
import torch
def save_checkpoint(model, optimizer, epoch, loss, path):
torch.save({
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'loss': loss
}, path)
def load_checkpoint(model, optimizer, path):
ckpt = torch.load(path)
model.load_state_dict(ckpt['model_state_dict'])
optimizer.load_state_dict(ckpt['optimizer_state_dict'])
return ckpt['epoch'], ckpt['loss']Complete Training Loop: Putting It Together
A production-quality training loop combines all the pieces: DataLoader for batching, the four-step update per batch, a validation pass per epoch, learning rate scheduling, loss and accuracy tracking, and checkpoint saving. The code below shows the full pattern with a simple classification model. Adapting this template to any supervised learning problem requires changing only the model, dataset, and loss function.
import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader
X_tr = torch.randn(160, 4); y_tr = torch.randint(0, 3, (160,))
X_val = torch.randn(40, 4); y_val = torch.randint(0, 3, (40,))
tr_loader = DataLoader(TensorDataset(X_tr, y_tr), 32, shuffle=True)
val_loader = DataLoader(TensorDataset(X_val, y_val), 32)
model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 3))
optimizer = optim.Adam(model.parameters())
criterion = nn.CrossEntropyLoss()
for epoch in range(5):
model.train()
for Xb, yb in tr_loader:
optimizer.zero_grad()
loss = criterion(model(Xb), yb)
loss.backward(); optimizer.step()
model.eval()
with torch.no_grad():
val_loss = sum(criterion(model(Xb), yb).item()
for Xb, yb in val_loader) / len(val_loader)
print(f'Epoch {epoch}: val_loss={val_loss:.3f}')Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: the four-step training loop (zero_grad, forward, backward, step) is the foundation of all PyTorch training, DataLoader handles batching and shuffling automatically, and CrossEntropyLoss with logits is the standard choice for classification. Next up we explore learning rate scheduling and the impact of the learning rate on training dynamics.
Frequently asked questions
Is the “Training Loop: Loss, Optimizer, and Epochs” lesson free?
Yes — the full text of “Training Loop: Loss, Optimizer, and Epochs” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Training Loop: Loss, Optimizer, and Epochs”?
Learners will write the PyTorch training loop: zero_grad, forward, compute CrossEntropyLoss, backward, optimizer.step, and track loss and accuracy over epochs. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Training Loop: Loss, Optimizer, and Epochs” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- PyTorch Tensors: Creation, Operations, and GPU Transfer
- Autograd: Automatic Differentiation for Backpropagation
- Building a Feedforward Network with nn.Module
- Training Loop: Loss, Optimizer, and Epochs