0PricingLogin▼
Machine Learning Academy · Lesson

Autograd: Automatic Differentiation for Backpropagation

Learners will define a scalar computation graph, call .backward(), and inspect .grad on leaf tensors to understand how gradients flow through the network.

What Is Automatic Differentiation?

Automatic differentiation (autograd) is the engine that computes gradients in PyTorch without requiring the programmer to derive them by hand. Unlike numerical differentiation (finite differences) or symbolic differentiation (algebra), autograd works by recording operations on tensors at runtime and replaying them in reverse. This makes it possible to train models of any architecture efficiently and correctly.

import torch

# A simple differentiable computation
x = torch.tensor(3.0, requires_grad=True)
y = x ** 2 + 2 * x + 1   # y = (x+1)^2

# Compute gradient dy/dx
y.backward()
print(x.grad)   # tensor(8.) because dy/dx = 2x+2 = 2*3+2 = 8

requires_grad: Enabling Gradient Tracking

Gradient tracking is disabled by default. Setting requires_grad=True tells PyTorch to record all operations on that tensor so gradients can be computed later. Leaf tensors (parameters) require gradients; intermediate tensors inherit gradient-tracking from their parents automatically. Model parameters created by nn.Module have requires_grad=True set automatically.

import torch

# Leaf tensor with gradient tracking
w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(0.5, requires_grad=True)

# Forward pass
x = torch.tensor(3.0)   # input, no grad needed
y_pred = w * x + b      # y = wx + b = 6.5

print(y_pred.requires_grad)  # True (inherited)
print(w.is_leaf)              # True
print(y_pred.is_leaf)         # False

All lessons in this course

  1. PyTorch Tensors: Creation, Operations, and GPU Transfer
  2. Autograd: Automatic Differentiation for Backpropagation
  3. Building a Feedforward Network with nn.Module
  4. Training Loop: Loss, Optimizer, and Epochs
← Back to Machine Learning Academy