Forward Caches, Backward Reuses
Why activations are stored during the forward pass.
Forward Caches, Backward Reuses is a free Deep Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Two Passes, One Goal
Training each batch runs two passes: a forward pass to predict and compute loss, then a backward pass to compute gradients. They work as a pair.
The Forward Pass Computes Values
Going forward, each layer turns its input into an output and sends it onward. By the end you have a prediction and a single loss number.
Backward Needs Forward Values
To compute a layer's gradient, the chain rule needs the very activations that layer produced going forward. Those values are not optional.
So We Cache Them
During the forward pass the network quietly caches each layer's inputs and outputs in memory, ready for the backward pass to grab. 💾
A Concrete Example
The derivative of a layer often reuses its own output. For a sigmoid, the gradient depends on the saved output value, so caching it saves recomputation.
sigmoid_grad = saved_output * (1 - saved_output)Backward Reuses the Cache
The backward pass walks layers in reverse, and at each one it pulls the matching cached values to multiply into the gradient. Nothing is recomputed.
Cache Costs Memory
Storing every activation is why training a deep net uses far more memory than just running it for predictions. Bigger nets need bigger caches.
Inference Skips the Cache
When you only need predictions, there is no backward pass, so PyTorch skips the cache entirely. That is why inference is lighter on memory.
with torch.no_grad():
preds = model(x)PyTorch Does This for You
Every operation on a tensor with requires_grad records what it needs into the computation graph, building the cache automatically as you go.
One Backward Frees It
By default, calling backward() consumes the cached graph and frees it. That is why a second backward() on the same graph raises an error.
loss.backward()Why This Design Wins
Caching forward values means each gradient is one cheap lookup-and-multiply instead of a fresh recomputation, making backprop fast and exact.
Quick Check
Let's check the cache idea.
Recap
The forward pass caches activations, and the backward pass reuses them to build gradients. That trade of memory for speed is what makes backprop practical. 💾
Frequently asked questions
Is the “Forward Caches, Backward Reuses” lesson free?
Yes — the full text of “Forward Caches, Backward Reuses” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Forward Caches, Backward Reuses”?
Why activations are stored during the forward pass. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Forward Caches, Backward Reuses” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- The Chain Rule, Layer by Layer
- Forward Caches, Backward Reuses
- Backprop a Tiny Net by Hand
- Vanishing & Exploding Gradients