Vanishing & Exploding Gradients
What breaks deep nets and early fixes.
Vanishing & Exploding Gradients is a free Deep Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Gradients Are a Product
Through many layers, backprop multiplies many local derivatives together. The size of that long product decides whether learning works or breaks.
Multiplying Small Numbers
If each link's derivative is below one, the product shrinks fast. After many layers the gradient becomes tiny, almost zero by the time it reaches early layers.
That Is Vanishing Gradients
When gradients shrink to near zero, early layers barely update and stop learning. This is the vanishing gradient problem that long stalled deep nets.
Multiplying Big Numbers
If each derivative is above one, the product blows up instead. Gradients grow huge as they travel back, and the weights lurch wildly.
That Is Exploding Gradients
Runaway gradients are the exploding gradient problem. Loss often jumps to NaN as updates overshoot far past any useful value. 💥
Sigmoid Made It Worse
Sigmoid and tanh squash inputs, so their derivatives stay well below one. Stacking them multiplied small numbers and made vanishing gradients common.
ReLU to the Rescue
ReLU has a derivative of exactly one for positive inputs, so it does not shrink the gradient. Switching to ReLU was a key early fix.
relu_grad = 1.0 if x > 0 else 0.0Careful Weight Initialization
Smart schemes like Xavier and He initialization scale starting weights so the gradient product stays near one across many layers.
Clip Exploding Gradients
For explosions, gradient clipping caps the gradient's size before the update, keeping a single huge step from wrecking the model.
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)Skip Connections Help
ResNet adds skip connections that let gradients flow straight back, sidestepping the long chain of multiplications that causes vanishing.
Why It All Matters
Keeping the gradient product near one is what lets very deep nets train at all. Every fix here exists to protect that balance.
Quick Check
Let's check the failure modes.
Recap
Long products of derivatives can vanish or explode. ReLU, careful init, clipping, and skip connections keep gradients healthy through deep nets. 💥
Frequently asked questions
Is the “Vanishing & Exploding Gradients” lesson free?
Yes — the full text of “Vanishing & Exploding Gradients” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Vanishing & Exploding Gradients”?
What breaks deep nets and early fixes. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Vanishing & Exploding Gradients” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- The Chain Rule, Layer by Layer
- Forward Caches, Backward Reuses
- Backprop a Tiny Net by Hand
- Vanishing & Exploding Gradients