Sigmoid & Tanh: Squashing to a Range
Bounded outputs and the vanishing-gradient trap.
Sigmoid & Tanh: Squashing to a Range is a free Deep Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Squashing Activations
Some activations squeeze any input into a fixed range. The two classics are sigmoid and tanh, and both bend big numbers gently inward. 🤏
Sigmoid's Range
Sigmoid maps every value into the open interval from 0 to 1. That makes its output read naturally as a probability.
import torch
y = torch.sigmoid(x) # outputs between 0 and 1Its S-Shaped Curve
Sigmoid has a smooth S shape: near zero it changes fast, but far out it flattens. Large inputs all map to nearly the same value.
Tanh's Range
Tanh is sigmoid's cousin, but it squashes inputs into the range from minus 1 to plus 1, centered neatly on zero.
y = torch.tanh(x) # outputs between -1 and 1Why Centering Helps
Because tanh is zero-centered, its outputs balance around zero. That often gives smoother, faster learning than sigmoid for hidden layers.
The Flat Tails
Out at the edges, both curves go almost flat. A flat region means a tiny slope, and a tiny slope means a tiny gradient.
Vanishing Gradients
When gradients shrink toward zero, early layers barely update. This vanishing gradient trap stalls deep networks during training. 😴
Why ReLU Took Over
This vanishing problem is exactly why ReLU replaced sigmoid and tanh in most hidden layers. ReLU keeps a healthy gradient for positives.
Where Sigmoid Still Wins
Sigmoid stays useful at the output of a binary classifier, where you genuinely want a single probability between 0 and 1.
Where Tanh Still Wins
Tanh still appears inside recurrent cells like LSTMs, where its bounded, zero-centered output helps keep the hidden state stable.
Choosing Wisely
Rule of thumb: avoid sigmoid and tanh in deep hidden stacks, but keep sigmoid for a single probability output. Match the tool to the job.
Quick Check
Recall what happens at the flat tails of these curves.
Recap
Sigmoid squashes to 0 to 1 and tanh to minus 1 to 1. Their flat tails cause vanishing gradients, so save them for outputs and special cells. 🎯
Frequently asked questions
Is the “Sigmoid & Tanh: Squashing to a Range” lesson free?
Yes — the full text of “Sigmoid & Tanh: Squashing to a Range” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Sigmoid & Tanh: Squashing to a Range”?
Bounded outputs and the vanishing-gradient trap. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Sigmoid & Tanh: Squashing to a Range” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Nonlinearity Unlocks Real Power
- ReLU and Its Leaky & GELU Cousins
- Sigmoid & Tanh: Squashing to a Range
- Softmax for Probabilities