ReLU and Its Leaky & GELU Cousins
The default activation and modern variants.
ReLU and Its Leaky & GELU Cousins is a free Deep Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Meet ReLU
The most popular activation is ReLU: it keeps positive values and turns every negative one into zero. Simple and fast. ⚡
import torch.nn.functional as F
y = F.relu(x) # max(0, x), elementwiseWhy It Caught On
ReLU is cheap to compute and its gradient is a clean 1 for positives. That keeps signals flowing and makes deep nets train quickly.
The Math
ReLU is just max(0, x). Positive inputs pass straight through; negatives flatten to zero. That single hinge is the whole trick.
The Dying ReLU Problem
If a neuron always outputs zero, its gradient is zero too, so it stops learning forever. We call this a dead neuron. 💀
Leaky ReLU to the Rescue
Leaky ReLU lets a tiny slope through for negatives instead of a hard zero. That small leak keeps dead neurons alive.
y = F.leaky_relu(x, negative_slope=0.01)Parametric ReLU
PReLU goes further: it learns the negative slope during training instead of fixing it. The network tunes the leak itself.
Meet GELU
GELU smooths the ReLU corner into a soft curve. It gates inputs by how likely they are to be useful, not with a hard cutoff.
y = F.gelu(x)Why Transformers Love GELU
Modern models like transformers favor GELU because its smooth shape gives gentler gradients. That often means steadier training. 🤖
SiLU and Friends
SiLU, also called Swish, multiplies the input by its own sigmoid. Like GELU, it is smooth and frequently edges out plain ReLU.
A Sensible Default
Start with ReLU for hidden layers; it is fast and reliable. Reach for Leaky ReLU or GELU only if you see dead neurons or want extra smoothness.
Use It as a Layer
You can drop these in as modules inside a model, not just as functions. That makes them easy to chain in nn.Sequential.
import torch.nn as nn
net = nn.Sequential(nn.Linear(4, 8), nn.ReLU())Quick Check
Think about a neuron that always lands in the negative zone.
Recap
ReLU is the fast default, but it can let neurons die. Leaky ReLU, PReLU, and GELU smooth or leak the negatives to keep learning healthy. 🌟
Frequently asked questions
Is the “ReLU and Its Leaky & GELU Cousins” lesson free?
Yes — the full text of “ReLU and Its Leaky & GELU Cousins” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “ReLU and Its Leaky & GELU Cousins”?
The default activation and modern variants. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “ReLU and Its Leaky & GELU Cousins” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Nonlinearity Unlocks Real Power
- ReLU and Its Leaky & GELU Cousins
- Sigmoid & Tanh: Squashing to a Range
- Softmax for Probabilities