ResNet: Skip Connections Go Deep
Residuals that beat vanishing gradients.
ResNet: Skip Connections Go Deep is a free Deep Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Deeper Got Worse
Surprisingly, plain nets past a point trained worse as they got deeper. More layers should not hurt, yet they did. Something was broken.
The Real Culprit
Gradients shrank as they flowed back through many layers, a problem called vanishing gradients. Deep nets stopped learning.
The Skip Connection
ResNets fix is elegant: add a shortcut that lets the input jump past a block and add itself to the output.
Learn the Residual
Instead of learning the full mapping, each block only learns the residual, the small change to add on top of the input.
Easy to Stay Still
If a block is not useful, it can just learn zeros and pass the input straight through. Doing nothing becomes effortless.
A Gradient Highway
The shortcut gives gradients a clean path backward, so even very deep networks keep learning.
The Residual Block
A block computes some layers, then adds the original input back. That single add is the whole trick.
def forward(self, x):
out = self.conv_layers(x)
return self.relu(out + x)Going Very Deep
With shortcuts, ResNet trained 50, 101, even 152 layers and won ImageNet 2015. Depth finally paid off.
The Bottleneck Block
Deeper ResNets use a 1x1, 3x3, 1x1 bottleneck to cut compute while keeping representational power.
Identity Everywhere
This residual idea spread far beyond vision. Transformers and many modern nets rely on the same identity shortcut.
A Tiny ResNet Block
You can sketch the core in a few lines: convolve, then add the input back before the activation. Here is the shape of it.
out = bn(conv(x))
out = relu(out + x) # the skip connectionQuick Check
Focus on what the skip connection actually achieves.
Recap: Shortcuts Win
ResNets skip connections made extreme depth trainable by letting blocks learn residuals. One add changed everything. Great progress!
Frequently asked questions
Is the “ResNet: Skip Connections Go Deep” lesson free?
Yes — the full text of “ResNet: Skip Connections Go Deep” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “ResNet: Skip Connections Go Deep”?
Residuals that beat vanishing gradients. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “ResNet: Skip Connections Go Deep” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- LeNet & AlexNet: The First Wins
- VGG: Stacks of Small Filters
- ResNet: Skip Connections Go Deep
- Load torchvision Models