Stack a Transformer Encoder Block
Attention plus feedforward and norms.
Stack a Transformer Encoder Block is a free Deep Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Building Block
A transformer is just one encoder block repeated. Learn the block and you understand the whole tower, from tiny models to giant ones.
Two Sub-Layers
Each block has two parts: a multi-head attention sub-layer, then a small feedforward network. Both wrapped with residuals and normalization.
Attention First
The block starts with self-attention, letting every token mix in context from the rest of the sequence before any further processing.
attn_out, _ = self.attn(x, x, x)The Residual Connection
A residual adds the sub-layer's input back to its output. This shortcut lets gradients flow and keeps deep stacks trainable.
x = x + attn_outLayer Normalization
After adding the residual, layer norm rescales each token's features to a stable distribution, steadying training across layers.
x = self.norm1(x)The Feedforward Net
Next comes a position-wise feedforward network: expand to a wider hidden size, apply a nonlinearity, then project back down.
ff = nn.Sequential(nn.Linear(d, 4*d), nn.GELU(), nn.Linear(4*d, d))Per-Token Processing
The feedforward layer treats each token independently. Attention shared information; this step refines each token on its own.
Second Residual and Norm
The feedforward output gets the same treatment: a residual add plus another layer norm, finishing the block.
x = self.norm2(x + ff(x))Stack Them Deep
Stack many identical blocks and the model builds richer representations layer by layer. Depth is where transformer power comes from.
layers = nn.ModuleList([Block(d) for _ in range(N)])Pre-Norm vs Post-Norm
Many modern models apply layer norm before each sub-layer instead of after. Pre-norm trains more stably in very deep stacks.
Use the Built-in
PyTorch gives you nn.TransformerEncoderLayer and nn.TransformerEncoder, so you can assemble a full stack in just a couple of lines.
layer = nn.TransformerEncoderLayer(d_model, nhead)
enc = nn.TransformerEncoder(layer, num_layers=6)Quick Check
Let's review the parts of an encoder block.
Recap
You assembled an encoder block: attention, residual, norm, feedforward, residual, norm. Stack it deep and you have a transformer. Amazing work!
Frequently asked questions
Is the “Stack a Transformer Encoder Block” lesson free?
Yes — the full text of “Stack a Transformer Encoder Block” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Stack a Transformer Encoder Block”?
Attention plus feedforward and norms. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Stack a Transformer Encoder Block” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Self-Attention: Query, Key & Value
- Scaled Dot-Product & Multi-Head
- Positional Encoding for Order
- Stack a Transformer Encoder Block