Scaled Dot-Product & Multi-Head
Attend in parallel subspaces.
Scaled Dot-Product & Multi-Head is a free Deep Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Scaling Problem
When key vectors are long, dot-product scores grow huge and softmax becomes razor-sharp. That kills gradients, so we need to scale the scores down.
Divide by Root d
The fix is to divide each score by the square root of the key dimension. This keeps the numbers in a stable range before softmax.
scores = (q @ k.transpose(-2, -1)) / d_k ** 0.5Scaled Dot-Product Attention
Put it together: score, scale, softmax, then blend values. This four-step recipe is the famous scaled dot-product attention.
w = scores.softmax(dim=-1)
out = w @ vOne Head, One View
A single attention head learns just one way to relate tokens. But language has many relationships at once, so one head is limiting.
Many Heads in Parallel
Multi-head attention runs several attention heads side by side, each with its own Q, K, V projections, each attending in a different subspace.
Split the Dimension
You do not make the model wider. You split the model dimension across heads, so each head works on a smaller slice in parallel.
d_head = d_model // num_headsReshape into Heads
To run heads in parallel, you reshape Q, K, V to add a heads axis, then attend within each head independently.
q = q.view(B, T, H, d_head).transpose(1, 2)Different Patterns
One head may track grammar, another may link a pronoun to its noun. Multiple heads capture diverse patterns the same input contains.
Concatenate Heads
After each head produces its output, you concatenate them back into one vector of the original model dimension.
out = out.transpose(1, 2).reshape(B, T, d_model)Final Projection
A last output projection mixes the concatenated head results, letting the model combine what every head discovered.
out = self.W_o(out)Use the Built-in
In practice PyTorch ships nn.MultiheadAttention, so you rarely hand-roll the reshapes. Knowing the math still helps you debug.
attn = nn.MultiheadAttention(d_model, num_heads)Quick Check
Let's confirm why we scale the scores.
Recap
You saw how scaling stabilizes softmax and how multi-head attention splits the work across parallel heads, then concatenates and projects. Great progress!
Frequently asked questions
Is the “Scaled Dot-Product & Multi-Head” lesson free?
Yes — the full text of “Scaled Dot-Product & Multi-Head” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Scaled Dot-Product & Multi-Head”?
Attend in parallel subspaces. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Scaled Dot-Product & Multi-Head” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Self-Attention: Query, Key & Value
- Scaled Dot-Product & Multi-Head
- Positional Encoding for Order
- Stack a Transformer Encoder Block