Multi-Head Attention and Positions
Many views plus order awareness.
Multi-Head Attention and Positions is a free NLP Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the NLP Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
One Head Is Limiting
A single attention head can track only one kind of relationship at a time. Real language needs many patterns noticed at once. 🧠
Many Heads, Many Views
Multi-head attention runs several attention computations in parallel, each with its own learned projections and its own focus.
What Each Head Learns
One head might track subject-verb links while another follows pronouns. Together they capture a far richer picture of the sentence.
Split the Dimensions
The model splits its vector size across heads, so each head works in a smaller subspace. The total compute stays roughly the same.
d_head = d_model // num_headsCombine the Heads
After each head produces an output, you concatenate them and pass the result through one more linear layer to mix the views.
out = concat(head_1, head_2, ...) @ W_oAttention Ignores Order
Self-attention treats input as a set, so by itself it cannot tell "dog bites man" from "man bites dog." It is order-blind.
Adding Position Information
To fix this, we inject a positional encoding into each word so the model knows where every token sits in the sequence.
Sinusoidal Encodings
The original Transformer uses fixed sine and cosine waves of different frequencies to give each position a unique, smooth signature.
pe[pos, 2i] = sin(pos / 10000 ** (2*i/d))Added, Not Appended
Position vectors are added to the word embeddings, not stuck on the end. So each token carries both meaning and place together.
x = token_embeddings + positional_encodingLearned Positions Too
Many modern models replace fixed waves with learned position embeddings, trained alongside everything else for flexibility.
Why Both Matter
Multi-head attention sees many relationships; positional encodings restore order. Together they let the Transformer truly understand sequences. ✨
Quick Check
Let us test positions and heads.
Recap
You saw how multi-head attention captures many relationships in parallel, while positional encodings give the model a sense of order. 🎯
Frequently asked questions
Is the “Multi-Head Attention and Positions” lesson free?
Yes — the full text of “Multi-Head Attention and Positions” is free to read here on the web, and the NLP Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the NLP Academy course, upgrade to CoddyKit PRO.
What will I learn in “Multi-Head Attention and Positions”?
Many views plus order awareness. You practise NLP Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start NLP Academy?
No prior experience is required. NLP Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Multi-Head Attention and Positions” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this NLP Academy lesson?
Yes. Every NLP Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- The Idea of Attention
- Self-Attention, Step by Step
- Multi-Head Attention and Positions
- Inside the Transformer Block