Positional Encoding for Order
Inject sequence position into tokens.
Positional Encoding for Order is a free Deep Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Attention Ignores Order
Self-attention treats a sentence like a bag of tokens. Shuffle the words and the math barely changes, so the model has no built-in sense of order.
Order Carries Meaning
But order matters: "dog bites man" is not "man bites dog". We must hand the model some signal about each token's position.
Add, Do Not Append
The trick is to add a position vector directly to each token's embedding. Same shape, so attention sees content and place fused together.
x = token_emb + pos_embSinusoidal Encoding
The original transformer used fixed sine and cosine waves of different frequencies, one pattern per dimension, to mark each position.
Why Sine Waves
Mixing frequencies gives every position a unique fingerprint, and the smooth waves let the model generalize to lengths it never saw in training.
The Formula
Even dimensions use sine, odd dimensions use cosine, with the wavelength growing across dimensions. That is the whole encoding recipe.
pe[:, 0::2] = torch.sin(pos / div)
pe[:, 1::2] = torch.cos(pos / div)Relative Distance
A neat bonus: with sinusoids, the offset between two positions is easy to express, helping attention reason about relative distance.
Learned Positions
Modern models often skip sinusoids and use a learned embedding table, one trainable vector per position, just like word embeddings.
self.pos_emb = nn.Embedding(max_len, d_model)Fixed vs Learned
Fixed sinusoids extrapolate to new lengths for free; learned tables fit your data but are capped at the maximum length you trained on.
Rotary Encoding
Newer transformers favor rotary position encoding, which rotates query and key vectors by an angle tied to position, baking order into attention itself.
Where It Goes
Whatever scheme you pick, positional info is injected once at the input, before the first attention layer ever runs.
Quick Check
Let's check why positional encoding exists.
Recap
You learned that attention ignores order, so we add positional encodings, whether sinusoidal, learned, or rotary, to tell tokens where they sit. Well done!
Frequently asked questions
Is the “Positional Encoding for Order” lesson free?
Yes — the full text of “Positional Encoding for Order” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Positional Encoding for Order”?
Inject sequence position into tokens. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Positional Encoding for Order” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Self-Attention: Query, Key & Value
- Scaled Dot-Product & Multi-Head
- Positional Encoding for Order
- Stack a Transformer Encoder Block