Self-Attention: Query, Key & Value
Let every token look at every other.
Self-Attention: Query, Key & Value is a free Deep Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Tokens That Talk
In a sequence, the meaning of one word depends on others. Self-attention lets every token look at every other token to gather the context it needs.
Three Roles per Token
Each token plays three roles: a query that asks, a key that answers, and a value that carries content. These come from the same word, used three ways.
The Query
A token's query describes what it is looking for. Think of it as the question this word is asking about the rest of the sentence.
The Key
Every token also exposes a key, a label advertising what it offers. A query is compared against all keys to find good matches.
The Value
Once a match is found, the value is the actual information that gets passed along. Keys decide how much, values decide what.
Make Q, K, V
You build queries, keys, and values by projecting the input through three learned linear layers. Same input, three different weight matrices.
q = self.W_q(x)
k = self.W_k(x)
v = self.W_v(x)Score by Similarity
To see how well a query matches a key, you take their dot product. A bigger score means the two tokens are more relevant to each other.
scores = q @ k.transpose(-2, -1)Scores to Weights
Raw scores become attention weights with softmax, so each query's weights are positive and sum to one across all keys.
weights = scores.softmax(dim=-1)Blend the Values
The output for each token is a weighted sum of all values, mixed by the attention weights. Relevant tokens contribute more.
out = weights @ vWhy It Beats RNNs
Self-attention connects any two tokens in one step, so distance does not matter. This parallel view is why transformers handle long context so well.
Learned, Not Fixed
The Q, K, V projections are trained by gradient descent. The network learns what to ask, what to advertise, and what to share, all from data.
Quick Check
Let's test how attention combines its pieces.
Recap
You learned that self-attention turns each token into a query, key, and value, scores query-key matches, and blends values by softmax weights. Nice work!
Frequently asked questions
Is the “Self-Attention: Query, Key & Value” lesson free?
Yes — the full text of “Self-Attention: Query, Key & Value” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Self-Attention: Query, Key & Value”?
Let every token look at every other. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Self-Attention: Query, Key & Value” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Self-Attention: Query, Key & Value
- Scaled Dot-Product & Multi-Head
- Positional Encoding for Order
- Stack a Transformer Encoder Block