Self-Attention, Step by Step

In self-attention, every word looks at every other word in the same sentence to build a richer, context-aware version of itself. 🔍

Each word is projected into three vectors: a query, a key, and a value. These three roles drive the whole self-attention computation.

The query represents what a word is looking for. Think of it as the question this word asks about the rest of the sentence.

Each word also offers a key that advertises what it contains, and a value holding the actual information to pass along if chosen.

Compare a query to every key using a dot product. A larger product means the query and key align, so that word deserves more focus.

Divide scores by the square root of the key dimension. This scaling keeps numbers stable so softmax does not become too sharp.

Run the scaled scores through softmax to get attention weights that are positive and sum to 1 across all words in the sentence.

Multiply each value by its weight and add them up. The result is a new vector that blends information from the most relevant words.

All steps combine into one tidy expression: scaled dot-product attention over queries, keys, and values, as introduced in the Transformer paper.

It is called self-attention because the queries, keys, and values all come from the same sequence, letting words attend to their own neighbors.

In "it was tired," self-attention links it to the right noun by weighting nearby words, giving every token clearer context.

Let us confirm the self-attention steps.

You walked through self-attention: turn words into queries, keys, and values, score, scale, softmax, then blend the values. That is the engine. ⚙️