順序のための位置エンコーディング

Self-attention treats a sentence like a bag of tokens. Shuffle the words and the math barely changes, so the model has no built-in sense of order.

But order matters: "dog bites man" is not "man bites dog". We must hand the model some signal about each token's position.

The trick is to add a position vector directly to each token's embedding. Same shape, so attention sees content and place fused together.

The original transformer used fixed sine and cosine waves of different frequencies, one pattern per dimension, to mark each position.

Mixing frequencies gives every position a unique fingerprint, and the smooth waves let the model generalize to lengths it never saw in training.

Even dimensions use sine, odd dimensions use cosine, with the wavelength growing across dimensions. That is the whole encoding recipe.

A neat bonus: with sinusoids, the offset between two positions is easy to express, helping attention reason about relative distance.

Modern models often skip sinusoids and use a learned embedding table, one trainable vector per position, just like word embeddings.

Fixed sinusoids extrapolate to new lengths for free; learned tables fit your data but are capped at the maximum length you trained on.

Newer transformers favor rotary position encoding, which rotates query and key vectors by an angle tied to position, baking order into attention itself.

Whatever scheme you pick, positional info is injected once at the input, before the first attention layer ever runs.

Let's check why positional encoding exists.

You learned that attention ignores order, so we add positional encodings, whether sinusoidal, learned, or rotary, to tell tokens where they sit. Well done!