ทรานส์ฟอร์เมอร์และกลไกความสนใจในภาษาที่เข้าใจง่าย

In "the trophy didn't fit in the bag because it was too big," what is "it"? Old models lost track over long sentences. This is the long-range dependency problem.

Attention is how a model decides which words matter most for each word — like highlighting the key parts of a passage instead of treating every word the same.

Attention works like a search: each word sends a Query, matches it against every Key, and pulls in the most relevant Values. The code below shows the core idea.

One attention head catches one kind of link. Multi-head attention runs many in parallel, so the model sees words through several lenses at once.

Attention alone ignores word order — "dog bites man" looks like "man bites dog." Positional encodings add a sense of position so order isn't lost.

After attention shares context, a feed-forward network processes each word on its own. Interestingly, much of the model's factual knowledge seems to live here.

BERT-style encoder-only models read all the text at once to understand it. GPT-style decoder-only models read left-to-right to generate it — which is what ChatGPT does.

Modern LLMs stack dozens of Transformer blocks. Each layer refines the last — early ones catch grammar, deeper ones handle reasoning. More depth, more thinking.

Stacking many layers is tricky. Residual connections and layer normalization keep the signal stable, so deep models can train reliably at 100+ layers.

Attention is easy to run in parallel, so more GPUs mean faster training. That's how researchers trained on huge data and uncovered the famous scaling laws.

Long inputs make standard attention very memory-hungry. FlashAttention computes the same result far more efficiently, making big context windows practical.

Test your understanding of AI Engineering concepts from this lesson.

Recap: self-attention links every word to every other, multi-head attention captures many relationships at once, and residuals plus normalization let models go deep. Next: how LLMs are trained.