AutocompleteからChatGPTまで

Before ChatGPT, computers guessed the next word by counting which words usually follow others — an n-gram model. Fast, but no real understanding.

Next came RNNs: they read text word by word and carried a memory of what came before. Smarter than counting, but slow and forgetful over long text.

In 2017, Google's Transformer let a model look at every word at once instead of one at a time. This unlocked huge, fast training — and powers nearly every LLM today. 🚀

OpenAI's GPT models kept growing — GPT-3 hit 175 billion parameters. At that size, surprising emergent capabilities appeared, like doing math it was never taught.

Raw GPT-3 often ignored you or rambled. RLHF fixed that by letting humans rate answers, teaching the model to follow instructions — and ChatGPT was born.

ChatGPT's real magic was the chat interface. It remembered the conversation, so you could just talk to it — and millions of people instantly could. 💬

Old NLP needed a new model for every task. An LLM is a single general-purpose model: it can translate, summarize, code, and chat — all from plain instructions.

Parameters are the numbers a model learns during training — its knowledge. More parameters mean smarter answers, but also more cost to run each request.

An LLM learns during pre-training: it reads huge amounts of text and just predicts the next word. From that one task, it picks up grammar, facts, and reasoning.

After ChatGPT, the race was on: Claude, Gemini, Llama, and more. Now the real question isn't "can AI do this?" but which model, at what cost and trade-offs.

Knowing this history pays off. A refusal is RLHF guardrails; a confident wrong answer is a training-data gap. Understanding the why helps you build better.

Test your understanding of AI Engineering concepts from this lesson.

Recap: n-grams and RNNs came first but were limited, Transformers made models scale, and RLHF turned raw GPT-3 into ChatGPT. Next: how attention actually works. ✨