How LLMs Are Trained
Learn the stages of LLM training: pre-training on massive corpora, supervised fine-tuning, and RLHF, and why each stage matters for model behavior.
How LLMs Are Trained is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Three Stages of LLM Training
LLMs are built in three stages: pre-training teaches language and knowledge, fine-tuning teaches instruction-following, and RLHF aligns the model with what people want.
Pre-Training: Next Token Prediction at Scale
Pre-training feeds the model tons of text with one job: predict the next token. That simple goal is enough to teach grammar, facts, and reasoning along the way.
Training Data Quality and Curation
Raw internet text is messy. Teams clean it with data pipelines that filter, deduplicate, and score quality. The rule of thumb: better data beats a bigger model.
The Loss Function and Optimization
Training minimizes cross-entropy loss — how far the model's guess is from the right next token. Tiny weight nudges, repeated billions of times. The code shows the math.
# Illustrative cross-entropy loss for a single token prediction
import math
vocab_size = 50000
correct_token_index = 4217 # index for the word 'Paris'
# Model output probabilities (softmax of logits)
model_probs = [0.00002] * vocab_size # simplified uniform base
model_probs[correct_token_index] = 0.70 # model is 70% confident in 'Paris'
loss = -math.log(model_probs[correct_token_index])
print(f'Cross-entropy loss: {loss:.4f}') # ~0.3567Emergent Capabilities from Scale
At enough scale, new emergent capabilities appear — like step-by-step reasoning — that small models simply don't have. Nobody trained them in directly; scale brought them.
Supervised Fine-Tuning on Instruction Pairs
A raw model just continues text. Supervised fine-tuning trains it on (instruction, ideal answer) pairs, so it learns to actually answer instead of rambling on.
# Illustrative SFT data format
training_example = {
'messages': [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': 'What is the capital of France?'},
{'role': 'assistant', 'content': 'The capital of France is Paris.'}
]
}
# During SFT, loss is computed only on the assistant turn tokens
# The system and user tokens are provided as context but not trained onRLHF Phase 1: Training the Reward Model
RLHF starts with a reward model. People pick the better of two answers, and the reward model learns to predict those preferences — a stand-in for human taste.
RLHF Phase 2: PPO Fine-Tuning
Next, PPO tunes the LLM to score higher on the reward model. A KL penalty keeps it from drifting weird and gaming the reward instead of truly helping.
Direct Preference Optimization: Simpler RLHF
PPO is complex. DPO is a simpler alternative: it trains the model straight from preferred-vs-rejected answer pairs — no separate reward model needed.
Chinchilla Scaling Laws: Compute-Optimal Training
The Chinchilla finding: older models were undertrained. For a given budget, scale data and size together — roughly 20 tokens per parameter for the best results.
What Each Training Stage Affects
Each stage controls something: pre-training sets what it knows, fine-tuning sets how it behaves, and RLHF sets its values. That tells you which part to fix.
Quick Check
Test your understanding of AI Engineering concepts from this lesson.
Lesson Recap
Recap: pre-training builds knowledge, fine-tuning builds instruction-following, and RLHF or DPO aligns the model with human values. Next: what LLMs can and can't do. 💡
Frequently asked questions
Is the “How LLMs Are Trained” lesson free?
Yes — the full text of “How LLMs Are Trained” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “How LLMs Are Trained”?
Learn the stages of LLM training: pre-training on massive corpora, supervised fine-tuning, and RLHF, and why each stage matters for model behavior. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “How LLMs Are Trained” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.