Evaluating Tuned Models vs Base
A/B against the base model on your eval set — sometimes fine-tuning hurts more than helps.
Evaluating Tuned Models vs Base is a free AI Agents lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Don't Trust Training Loss
Low training loss does not mean better in production. The model may overfit, lose general capability, or hurt unseen tasks.
Always evaluate the tuned model on a held-out eval set.
Three Must-Have Eval Splits
- Train — used in fine-tuning
- Validation — used to pick best checkpoint
- Test — never seen during training; ONLY for final eval
A/B Against Base
results_base = run_eval(base_model, eval_set)
results_tuned = run_eval(tuned_model, eval_set)
print(f'Base: {results_base.score}')
print(f'Tuned: {results_tuned.score}')
print(f'Delta: {results_tuned.score - results_base.score:+.2f}')Watch for Catastrophic Forgetting
Fine-tuning on narrow data can make the model worse at OTHER tasks. Test on a broad eval set too — not just your new specialization.
Per-Category Reporting
Tag your eval set; report scores per category:
Category 'extraction': base 0.78 -> tuned 0.91 (+0.13)
Category 'reasoning': base 0.85 -> tuned 0.78 (-0.07) <-- regression
Category 'chat': base 0.82 -> tuned 0.83 (+0.01)Calibration
Tuned models often become overconfident. Compare predicted vs actual probability of correctness — calibrate if needed.
Length and Style Drift
Tuned models drift toward the training distribution. If training data is shorter, outputs get shorter. Sometimes desirable; sometimes not.
Real-World A/B Test
Beyond offline evals, A/B test in production:
if user.bucket == 'tuned':
response = tuned_model.run(...)
else:
response = base_model.run(...)
log_metric('thumbs_up', response.feedback)Compare Quality and Cost
The tuned model may be smaller/cheaper. Total cost-quality:
- Base GPT-4o: $X per call, quality Y
- Tuned Llama 8B (self-hosted): $X/10 per call, quality 0.9Y
- Likely a winner if Y stays high enough
Hidden Failure Modes
Try adversarial inputs:
- Prompts that try to bypass guardrails
- Out-of-distribution inputs
- Edge-case prompts
Sometimes fine-tunes weaken safety; verify before shipping.
LLM-as-Judge Caveat
If you use an LLM judge, use a DIFFERENT model family than your tuned model. Otherwise the judge gives biased high scores.
When to Iterate the Dataset
If eval shows specific categories regressed:
- Find similar examples in production traces
- Add them to the training set
- Re-train
- Re-evaluate
Roll Back is OK
If the tune underperforms, throw it away and try again. Sunk cost is not a reason to ship a worse model.
Catastrophic Forgetting
What is catastrophic forgetting in fine-tuning?
Recap
Eval tuned model against base on YOUR gold set. Watch per-category regressions. A/B in production. Roll back if it loses; iterate dataset if it half-wins.
Frequently asked questions
Is the “Evaluating Tuned Models vs Base” lesson free?
Yes — the full text of “Evaluating Tuned Models vs Base” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Evaluating Tuned Models vs Base”?
A/B against the base model on your eval set — sometimes fine-tuning hurts more than helps. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Evaluating Tuned Models vs Base” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- When Fine-Tuning Beats Prompting
- Data Collection: Trajectories and Trace Replay
- LoRA and QLoRA for Cost-Efficient Tuning
- Evaluating Tuned Models vs Base