When to Fine-Tune
Signals that prompting hit limits.
When to Fine-Tune is a free AI Prompt Engineering lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Fine-Tuning Is an Evidence Decision
Fine-tuning is justified only when you can point at data showing prompting hit a wall. The trigger is never a hunch - it is a held-out eval where the best honest prompt plateaus below your quality bar despite climbing the optimization ladder.
- Tuning trades flexibility for consistency, lower per-call cost, and learned behavior
- The cost is a data pipeline, eval infra, and re-tuning on base-model churn
- You must be able to name the specific failure prompting could not fix
Signal 1: The Prompt Plateau
The clearest signal is a plateau on a frozen eval set. You add exemplars, decompose, add verifiers - and the score stops improving while errors remain systematic, not random.
Systematic residual errors (the model consistently mishandles the same construct) mean the behavior is hard to elicit through instruction. That is a tuning-shaped problem. Random scattered errors usually mean the prompt or data is still noisy - keep iterating instead.
# Track eval score vs prompt-iteration; flat tail = plateau
scores = [0.62, 0.71, 0.78, 0.79, 0.795, 0.796] # diminishing returns
def plateaued(scores, window=3, eps=0.01):
tail = scores[-window:]
return (max(tail) - min(tail)) < eps
print(plateaued(scores)) # True -> prompting has stalledSignal 2: Prompt Length Becomes the Product
When the prompt has grown to thousands of tokens of exemplars and rules just to hold quality, you are paying a latency and cost tax on every call to simulate learned behavior.
If those tokens encode stable, repetitive behavior (a fixed format, a consistent style, a routing decision), fine-tuning can fold them into weights - shrinking the prompt by an order of magnitude while keeping behavior. This is prompt distillation, the most common legitimate tuning use case.
Signal 3: Hard Latency or Cost Floor
If you need a smaller, faster, cheaper model to match a larger model's behavior on a narrow task, tuning is the tool. You distill the large model's outputs into a small tuned model.
This is justified when: volume is above break-even, latency budget is tight, and the task is narrow enough that a small model can master it. Outside those conditions, the engineering overhead is not worth it.
# Distillation data: teacher (big model) labels -> student (small) trains
def make_distill_pair(prompt, teacher_fn):
completion = teacher_fn(prompt) # high-quality big-model output
return {'messages': [
{'role': 'user', 'content': prompt},
{'role': 'assistant', 'content': completion},
]}Signal 4: Idiosyncratic Format or Style
Some outputs are so specific that describing them costs more than demonstrating them at scale: a proprietary DSL, a house writing voice with a thousand micro-rules, a rigid domain schema with countless conditional fields.
When format compliance must be near-perfect and the rules are too numerous to enumerate in a prompt, hundreds of examples teach the pattern more reliably than prose. Tuning excels at internalizing implicit structure that resists explicit instruction.
Signal 5: Behavior the Model Resists
Occasionally a model fights an instruction: it keeps adding caveats you forbade, refuses a benign task, or reverts to a default style under load. If strong, repeated instruction plus exemplars cannot reliably suppress the behavior, that resistance is a prior baked into the base weights.
Tuning can override such priors. But verify first that the resistance is genuine and not a prompt-clarity problem - misattributed resistance leads to unnecessary training runs.
Anti-Signals: When NOT to Tune
Equally important is recognizing false alarms. Do NOT fine-tune when:
- The gap is knowledge - retrieve it instead; tuning bakes in stale, lossy facts
- The spec is still changing weekly - you will re-train constantly
- You have fewer than a few hundred clean examples - too little signal, high overfit risk
- Errors are random, not systematic - the data or prompt is still noisy
- You lack an eval harness - you cannot tell if tuning even helped
Data Readiness Gate
Fine-tuning quality is bounded by data quality. Before committing, pass a readiness gate: enough examples, balanced across the cases you care about, label-consistent, and free of the leakage that inflates eval scores.
A few hundred meticulously curated examples beat tens of thousands of noisy ones. If your labels disagree with each other, the model will learn the noise.
def data_ready(examples, min_n=300, max_dupe_ratio=0.05):
n = len(examples)
texts = [e['messages'][0]['content'] for e in examples]
dupe_ratio = 1 - (len(set(texts)) / n)
return n >= min_n and dupe_ratio <= max_dupe_ratio
# Returns False until you have enough deduped, curated examplesChoose the Tuning Method
Not all tuning is full-weight training. Match the method to the signal:
- LoRA / adapters - cheap, fast, reversible; ideal for style and format distillation
- Full fine-tune - heavier; for deep behavioral shifts on capable open models
- Preference tuning (DPO-style) - when you have pairwise good/bad judgments rather than gold completions
Start with the lightest method that the signal demands. LoRA-style adapters cover the majority of production cases at a fraction of the cost.
Pre-Commit Protocol
Before launching a training run, lock in the experiment so the result is interpretable:
- Freeze a held-out eval set the model will never see in training
- Record the best prompt-only baseline score on that set
- State the target lift and the cost ceiling in advance
- Define the rollback: if tuned does not beat baseline by the target, you ship the prompt
Without a pre-committed baseline and target, you cannot prove tuning earned its keep.
Putting Signals Together
Combine the signals into a single go/no-go gate. Tuning proceeds only when prompting has plateaued AND the data is ready AND the gap is behavior - not when any single signal fires in isolation.
def should_fine_tune(plateaued, data_ready, gap_is_behavior,
spec_stable, has_eval_harness):
return all([
plateaued, # prompting stalled on frozen eval
data_ready, # enough clean, deduped examples
gap_is_behavior, # not a knowledge gap (else use RAG)
spec_stable, # task definition has settled
has_eval_harness, # can measure the lift
])
print(should_fine_tune(True, True, True, True, True)) # True -> proceedQuick Check
On a frozen eval set, a model's errors are random and scattered across many different input types, and the score still jumps when you tweak exemplars. What does this indicate about fine-tuning readiness?
Recap
Fine-tune on evidence, not intuition. The legitimate signals: a true plateau with systematic errors, prompt length that has become the product, a hard latency/cost floor, idiosyncratic formats, or behavior the base model resists.
- Anti-signals: knowledge gaps, moving specs, too little data, random errors, no eval harness
- Pass a data-readiness gate before training - quality bounds the result
- Choose the lightest method (LoRA-style first) that the signal demands
- Pre-commit a frozen eval, a baseline, a target lift, and a rollback
- Proceed only when plateau AND data-ready AND behavior-gap all hold
Frequently asked questions
Is the “When to Fine-Tune” lesson free?
Yes — the full text of “When to Fine-Tune” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.
What will I learn in “When to Fine-Tune”?
Signals that prompting hit limits. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Prompt Engineering?
No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “When to Fine-Tune” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Prompt Engineering lesson?
Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.