Model Routing (Cheap -> Expensive)
Try a small model first, escalate to a frontier model only when the small one fails or low-confidence.
Model Routing (Cheap -> Expensive) is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Don't Use Big Models For Easy Tasks
Calling GPT-4o for "is this email spam?" wastes money. Route easy tasks to cheap models, hard tasks to expensive ones.
The Routing Pattern
- Try small/cheap model first
- If output is low-confidence or fails validation, escalate to big model
- Log which path was taken
Confidence-Based Routing
cheap_response = call('gpt-4o-mini', messages)
confidence = parse_confidence(cheap_response)
if confidence > 0.85:
return cheap_response
# Otherwise escalate
return call('gpt-4o', messages)Validation-Based Routing
If the cheap model's output fails schema validation, escalate:
try:
result = Schema.model_validate_json(cheap_response.content)
return result
except ValidationError:
return Schema.model_validate_json(call('gpt-4o', messages).content)Per-Step Model Choice
Different parts of an agent need different models:
MODEL_BY_STEP = {
'classify_intent': 'gpt-4o-mini', # easy
'plan': 'gpt-4o', # critical
'extract': 'gpt-4o-mini', # routine
'write_response': 'claude-sonnet-4-5' # quality matters
}
# --- demo ---
for step in ['classify_intent', 'plan', 'extract', 'write_response']:
print(f'{step:16s} -> {MODEL_BY_STEP[step]}')
Routing Across Providers
Use Groq for latency, OpenAI for quality, Anthropic for long context:
if need_low_latency:
return call_groq('llama-3.1-70b')
elif need_long_context:
return call_anthropic('claude-sonnet-4-5')
else:
return call_openai('gpt-4o-mini')LLM-as-Router
A small model classifies the request and picks the next model:
router = call('gpt-4o-mini', 'Classify this request as simple/complex/code. Return one word.')
if router == 'simple':
return call('gpt-4o-mini', user_msg)
else:
return call('gpt-4o', user_msg)Fallback Chain
If primary fails (rate limit, error), try secondary:
for model in ['gpt-4o', 'claude-sonnet-4-5', 'gpt-4o-mini']:
try:
return call(model, messages)
except RateLimitError:
continue
raise AllModelsFailed()Embedding-Based Routing
Embed the query; if similar to a known easy case, use cheap; else use big model:
embedding = embed(query)
matches = vector_db.query(embedding, k=3, filter={'category': 'easy'})
if max(m.score for m in matches) > 0.9:
use_cheap_model()Routing Hierarchy Example
| Tier | Cost | Use |
|---|---|---|
| Tier 1 | $ — Llama 8B | Classification, extraction |
| Tier 2 | $$ — gpt-4o-mini | Standard agent steps |
| Tier 3 | $$$ — gpt-4o / claude | Hard reasoning, final synthesis |
Measure Net Savings
Track:
- % of requests served by each tier
- Quality (eval pass rate) per tier
- Total cost per request
Compare to baseline (always-big-model) to verify routing helps.
Calibrate Confidence
Self-reported confidence is unreliable. Calibrate against an eval set; if model says 90% but only correct 60% of the time, adjust your threshold.
Open-Source Routers
RouteLLM (LMSYS) is an OSS framework for trained model routers. Worth exploring if routing is critical to your cost structure.
Routing Strategy
What's the simplest and most effective routing strategy?
Recap
Tiered models, cheap-first with escalation, per-step routing, fallback chains. Measure tier mix and quality. Routing is the single biggest production cost lever.
Frequently asked questions
Is the “Model Routing (Cheap -> Expensive)” lesson free?
Yes — the full text of “Model Routing (Cheap -> Expensive)” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Model Routing (Cheap -> Expensive)”?
Try a small model first, escalate to a frontier model only when the small one fails or low-confidence. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Model Routing (Cheap -> Expensive)” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Token Budgets Per Step
- Model Routing (Cheap -> Expensive)
- Caching Prompts and Results (Anthropic, Vertex)
- Quantisation and Speculative Decoding