Trade-offs: Latency, Cost, Capability
Open weights save money but cost engineer hours; benchmark before committing.
Trade-offs: Latency, Cost, Capability is a free AI Agents lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Three Axes, Pick Two
Every model choice trades off:
- Latency — time per response
- Cost — per million tokens
- Capability — quality on hard tasks
No model is best on all three.
Capability Landscape
| Top tier | Mid tier | Small tier | |
|---|---|---|---|
| Closed | GPT-4o, Claude Sonnet 4.5 | GPT-4o-mini, Haiku | — |
| Open | Llama 3.1 405B | Llama 3.1 70B, Qwen 2.5 72B | Llama 3.1 8B, Phi-3 |
Latency Landscape
Approximate p50 latency for a typical agent turn:
- Groq Llama 3.1 70B: ~200ms (lightning fast)
- gpt-4o-mini: ~600ms
- gpt-4o: ~1500ms
- Self-hosted Llama 70B on 1xH100: ~800ms
- Self-hosted Llama 8B on RTX 4090: ~200ms
Cost Per Million Tokens (Approx)
Mid-2025 ballpark, in/out:
- GPT-4o: $2.50 / $10
- GPT-4o-mini: $0.15 / $0.60
- Claude Sonnet 4.5: $3 / $15
- Claude Haiku: $0.80 / $4
- Together Llama 70B: $0.88 / $0.88
- Groq Llama 70B: $0.59 / $0.79
- Self-hosted (your GPU): essentially free per token
Calculate Your Crossover Point
Self-hosting saves money only above a volume threshold. Rough estimate:
GPU_COST_PER_HOUR = 1.5
GPU_TOKENS_PER_SECOND = 50
TOKENS_PER_HOUR = GPU_TOKENS_PER_SECOND * 3600
# 180,000 tokens/hour
print(f"Tokens per hour: {TOKENS_PER_HOUR:,}")
breakeven_tokens = GPU_COST_PER_HOUR / 0.6e-6
print(f"Breakeven vs $0.60/1M closed-model price: {breakeven_tokens:,.0f} tokens/hour")
print("Far above typical small-app traffic")
Model Routing
Use cheap models by default, escalate to expensive ones only when needed:
def answer(query):
cheap = call_model('gpt-4o-mini', query)
if confidence(cheap) > 0.8:
return cheap
return call_model('gpt-4o', query)Per-Step Routing
Inside an agent, route per-step:
- Plan with GPT-4o (hard reasoning)
- Search with Llama 8B (cheap loops)
- Synthesise with Claude (writing quality)
Latency-Sensitive Paths
For voice / real-time chat:
- Use Groq, SambaNova, or Cerebras for sub-200ms
- Stream tokens aggressively
- Avoid multi-step agents on the critical path
Quality-Sensitive Paths
For final answers and high-stakes work:
- Use the best model you can afford
- Add cross-model critique (GPT-4 writes, Claude reviews)
- Add verification (run code, check facts)
Total Cost of Ownership
Self-hosting cost includes:
- GPU rental or capex
- Engineer time to manage infra
- Monitoring, on-call
- Lower throughput than hosted services
For most teams, hosted APIs are cheaper TCO until very high volume.
When to Pay for Big Closed Models
- Final user-facing answers
- Frontier reasoning
- Multimodal
- 200k+ context
Benchmark on Your Task
Public benchmarks tell only part of the story. Build a 50-question eval set on your actual data and measure each candidate model on it. Pick the cheapest that meets quality.
Routing Strategy
What is the cheapest model-routing strategy that still meets quality?
Recap
Latency, cost, capability — pick two. Use cheap models per default, escalate when needed. Hosted open-model APIs (Groq, Together) often beat both extremes.
Frequently asked questions
Is the “Trade-offs: Latency, Cost, Capability” lesson free?
Yes — the full text of “Trade-offs: Latency, Cost, Capability” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Trade-offs: Latency, Cost, Capability”?
Open weights save money but cost engineer hours; benchmark before committing. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Trade-offs: Latency, Cost, Capability” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Llama, Mistral and Qwen Overview
- Running Local Models with Ollama and llama.cpp
- Function-Calling Open Models (Hermes, Functionary)
- Trade-offs: Latency, Cost, Capability