LLM Apps in Production (RAG + Vector DB + Caching) · Lesson

Choosing the Right Model for the Task

Learn how to cut RAG costs and latency by routing each request to the cheapest model that can do the job well, using model tiers, cascades, and quality gates.

Choosing the Right Model for the Task is a free LLM Apps in Production (RAG + Vector DB + Caching) lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the LLM Apps in Production (RAG + Vector DB + Caching) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Model Choice Drives Cost

In a RAG pipeline the LLM call is usually the single biggest cost and latency driver. The same prompt sent to a flagship model can cost 20-50x more than a small model.

Optimizing model selection is often the highest-leverage change you can make.

Token price differs per model
Latency scales with model size
Not every query needs the biggest brain

Model Tiers

Group your available models into tiers by capability and price:

Small / cheap — classification, extraction, simple Q&A
Mid — most RAG answers grounded in retrieved context
Large / flagship — multi-step reasoning, ambiguous queries

Default to the smallest tier that meets your quality bar.

A Simple Router

A router inspects the request and picks a model. Start with rule-based routing before adding ML.

def pick_model(query, context_len):
    if len(query) < 80 and context_len < 2000:
        return 'small-model'
    if 'explain' in query or 'compare' in query:
        return 'large-model'
    return 'mid-model'

print(pick_model('What is the price?', 500))

Model Cascades

A cascade tries a cheap model first, then escalates only if the answer is low confidence. Most queries resolve cheaply; only the hard ones reach the expensive model.

Run small model
Score confidence / check guardrails
Escalate only on failure

Cascade in Code

A minimal cascade with a confidence check.

def answer(query):
    cheap = call('small-model', query)
    if cheap['confidence'] >= 0.8:
        return cheap['text']
    return call('large-model', query)['text']

def call(model, query):
    return {'text': 'stub', 'confidence': 0.9}

print(answer('hello'))

Confidence Signals

How do you know the cheap answer is good enough? Useful signals:

Self-reported confidence from the model
Whether the answer cites retrieved context
Output length / refusal patterns
A small judge model scoring the answer

Matching Context Size to Model

Large context windows are expensive. A model that accepts 200k tokens charges you for every token you send. Trim retrieved chunks aggressively and reserve big windows for queries that truly need them.

Right-sizing context is part of right-sizing the model.

Measuring Quality per Tier

Before downgrading a model, measure quality on a fixed eval set. Track accuracy per tier so you know the real trade-off.

scores = {'small': 0.81, 'mid': 0.90, 'large': 0.93}
bar = 0.88
cheapest_ok = next(m for m, s in scores.items() if s >= bar)
print('Use:', cheapest_ok)

Cost vs Quality Curve

Plotting cost against quality usually shows diminishing returns: jumping to the flagship model buys a few points of accuracy at multiples of the cost.

Pick the point where quality crosses your acceptance bar at the lowest cost.

Fallbacks for Reliability

Routing also helps reliability. If your primary model is rate-limited or down, route to an alternative provider of similar tier so users still get answers.

Primary -> secondary provider
Same tier, comparable quality
Log which path served the request

Putting It Together

A production router combines: tier rules, a cascade for hard queries, context trimming, and provider fallbacks. Continuously evaluate so routing stays calibrated as models change.

Quick Check

Test your understanding of model cascades.

Recap

You learned to cut RAG cost and latency by choosing the right model: tier your models, default to the smallest that meets your bar, use cascades to escalate only hard queries, right-size context, and keep provider fallbacks for reliability. Always validate routing against an eval set.

Frequently asked questions

Is the “Choosing the Right Model for the Task” lesson free?

Yes — the full text of “Choosing the Right Model for the Task” is free to read here on the web, and the LLM Apps in Production (RAG + Vector DB + Caching) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the LLM Apps in Production (RAG + Vector DB + Caching) course, upgrade to CoddyKit PRO.

What will I learn in “Choosing the Right Model for the Task”?

Learn how to cut RAG costs and latency by routing each request to the cheapest model that can do the job well, using model tiers, cascades, and quality gates. You practise LLM Apps in Production (RAG + Vector DB + Caching) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start LLM Apps in Production (RAG + Vector DB + Caching)?

No prior experience is required. LLM Apps in Production (RAG + Vector DB + Caching) on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Choosing the Right Model for the Task” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this LLM Apps in Production (RAG + Vector DB + Caching) lesson?

Yes. Every LLM Apps in Production (RAG + Vector DB + Caching) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to LLM Apps in Production (RAG + Vector DB + Caching)