0Pricing
AI Agents · Lesson

RAG Evaluation (RAGAS, Recall@K)

Measure faithfulness, answer relevance, context precision, and recall@K to know if changes help.

RAG Evaluation (RAGAS, Recall@K) is a free AI Agents lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

You Cannot Improve What You Cannot Measure

RAG has many knobs: chunk size, top-K, re-ranker, prompt, model. Without metrics, every change is guesswork.

Key Metrics for RAG

  1. Retrieval: did we fetch the right chunks?
  2. Faithfulness: does the answer stay grounded in the chunks?
  3. Answer relevance: does the answer address the question?
  4. Context precision/recall: ratio of useful chunks fetched

Recall@K

Build a gold set of (question, list-of-relevant-chunk-ids) pairs. Measure how often the top-K retrieved chunks contain a relevant one:

def recall_at_k(eval_set, k=5):
    hits = 0
    for item in eval_set:
        retrieved = retriever(item['question'], k=k)
        if any(c.id in item['relevant_ids'] for c in retrieved):
            hits += 1
    return hits / len(eval_set)

Precision@K

What fraction of retrieved chunks are relevant?

def precision_at_k(eval_set, k=5):
    total_relevant = 0
    total_retrieved = 0
    for item in eval_set:
        retrieved = retriever(item['question'], k=k)
        total_relevant += sum(1 for c in retrieved if c.id in item['relevant_ids'])
        total_retrieved += k
    return total_relevant / total_retrieved

MRR (Mean Reciprocal Rank)

How highly is the FIRST relevant doc ranked?

def mrr(eval_set, k=10):
    total = 0
    for item in eval_set:
        retrieved = retriever(item['question'], k=k)
        for rank, c in enumerate(retrieved, start=1):
            if c.id in item['relevant_ids']:
                total += 1 / rank
                break
    return total / len(eval_set)

Faithfulness (LLM-as-Judge)

Use an LLM to check whether each claim in the answer is supported by the context:

judge_prompt = '''
Given the context and answer, identify each factual claim in the answer.
For each claim, judge whether the context supports it.
Return {claims: [{claim, supported: true/false}]}.
'''
result = judge_llm.invoke(judge_prompt.format(context=ctx, answer=ans))

Answer Relevance

Reverse-judge: ask an LLM "Could this answer be the answer to this question?" Catches off-topic outputs.

RAGAS Framework

RAGAS automates the above metrics:

# pip install ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

results = evaluate(
    dataset,            # HF dataset with question, contexts, answer, ground_truth
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(results)

Building a Gold Set

20-200 (question, expected_answer, relevant_chunks) tuples curated by humans. Cover:

  • Common queries
  • Edge cases
  • Adversarial inputs (out-of-corpus questions)

Synthetic Eval Sets

Bootstrapping: ask an LLM to generate Q&A pairs from your documents. Lower quality but fast:

synth_prompt = 'Read this document and write 3 questions a user might ask, with answers from the doc:\n{doc}'
# Use as a starting point; humans curate later.
print(synth_prompt)

LangSmith and Langfuse Datasets

Both tools let you turn production traces into eval datasets. Pick 50 real questions that went poorly, label correct answers, and use that as your benchmark.

Don't Over-Optimise One Metric

Maximising Recall@K can hurt Precision@K (too many false positives). Track multiple metrics and a composite.

A/B Tests in Production

Once you have an offline eval that correlates with user satisfaction, you can A/B test changes in prod and compare both metrics and user thumbs-up rates.

Recall@K Definition

What does Recall@5 = 0.8 mean?

Recap

Build a gold set. Measure Recall@K, Precision@K, MRR, Faithfulness, Answer Relevance. Use RAGAS to automate. Iterate against your metrics.

Frequently asked questions

Is the “RAG Evaluation (RAGAS, Recall@K)” lesson free?

Yes — the full text of “RAG Evaluation (RAGAS, Recall@K)” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “RAG Evaluation (RAGAS, Recall@K)”?

Measure faithfulness, answer relevance, context precision, and recall@K to know if changes help. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “RAG Evaluation (RAGAS, Recall@K)” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Re-ranking with Cross-Encoders
  2. HyDE: Hypothetical Document Embeddings
  3. Multi-Vector Retrieval (ColBERT)
  4. RAG Evaluation (RAGAS, Recall@K)
← Back to AI Agents