Valutazione RAG (RAGAS, Recall@K)

Non si può migliorare ciò che non si può misurare

Metriche chiave per il RAG

Recall@K

def recall_at_k(eval_set, k=5):
    hits = 0
    for item in eval_set:
        retrieved = retriever(item['question'], k=k)
        if any(c.id in item['relevant_ids'] for c in retrieved):
            hits += 1
    return hits / len(eval_set)

Precision@K

def precision_at_k(eval_set, k=5):
    total_relevant = 0
    total_retrieved = 0
    for item in eval_set:
        retrieved = retriever(item['question'], k=k)
        total_relevant += sum(1 for c in retrieved if c.id in item['relevant_ids'])
        total_retrieved += k
    return total_relevant / total_retrieved

MRR (Mean Reciprocal Rank)

def mrr(eval_set, k=10):
    total = 0
    for item in eval_set:
        retrieved = retriever(item['question'], k=k)
        for rank, c in enumerate(retrieved, start=1):
            if c.id in item['relevant_ids']:
                total += 1 / rank
                break
    return total / len(eval_set)

Fedeltà (LLM-as-Judge)

judge_prompt = '''
Given the context and answer, identify each factual claim in the answer.
For each claim, judge whether the context supports it.
Return {claims: [{claim, supported: true/false}]}.
'''
result = judge_llm.invoke(judge_prompt.format(context=ctx, answer=ans))

Rilevanza della risposta

Framework RAGAS

# pip install ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

results = evaluate(
    dataset,            # HF dataset with question, contexts, answer, ground_truth
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(results)

Creazione di un gold set

Dataset di valutazione sintetici

synth_prompt = 'Read this document and write 3 questions a user might ask, with answers from the doc:\n{doc}'
# Use as a starting point; humans curate later.
print(synth_prompt)

Dataset di LangSmith e Langfuse

Non ottimizzi eccessivamente una sola metrica

Test A/B in produzione

Definizione di Recall@K

Riepilogo