การทดสอบและประเมินแอป RAG ของคุณ

A RAG app can look fine on a few queries and fail badly on others. Without measurement you cannot tell if a change helped or hurt.

Evaluation gives you a repeatable score to guide improvements.

RAG quality has two parts:

• Retrieval: did we fetch the right documents?
• Generation: did the answer use them correctly?

A bad answer can come from either, so measure both.

Create a small set of questions with known correct answers and the documents that contain them. Even 20 to 50 examples are enough to start.

Hit rate (or recall@k) checks whether the correct source appears in the top-k retrieved chunks. High hit rate means retrieval is doing its job.

Faithfulness asks: is the answer supported by the retrieved context, or did the model make things up? An LLM judge can score this automatically.

Answer relevance measures whether the response actually addresses the question, regardless of sources. A faithful answer can still be off-topic.

You can use a strong model to grade outputs against the expected answer, returning a pass or score with a reason.

Loop over the test set, run your pipeline, and aggregate scores into a single report you can compare across versions.

Change one variable — chunk size, k, prompt, model — rerun the same test set, and compare scores. This turns guesswork into evidence-based tuning.

Keep the test suite in CI. When a change drops a metric, you catch the regression before users do. Treat evaluation like unit tests for AI quality.

Use failures to guide fixes:

• Low hit rate? Adjust chunking or retrieval
• Low faithfulness? Strengthen grounding instructions
• Low relevance? Improve the prompt

You learned to evaluate your RAG app:

• Measure both retrieval and generation
• Build a small test set with known answers
• Use hit rate, faithfulness, and answer relevance
• Let an LLM judge grade outputs
• Compare configs and guard against regressions in CI

Evaluation turns RAG improvement into a measurable, repeatable process.