Building a RAG Q&A System End-to-End
RetrievalQA chain, custom prompts, source attribution, evaluating RAG with RAGAS.
Building a RAG Q&A System End-to-End is a free Learn AI with Python lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What is RAG?
Retrieval-Augmented Generation grounds an LLM in your own data. At query time you retrieve relevant chunks from a vector store and inject them into the prompt, so the model answers from facts instead of guessing.
The RAG Flow
Every RAG query follows four steps: (1) embed the question, (2) retrieve similar chunks, (3) stuff them into the prompt as context, (4) the LLM generates an answer grounded in that context.
Turning a Store into a Retriever
Convert your vector store to a retriever with as_retriever. The search_kwargs dict controls retrieval, most importantly k, the number of chunks to fetch per question.
retriever = db.as_retriever(
search_kwargs={"k": 4}
)Tuning search_kwargs
search_kwargs can also set a score threshold or MMR options. Start with k=4; raise it if answers miss context, lower it to cut token cost and noise.
retriever = db.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={"k": 6, "score_threshold": 0.5}
)RetrievalQA Chain
The RetrievalQA chain wires retrieval and generation together. Build it with from_chain_type, passing your llm and retriever. One call now answers questions from your documents.
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o"),
retriever=retriever
)chain_type: How Context is Combined
The chain_type controls how retrieved chunks enter the prompt. "stuff" (default) puts all chunks in one prompt, simple and best when they fit the context window. Other types like "map_reduce" handle many large chunks.
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=retriever
)Asking a Question
Run the chain with invoke, passing your question under the query key. The chain retrieves, builds the prompt, and returns the grounded answer.
response = qa.invoke({"query": "What is the refund window?"})
print(response["result"])Returning Source Documents
Set return_source_documents=True to get the chunks used for the answer. This enables citations and lets users verify where each fact came from, crucial for trust.
qa = RetrievalQA.from_chain_type(
llm=llm,
retriever=retriever,
return_source_documents=True
)
resp = qa.invoke({"query": "Refund window?"})
for d in resp["source_documents"]:
print(d.metadata)Why Evaluate RAG?
RAG can fail in two ways: retrieval brings the wrong chunks, or generation hallucinates despite good chunks. You need metrics to know which part to fix. Eyeballing answers does not scale.
RAGAS Metrics
RAGAS is a library that scores RAG quality. Key metrics: faithfulness (answer supported by context), answer_relevancy (answer addresses the question), and context_precision (retrieved chunks are relevant).
pip install ragas
from ragas.metrics import faithfulness, answer_relevancy, context_precisionRunning an Evaluation
Build a dataset of questions, generated answers, retrieved contexts, and ground-truth answers, then call evaluate. The scores tell you whether to improve chunking, retrieval k, or the prompt.
from ragas import evaluate
result = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy, context_precision]
)
print(result)Quick Check
Test your RAG knowledge.
Recap: End-to-End RAG
You built a full RAG pipeline: db.as_retriever with search_kwargs, a RetrievalQA.from_chain_type chain over your llm and retriever, and return_source_documents=True for citations. You also learned to measure quality with RAGAS metrics like faithfulness and context precision.
Frequently asked questions
Is the “Building a RAG Q&A System End-to-End” lesson free?
Yes — the full text of “Building a RAG Q&A System End-to-End” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Building a RAG Q&A System End-to-End”?
RetrievalQA chain, custom prompts, source attribution, evaluating RAG with RAGAS. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Building a RAG Q&A System End-to-End” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- LangChain Architecture and LCEL
- Document Loading, Splitting, and Embedding
- Vector Stores: Chroma and FAISS
- Building a RAG Q&A System End-to-End