AI Agents · Lesson

Multi-Document Q&A Agents

Indexing a document corpus and answering questions across all documents.

Multi-Document Q&A Agents is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Multi-Document Q&A Overview

A multi-document Q&A agent answers questions by retrieving relevant content from a collection of N documents, synthesizing an answer, and attributing each claim to its source.

Unlike single-document Q&A, multi-doc agents must handle conflicting information across sources and reason about which documents are most relevant to the question.

Indexing Multiple Documents

Before answering any questions, all documents must be indexed: parsed, chunked, embedded, and stored in a vector database. Each chunk is stored with metadata linking it back to its source document.

import chromadb
from chromadb.utils import embedding_functions
import os

client = chromadb.PersistentClient(path='./doc_index')
ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv('OPENAI_API_KEY'),
    model_name='text-embedding-3-small'
)
collection = client.get_or_create_collection('documents', embedding_function=ef)

def index_document(doc_id, doc_path, doc_title):
    # Parse and chunk
    chunks = pdf_to_chunks(doc_path, chunk_size=800, overlap=150)
    for i, chunk in enumerate(chunks):
        chunk_id = f'{doc_id}_chunk_{i}'
        collection.add(
            ids=[chunk_id],
            documents=[chunk['text']],
            metadatas=[{
                'doc_id': doc_id,
                'title': doc_title,
                'page': chunk['page'],
                'source_file': doc_path
            }]
        )
    print(f'Indexed {len(chunks)} chunks from: {doc_title}')

Retrieving Relevant Chunks

On a question, query the vector store for the most semantically similar chunks across all indexed documents. The n_results parameter controls how many chunks to retrieve.

def retrieve_relevant_chunks(question, n_results=8):
    results = collection.query(
        query_texts=[question],
        n_results=n_results,
        include=['documents', 'metadatas', 'distances']
    )

    chunks = []
    for i in range(len(results['documents'][0])):
        chunks.append({
            'text':      results['documents'][0][i],
            'metadata':  results['metadatas'][0][i],
            'distance':  results['distances'][0][i],
            'relevance': 1 - results['distances'][0][i]  # cosine similarity proxy
        })

    # Sort by relevance
    chunks.sort(key=lambda x: x['relevance'], reverse=True)
    return chunks

Source Attribution in the Prompt

When passing retrieved chunks to the LLM, label each chunk with its document source. The LLM can then cite sources using numbered references in the answer.

def format_chunks_for_prompt(chunks, max_chars=4000):
    sections = []
    used_chars = 0

    for i, chunk in enumerate(chunks, 1):
        meta = chunk['metadata']
        header = f"[Source {i}: {meta['title']}, page {meta.get('page', '?')}]"
        content = chunk['text'][:600]
        entry = f'{header}\n{content}'

        if used_chars + len(entry) > max_chars:
            break

        sections.append(entry)
        used_chars += len(entry)

    return '\n\n'.join(sections)

QA_PROMPT = '''Answer the question based on the provided document excerpts.
Cite sources as [Source N]. If sources conflict, mention both views.

{context}

Question: {question}
Answer:'''

def answer_question(question):
    chunks = retrieve_relevant_chunks(question, n_results=6)
    context = format_chunks_for_prompt(chunks)
    return llm_call(QA_PROMPT.format(context=context, question=question))

Cross-Document Reasoning

Some questions require synthesizing information from multiple documents — not just finding a single matching chunk. For example: "Which of the three contracts has the lowest penalty clause?"

Use a two-step approach: retrieve relevant chunks from each document, then ask the LLM to compare and synthesize across them.

def cross_document_compare(question, doc_ids):
    # Retrieve best chunks per document
    per_doc_chunks = {}
    for doc_id in doc_ids:
        results = collection.query(
            query_texts=[question],
            n_results=3,
            where={'doc_id': {'$eq': doc_id}}  # filter by document
        )
        if results['documents'][0]:
            per_doc_chunks[doc_id] = results['documents'][0]

    # Format with document labels
    context_parts = []
    for doc_id, texts in per_doc_chunks.items():
        doc_label = f'Document {doc_id}'
        combined = ' '.join(texts[:2])[:600]
        context_parts.append(f'== {doc_label} ==\n{combined}')

    comparison_context = '\n\n'.join(context_parts)
    return llm_call(f'Compare these documents to answer: {question}\n\n{comparison_context}')

Handling Conflicting Information

Different documents may state conflicting facts — one contract says payment is due in 30 days, another says 60 days. The agent must detect and surface these conflicts rather than silently picking one.

CONFLICT_PROMPT = '''You are analyzing multiple document sources.
Some may contain conflicting information.

For each factual claim you make:
1. Cite the source document
2. If another source contradicts it, explicitly note the conflict
3. Indicate which source you believe is more authoritative, if possible

Document excerpts:
{context}

Question: {question}

Answer (with conflict notes where applicable):'''

def answer_with_conflict_detection(question):
    chunks = retrieve_relevant_chunks(question, n_results=8)
    context = format_chunks_for_prompt(chunks)
    return llm_call(CONFLICT_PROMPT.format(
        context=context, question=question
    ))

Relevance Threshold Filtering

Not all retrieved chunks are truly relevant — vector similarity has a recall/precision tradeoff. Set a minimum relevance threshold to exclude weakly matching chunks that could mislead the LLM.

MIN_RELEVANCE = 0.72  # cosine similarity threshold

def retrieve_above_threshold(question, n_results=10, threshold=MIN_RELEVANCE):
    chunks = retrieve_relevant_chunks(question, n_results=n_results)
    relevant = [c for c in chunks if c['relevance'] >= threshold]

    print(f'Retrieved: {len(chunks)}, Above threshold: {len(relevant)}')

    if not relevant:
        # Fallback: use top 3 even if below threshold
        return chunks[:3]

    return relevant

def answer_with_threshold(question):
    chunks = retrieve_above_threshold(question)
    if not chunks:
        return 'I could not find relevant information in the indexed documents.'
    context = format_chunks_for_prompt(chunks)
    return llm_call(QA_PROMPT.format(context=context, question=question))

Generating Source Citations

After generating an answer, extract which source documents were cited and return them as a structured list. This helps users find the original documents for verification.

import re

def extract_citations(answer_text, chunks):
    # Find all [Source N] references in the answer
    cited_nums = set(int(m) for m in re.findall(r'\[Source (\d+)\]', answer_text))

    citations = []
    for num in sorted(cited_nums):
        idx = num - 1
        if idx < len(chunks):
            meta = chunks[idx]['metadata']
            citations.append({
                'source_num': num,
                'title':     meta.get('title', 'Unknown'),
                'page':      meta.get('page', 'N/A'),
                'file':      meta.get('source_file', '')
            })

    return citations

def answer_with_citations(question):
    chunks = retrieve_above_threshold(question)
    context = format_chunks_for_prompt(chunks)
    answer = llm_call(QA_PROMPT.format(context=context, question=question))
    citations = extract_citations(answer, chunks)
    return {'answer': answer, 'citations': citations}

Re-ranking with Cross-Encoder

Initial vector retrieval uses bi-encoders (fast, approximate). A cross-encoder re-ranks the top results by scoring each (query, chunk) pair together — more accurate but slower. This two-stage approach improves final answer quality.

# pip install sentence-transformers
from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank_chunks(question, chunks, top_k=4):
    # Score each chunk against the question
    pairs = [(question, c['text']) for c in chunks]
    scores = reranker.predict(pairs)

    # Attach scores and re-sort
    scored_chunks = list(zip(scores, chunks))
    scored_chunks.sort(key=lambda x: x[0], reverse=True)

    top_chunks = [chunk for _, chunk in scored_chunks[:top_k]]
    print(f'Re-ranked {len(chunks)} chunks -> kept top {top_k}')
    return top_chunks

def answer_with_reranking(question):
    # Retrieve more initially
    initial_chunks = retrieve_relevant_chunks(question, n_results=12)
    # Re-rank for precision
    top_chunks = rerank_chunks(question, initial_chunks, top_k=4)
    context = format_chunks_for_prompt(top_chunks)
    return llm_call(QA_PROMPT.format(context=context, question=question))

Document-Level Metadata Filtering

When the user specifies a particular document or date range, filter at the metadata level before embedding search. This prevents irrelevant documents from polluting results.

def retrieve_filtered(question, filters=None, n_results=8):
    query_kwargs = {
        'query_texts': [question],
        'n_results': n_results,
        'include': ['documents', 'metadatas', 'distances']
    }

    # ChromaDB metadata filters
    # Example: {'doc_id': 'contract_2024', 'year': {'$gte': 2023}}
    if filters:
        query_kwargs['where'] = filters

    results = collection.query(**query_kwargs)
    return [
        {'text': t, 'metadata': m, 'relevance': 1 - d}
        for t, m, d in zip(
            results['documents'][0],
            results['metadatas'][0],
            results['distances'][0]
        )
    ]

# Example usage
chunks = retrieve_filtered(
    'What are the payment terms?',
    filters={'doc_id': {'$in': ['contract_a', 'contract_b']}}
)

Updating the Index with New Documents

Document collections change over time — new files are added, old ones are updated. The indexing pipeline must support incremental updates: add new documents, re-index updated ones, and remove deleted ones.

import os
import hashlib

# Track indexed documents by file hash
index_registry = {}  # {filepath: {hash, doc_id, indexed_at}}

def file_hash(filepath):
    with open(filepath, 'rb') as f:
        return hashlib.md5(f.read()).hexdigest()

def index_if_new_or_changed(filepath, title):
    fhash = file_hash(filepath)
    existing = index_registry.get(filepath)

    if existing and existing['hash'] == fhash:
        print(f'Skipping unchanged: {title}')
        return existing['doc_id']

    if existing:
        # Remove old chunks from vector store
        collection.delete(where={'doc_id': {'': existing['doc_id']}})
        print(f'Re-indexing updated: {title}')
    else:
        print(f'Indexing new: {title}')

    doc_id = hashlib.md5(filepath.encode()).hexdigest()[:8]
    index_document(doc_id, filepath, title)
    index_registry[filepath] = {'hash': fhash, 'doc_id': doc_id}
    return doc_id

def sync_document_directory(directory):
    pdf_files = [f for f in os.listdir(directory) if f.endswith('.pdf')]
    for fname in pdf_files:
        fpath = os.path.join(directory, fname)
        title = fname.replace('.pdf', '').replace('_', ' ').title()
        index_if_new_or_changed(fpath, title)
    print(f'Sync complete: {len(pdf_files)} files processed')

Knowledge Check

In a multi-document Q&A system using retrieval-augmented generation, what does the relevance threshold serve to prevent?

Recap: Multi-Document Q&A Agents

Multi-document Q&A: index all documents (parse → chunk → embed → store with metadata) → retrieve relevant chunks across all documents → format with source labels → synthesize answer with citations.

Advanced techniques: cross-document comparison for comparative questions, conflict detection prompting, relevance threshold filtering, cross-encoder re-ranking for precision, and metadata filtering to scope queries to specific documents or date ranges.

Frequently asked questions

Is the “Multi-Document Q&A Agents” lesson free?

Yes — the full text of “Multi-Document Q&A Agents” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Multi-Document Q&A Agents”?

Indexing a document corpus and answering questions across all documents. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Multi-Document Q&A Agents” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents