The RAG Architecture: Indexing and Retrieval

A RAG system operates in two fundamentally different phases that run at different times. The offline indexing phase processes your documents once (or when they change) and prepares a searchable index. The online retrieval phase runs in real time for every user query. Understanding this split is essential for designing systems that are both fast at query time and maintainable over time.

Indexing starts with document loading: reading raw files from your source systems. Documents can be PDFs, Word files, HTML pages, Markdown files, database rows, or any text source. Each document is loaded into memory as plain text, preserving structure where possible. Libraries like pypdf, python-docx, and unstructured handle the heavy lifting of format-specific parsing.

LLMs have finite context windows and retrieving entire documents is wasteful. The loaded text is split into smaller chunks of roughly 200-1000 tokens each. Good chunking preserves semantic coherence: a chunk should express a complete idea. A common strategy uses overlapping windows so that sentences near chunk boundaries appear in two chunks, preventing information loss at split points.

Each text chunk is converted to a dense vector embedding that captures its semantic meaning numerically. You call an embedding model — such as OpenAI's text-embedding-3-small — for each chunk and receive a high-dimensional float array. Chunks with similar meaning produce vectors that are close in this high-dimensional space, which is what enables semantic similarity search.

The embedded chunks are stored in a vector database alongside their metadata (source file, page number, section title). The vector store builds an index structure (typically HNSW) that enables fast approximate nearest neighbor search. This index is persisted to disk so it survives restarts. Indexing typically happens once at setup and incrementally when new documents are added.

When a user submits a query, the online retrieval phase begins. The first step is to embed the user's question using the same embedding model used during indexing. This is critical: if you indexed with text-embedding-3-small, you must query with text-embedding-3-small too. The query embedding is a vector that encodes the semantic meaning of what the user is asking.

The query embedding is sent to the vector store, which performs an approximate nearest neighbor (ANN) search to find the K chunks whose embeddings are most similar to the query vector. This search is extremely fast (typically sub-10ms) because HNSW indexes trade a small amount of recall for massive speed gains over brute-force search. You retrieve the top-K chunks — commonly K=5 to K=20.

The key insight is that indexing and retrieval are designed to work together. The embedding model must be identical in both phases, because the mathematical space where vectors live is model-specific. If you switch embedding models, you must re-index all documents. The vector store is the bridge: it accepts vectors during indexing and returns vectors during retrieval, decoupling the two phases in time while keeping them aligned in vector space.

Vector search finds semantically similar chunks, but sometimes you also need to filter by metadata. For example: only retrieve chunks from documents uploaded in 2025, or only from the HR department folder. Vector stores support pre-filtering or post-filtering on metadata fields. Pre-filtering (supported by Pinecone and Qdrant) applies the filter before ANN search, which is faster and more accurate than post-filtering on the top-K results.

In production, your document corpus changes over time. An effective indexing architecture supports incremental updates: when a document is edited, delete its existing vectors by ID and upsert the new ones. When documents are deleted, remove their vectors. Assign each chunk a deterministic ID based on the source document path and chunk position so you can always find and update the right vectors without re-indexing everything.

The complete RAG architecture looks like this: Offline: Documents → Loader → Chunker → Embedding Model → Vector Store. Online: User Query → Embedding Model → Vector Store (ANN search) → Top-K Chunks → Prompt Assembly → LLM → Answer. The offline pipeline runs once per document update. The online pipeline runs in milliseconds for every user query, with the LLM seeing only the relevant context, not the entire document corpus.

Test your understanding of AI Engineering concepts from this lesson.

In this lesson you learned: the offline indexing phase consisting of load, chunk, embed, and store steps that run once per document, the online retrieval phase that embeds the query, performs ANN search, and returns top-K chunks in milliseconds, and incremental indexing strategies for keeping the vector store up to date as documents change. Next up we explore how to craft effective augmented prompts that use retrieved context well.