0Pricing
AI Agents · Lesson

Indexing a Document Set

Embed every chunk and store the vectors in an index — the offline preparation step of any RAG system.

Indexing a Document Set is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Ingestion Pipeline

The offline RAG pipeline has four steps:

  1. Load documents (PDF, HTML, MD)
  2. Chunk each document
  3. Embed each chunk
  4. Store the vectors + metadata in an index

Step 1: Load Documents

Use specialized loaders for different formats:

# PDFs
from pypdf import PdfReader
text = ''
for page in PdfReader('doc.pdf').pages:
    text += page.extract_text()

# HTML
from bs4 import BeautifulSoup
text = BeautifulSoup(html, 'html.parser').get_text()

# Markdown — just read the file
text = open('doc.md').read()

Step 2: Chunk

Use the splitter from the previous lesson:

from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_text(text)

Step 3: Embed in Batches

Embed many chunks per API call:

from openai import OpenAI
client = OpenAI()

def embed_batch(texts, batch_size=100):
    vectors = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        resp = client.embeddings.create(model='text-embedding-3-small', input=batch)
        vectors.extend(d.embedding for d in resp.data)
    return vectors

Step 4: Store

For 10k-50k chunks, FAISS or Chroma is enough:

import chromadb
client = chromadb.Client()
collection = client.create_collection('docs')

collection.add(
    ids=[f'doc-{i}' for i in range(len(chunks))],
    embeddings=vectors,
    documents=chunks,
    metadatas=[{'source': 'doc.pdf', 'page': i} for i in range(len(chunks))]
)

Idempotent Ingestion

Make ingestion safe to re-run. Use deterministic IDs (hash of content) so re-running does not duplicate:

import hashlib

def chunk_id(source, text):
    h = hashlib.sha256(text.encode()).hexdigest()[:16]
    return f'{source}:{h}'

print(chunk_id('handbook.pdf', 'Employees get 20 vacation days per year.'))
print(chunk_id('handbook.pdf', 'Employees get 20 vacation days per year.'))

Incremental Updates

When a document changes:

  1. Compute new chunks
  2. Diff against existing IDs
  3. Delete removed, add new, keep unchanged

Naive approach (delete all + re-insert) is fine for small corpora but wastes embedding API calls.

Metadata for Filtering

Include any field you might want to filter by later:

metadata = {
    'source': '/docs/handbook.pdf',
    'page': 42,
    'department': 'engineering',
    'updated_at': '2024-08-12',
    'access_level': 'internal'
}
for k, v in metadata.items():
    print(f"{k}: {v}")

Progress and Checkpoints

For large ingestions, log progress and checkpoint:

for i, batch in enumerate(batches):
    embed_and_store(batch)
    if i % 10 == 0:
        save_checkpoint(i)
        print(f'Processed {i * 100} chunks')

Rate Limits

OpenAI embeddings have a high but real rate limit. Use semaphores or `tenacity` retries:

from asyncio import Semaphore
sem = Semaphore(10)  # 10 concurrent embed calls max

async def safe_embed(batch):
    async with sem:
        return await client.embeddings.create(...)

Sanity Check the Index

After ingestion, run a few test queries and inspect results manually. If nothing relevant comes up, your chunking or embedding is broken — find out before users do.

Versioning the Index

Version the index so you can swap to a new chunking or embedding model without downtime:

collection = client.create_collection(f'docs-v2-{date.today()}')
# old collection stays live until new one is validated

Idempotent IDs

Why use content hashes as chunk IDs?

Recap

Load -> chunk -> embed -> store, with idempotent IDs and metadata. The index is the foundation of every downstream retrieval step.

Frequently asked questions

Is the “Indexing a Document Set” lesson free?

Yes — the full text of “Indexing a Document Set” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Indexing a Document Set”?

Embed every chunk and store the vectors in an index — the offline preparation step of any RAG system. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Indexing a Document Set” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. What RAG Solves (Knowledge Cut-off, Hallucinations)
  2. Chunking Strategies (Fixed, Sentence, Semantic)
  3. Indexing a Document Set
  4. Building a Naive RAG with FAISS or Chroma
← Back to AI Agents