Chunking Strategies: Fixed vs Sentence vs Recursive

Chunking is the process of splitting loaded documents into smaller pieces that fit within the LLM's context window and can be individually indexed and retrieved. The way you chunk determines retrieval quality more than almost any other factor. A chunk that splits an answer across two pieces means neither chunk alone is sufficient to answer the question. A chunk that mixes two unrelated topics gets retrieved for questions about both but is useful for neither.

Fixed-size chunking splits text into chunks of exactly N characters or N tokens, regardless of sentence or paragraph boundaries. It is the simplest strategy and fast to implement. The major drawback is that it often cuts sentences in half, creating chunks that start or end mid-thought. This is acceptable for dense, uniformly formatted text like database export dumps, but produces poor retrieval quality on narrative prose or technical documentation.

The key improvement to fixed-size chunking is overlap: each chunk shares N characters with the previous chunk. This ensures that information near a chunk boundary appears in both adjacent chunks. With 50-100 token overlap, a sentence that crosses a boundary will be fully captured in at least one of the two chunks. More overlap means better coverage but larger index size and redundant content in retrieval results.

Sentence-boundary chunking uses NLP libraries like nltk or spacy to detect sentence ends before splitting. This guarantees that no sentence is cut in half. You accumulate sentences until adding the next one would exceed the target size, then start a new chunk. The result is chunks that always contain complete thoughts, which produces significantly better embedding quality than fixed-character splitting.

Recursive character splitting is the most widely used strategy in production RAG systems. It tries a hierarchy of separators in order: first paragraph breaks (\n\n), then newlines (\n), then sentence-ending periods, then spaces, and finally individual characters. It splits at the largest meaningful boundary that keeps the chunk under the target size, resulting in chunks that respect document structure as much as possible.

Character counts are an imprecise proxy for token counts. A 1000-character chunk might be 200 tokens or 400 tokens depending on word length and language. For precise control, split by token count using tiktoken. This matters for fitting chunks to the model's context window and for accurate cost estimation. LangChain's TokenTextSplitter wraps tiktoken for token-aware chunking.

Chunk size is a critical hyperparameter. Small chunks (100-200 tokens) are precise — they contain tightly focused information that embeds well. But they lose surrounding context, so the LLM may not have enough information to answer. Large chunks (500-1000 tokens) provide more context but their embeddings average over a broader topic, making them harder to retrieve for specific queries. Most production systems use 256-512 tokens with empirical testing on their data.

A powerful enhancement is contextual chunk headers: prepend the document title and section heading to each chunk's text before embedding. This means the embedding captures not just the chunk's content but also where it came from in the document. A chunk reading Benefits and Vacation Policy: Employees accrue 15 days annually... retrieves much more accurately for questions about vacation policy than the same text without the header.

No single chunking strategy is universally best. Build a quick evaluation: take 20 questions you know the answers to, run retrieval with each chunking strategy, and measure how often the correct chunk ranks in the top 5 results (hit rate@5). This takes an hour to set up and saves weeks of guessing. You will often find that your specific document format (dense legal prose vs. structured technical docs) strongly favors one strategy over others.

Some document types need special-purpose chunking. Code files should be split by function or class boundaries, not character count. Markdown files should be split at heading boundaries so each chunk corresponds to one section. Tables should be kept intact as a single chunk (splitting mid-table makes the content uninterpretable). Plan your chunking strategy based on the document types in your corpus rather than applying a one-size-fits-all splitter.

Every chunk needs a stable, unique ID derived from its source document and position. Use this ID to update specific chunks when documents change without re-indexing the entire corpus. A deterministic hash of the source path plus chunk index works well. Also record chunk position (chunk 3 of 12 from document X) in metadata — this helps re-assemble full sections when multiple adjacent chunks are retrieved together.

Test your understanding of AI Engineering concepts from this lesson.

In this lesson you learned: fixed-size chunking is simple but cuts sentences in half, sentence-boundary chunking preserves complete thoughts, recursive character splitting respects document structure and is the most common production choice, and advanced techniques including token-aware splitting, contextual headers, special-purpose splitters for Markdown and code, and stable chunk IDs for incremental updates. Next up we embed and store these chunks in a vector database.