Strategi Pembagian Potongan untuk RAG

Before embedding, documents are split into chunks. Chunk quality directly determines retrieval quality — too large dilutes relevance, too small loses context.

The simplest method splits text every N characters or tokens. Fast but can cut sentences mid-thought.

Add overlap between chunks so context spanning a boundary is not lost. A 10-20% overlap is common.

Split on sentence boundaries so each chunk stays grammatically whole. This usually improves embedding quality over raw character splits.

Recursive splitting tries large separators first (paragraphs), then smaller ones (sentences, words) until chunks fit the size limit. It respects natural structure.

Semantic chunking groups sentences by meaning similarity, starting a new chunk when topic shifts. More expensive but very precise.

Match chunk size to your embedding model's context window and your LLM's prompt budget.

• Small chunks: precise but fragmented
• Large chunks: rich but noisy

Store source, page, and position metadata with each chunk so you can cite sources and reconstruct context later.

Embed small child chunks for precise matching, but return their larger parent chunk to the LLM for full context. Best of both worlds.

Test different chunking configs with the same queries and measure retrieval hit rate. There is no universal best — measure for your data.

A solid starting point:

• Recursive splitting
• ~500 tokens per chunk
• ~50-100 token overlap
• Attach source metadata

You learned fixed, overlapping, sentence-aware, recursive, and semantic chunking, plus parent-child retrieval and metadata. Good chunking is the foundation of accurate RAG retrieval.