แบ่งข้อความเป็นส่วนเพื่อสร้างเวกเตอร์ฝังที่ดียิ่งขึ้น

Embedding models have a token limit and produce one vector per input. Feeding a whole document yields a vague, averaged vector. Chunking splits text into focused pieces so each vector captures a specific idea.

Chunk size is a balance:

• Too large — diluted meaning, mixed topics in one vector
• Too small — fragments lose context, more vectors to store

Aim for chunks that hold one coherent thought.

The simplest method: split every N characters or tokens.

Fixed splits can cut a sentence in half, losing context at the boundary. Adding overlap repeats the last few tokens of one chunk at the start of the next so ideas spanning a boundary survive.

Better than blind character splits: break on sentence boundaries, then group sentences up to a target size. Chunks end cleanly and read coherently.

Recursive splitting tries large separators first (paragraphs), then smaller ones (sentences, then words) until chunks fit the size limit. It respects document structure while guaranteeing size.

For Markdown, code, or HTML, split along structural elements:

• Markdown headers and sections
• Code functions or classes
• HTML sections and lists

This keeps related content together.

Models limit by tokens, not characters. Estimate tokens before embedding so chunks fit the model window.

Think about how users query. If questions target short facts, smaller chunks improve precision. If questions need broad context, larger chunks help. Sometimes you store both granularities.

A chunk taken out of context can be ambiguous. Prepend a short header (document title, section name) to each chunk before embedding so the vector knows where it came from.

There is no universal best chunk size. Pick a starting point (often a few hundred tokens with modest overlap), then measure retrieval quality and adjust. Chunking is an experiment, not a fixed rule.

Test your understanding of chunk overlap.

You learned that chunking turns documents into focused, embeddable pieces. Balance chunk size, add overlap to preserve boundary context, prefer sentence-aware or recursive splitting, count tokens, enrich chunks with context headers, and iterate while measuring retrieval quality.