문서 분할 및 임베딩

A long PDF holds many topics at once. To retrieve precisely, you first split it into smaller pieces called chunks.

A good chunk is one coherent thought: a paragraph or two. Too big buries the answer; too small loses the surrounding context.

The simplest method cuts text every fixed number of characters or tokens. It is quick but can slice a sentence in half.

To avoid cutting an idea apart, let chunks share an overlap. Repeating the last lines keeps context flowing across boundaries.

Smarter splitters break on paragraphs or headings first. Respecting structure keeps each chunk on a single, clean topic.

Search needs numbers, not words. An embedding turns each chunk into a dense vector that captures its meaning.

Embeddings place similar text close together. Two chunks about the same idea sit near each other in vector space.

You pass text to an embedding model and get back a list of floats. The same model must encode both chunks and queries.

Run every chunk through the model to build its vector. One call can encode the whole list at once for speed.

Each vector has a fixed length, its dimension. A small model may give 384 numbers; larger ones give 768 or more.

Keep each vector linked to its original text and source. Later you retrieve by vector but show the human-readable chunk.

Consider why we add overlap when splitting documents.

You split documents into chunks, optionally overlapping, then embed each into a vector. Store text and vector together for retrieval. ✅