Comprendre les stratégies de découpage du texte

Large Language Models (LLMs) have a 'context window' – a limit on how much text they can process at once. If you feed them a document that's too long, they simply can't handle it all.

Text splitting is the process of breaking down large documents into smaller, manageable chunks. This makes them suitable for LLMs and helps retrieval systems find more precise information.

Imagine an LLM as a very smart person with a short-term memory limit. The context window is like that limit. If you give it too much information, it might forget the beginning or get confused.

• LLMs can only process a certain number of tokens (words or sub-words).
• Going over this limit means information is truncated or ignored.
• Smaller chunks ensure all relevant information fits and is processed effectively.

One of the most straightforward ways to split text is using a CharacterTextSplitter. It simply breaks text based on a specified separator, usually a newline character (\n).

It's like cutting a long rope into smaller pieces at every knot you find. This method is easy to understand but can sometimes break sentences or paragraphs in awkward places.

Try running this example. Notice how the CharacterTextSplitter breaks the text primarily at each newline character.

The chunk_size parameter determines the maximum length of each piece of text. If a piece of text (before splitting by a separator) exceeds this size, the splitter will try to break it further.

Choosing the right size is crucial:

• Too small: Context might be lost across multiple chunks, making it harder for the LLM to understand the full picture.
• Too large: Might still exceed the LLM's context window or contain too much irrelevant information, diluting the focus.

chunk_overlap specifies how many characters (or tokens) each chunk shares with the previous one. This is vital to maintain continuity and prevent loss of context at the boundaries of chunks.

Imagine a sentence that gets split perfectly in half across two chunks. Without overlap, the LLM might miss the connection between the two halves. Overlap ensures that key phrases or ideas aren't cut off abruptly, providing a smoother flow of information.

The RecursiveCharacterTextSplitter is often preferred for general-purpose documents. Instead of just one separator, it tries a list of separators in order of preference (e.g., ["\n\n", "\n", " ", ""]).

It first tries to split by the largest, most semantically meaningful separator (like a double newline for paragraphs). If a chunk is still too big, it then tries the next smaller separator (like a single newline), and so on. This creates more semantically coherent chunks.

This example uses a recursive splitter. Notice how it prioritizes paragraph breaks (double newlines) to keep related sentences together.

When should you use which splitter?

• CharacterTextSplitter: Good for simple, highly structured text where you know the exact delimiters (e.g., CSV files, specific log formats).
• RecursiveCharacterTextSplitter: Generally the default and best choice for most general-purpose documents (like articles, reports), as it aims for more logical and semantically coherent breaks.
• Other splitters: LangChain offers specialized splitters for code, Markdown, and even semantic content. We'll touch on these in future lessons!

Great job! You've learned the fundamental concepts behind text splitting, a crucial step for preparing documents for LLMs.

• We split text due to LLM context window limits and for more effective retrieval.
• The CharacterTextSplitter provides basic splitting using a single separator.
• The RecursiveCharacterTextSplitter offers a smarter, hierarchical approach for general text.
• chunk_size controls the maximum length of your chunks.
• chunk_overlap preserves context by sharing text between adjacent chunks.

Next, we'll dive deeper into customizing splitting strategies for specific content types!