LangChain / RAG / Vector DBs · 课时

自定义文档切分

实施包括语义分块，以及处理代码或特定数据结构在内的高级文本切分技术

自定义文档切分 是 CoddyKit 上的免费 LangChain / RAG / Vector DBs 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 LangChain / RAG / Vector DBs 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 LangChain / RAG / Vector DBs 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Customize Text Splitting?

When preparing documents for Retrieval Augmented Generation (RAG), how you split them into chunks is crucial. Basic text splitters are a good start, but they often fall short for complex or highly structured content.

Customizing your text splitting strategy allows you to maintain better contextual integrity, leading to more accurate retrievals and better LLM responses.

Tailoring Character Splitters

LangChain's CharacterTextSplitter is simple but powerful. You can customize it by providing specific separator characters. This is useful when your documents have unique delimiters you want to respect, like a specific tag or a unique line break pattern.

By defining your own separators, you can ensure logical breaks rather than arbitrary character counts.

from langchain.text_splitter import CharacterTextSplitter

class Main:
    def run(self):
        text = "Chapter 1: Intro.Section 1.1: Basics.Section 1.2: Advanced."
        # Custom separator is "."
        splitter = CharacterTextSplitter(
            separator=".",
            chunk_size=20,
            chunk_overlap=0
        )
        chunks = splitter.split_text(text)
        for i, chunk in enumerate(chunks):
            print(f"Chunk {i+1}: {chunk}")

if __name__ == "__main__":
    Main().run()

Refining Recursive Splitters

The RecursiveCharacterTextSplitter attempts to split text using a list of separators in order, trying to keep chunks as large as possible. You can customize this list to match your document's inherent structure.

For example, you might prioritize splitting by double newlines, then single newlines, then spaces, and finally characters.

from langchain.text_splitter import RecursiveCharacterTextSplitter

class Main:
    def run(self):
        text = "Hello there!\n\nThis is a paragraph.\nAnd this is another sentence."
        # Custom list of separators
        splitter = RecursiveCharacterTextSplitter(
            separators=["\n\n", "\n", " ", ""],
            chunk_size=40,
            chunk_overlap=0
        )
        chunks = splitter.split_text(text)
        for i, chunk in enumerate(chunks):
            print(f"Chunk {i+1}: {chunk}")

if __name__ == "__main__":
    Main().run()

Intro to Semantic Chunking

Instead of relying solely on character counts or delimiters, what if we could split text based on its meaning?

Semantic chunking aims to create chunks that represent complete, coherent ideas or topics. This method helps prevent important concepts from being arbitrarily split across different chunks, which often happens with fixed-size or simple character splitters.

How Semantic Chunking Works

Semantic chunking typically involves a few steps:

Embed Sentences: Each sentence or a small unit of text is converted into a vector embedding.
Measure Similarity: The semantic similarity between adjacent sentences or units is measured using their embeddings.
Identify Breakpoints: Chunks are formed where semantic similarity drops significantly, indicating a topic change or a shift in discussion.

While LangChain doesn't have a single 'semantic splitter' out-of-the-box, it's a pattern you can build using embedding models and custom logic.

Specialized Code Splitters

Code has a unique structure, with functions, classes, comments, and specific syntax. Generic text splitters often break code in awkward places, making the resulting chunks hard to understand or use as context for an LLM.

LangChain provides specialized splitters for different programming languages. These splitters understand the syntax of a language and ensure that chunks are syntactically meaningful, like keeping a whole function or class together.

Python Code Splitter Demo

The RecursiveCharacterTextSplitter.from_language method allows you to specify a programming language. It then uses language-specific separators (like class definitions, function definitions, etc.) to create more intelligent chunks.

This ensures that code snippets passed to an LLM are more coherent.

from langchain.text_splitter import RecursiveCharacterTextSplitter, Language

class Main:
    def run(self):
        python_code = """
def calculate_sum(a, b):
    # This function adds two numbers
    return a + b

class MyCalculator:
    def __init__(self):
        self.result = 0

    def add(self, num):
        self.result += num

if __name__ == "__main__":
    total = calculate_sum(5, 3)
    print(f"Sum: {total}")
    calc = MyCalculator()
    calc.add(10)
    print(f"Calc result: {calc.result}")
"""

        # Initialize splitter for Python code
        python_splitter = RecursiveCharacterTextSplitter.from_language(
            language=Language.PYTHON,
            chunk_size=100, # Adjust chunk size to see more splits
            chunk_overlap=0
        )

        docs = python_splitter.create_documents([python_code])

        for i, doc in enumerate(docs):
            print(f"--- Chunk {i+1} ---")
            print(doc.page_content)

if __name__ == "__main__":
    Main().run()

Beyond Code: Other Structures

LangChain also offers specialized splitters for other structured formats, not just code:

MarkdownTextSplitter: Understands Markdown syntax (headings, code blocks, lists) to create logically grouped chunks.
LatexTextSplitter: Recognizes LaTeX sections, chapters, and environments, preserving the document's academic structure.

These specialized splitters are invaluable for processing documents where the formatting itself conveys important structural information.

Creating Custom Splitter Logic

For truly unique document structures or proprietary data formats, you might need to implement your own splitting logic. LangChain allows you to:

Subclass TextSplitter: Create a new class that inherits from TextSplitter and overrides its methods to define custom splitting rules.
Write a custom function: Develop a function that takes your text and returns a list of chunks based on your specific parsing requirements.

This approach offers maximum flexibility to handle complex regex patterns, custom delimiters, or nested structures unique to your dataset.

Check Your Understanding

You've learned about various ways to customize text splitting for different document types. Let's test your knowledge.

Custom Splitting Recap

In this lesson, we explored how to go beyond basic text splitting to handle diverse and complex document types more effectively:

We customized Character and Recursive Character splitters with specific lists of separators.
We introduced the concept of Semantic Chunking for meaning-based splits.
We learned about Language-specific splitters for code (e.g., Python, Java) and other structured formats like Markdown and LaTeX.
Finally, we discussed the power and flexibility of creating entirely custom splitting logic for unique data.

Mastering customized splitting strategies is a critical step in building accurate and robust RAG applications.

常见问题解答

「自定义文档切分」课时是免费的吗？

是的 — 「自定义文档切分」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 LangChain / RAG / Vector DBs 课程的其余内容，请升级到 CoddyKit PRO。 LangChain / RAG / Vector DBs 课程共包含 4 节课。

「自定义文档切分」这节课中我会学到什么？

实施包括语义分块，以及处理代码或特定数据结构在内的高级文本切分技术你通过在浏览器中直接运行的动手代码来练习 LangChain / RAG / Vector DBs，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 LangChain / RAG / Vector DBs 需要有经验吗？

无需任何先前经验。CoddyKit 上的 LangChain / RAG / Vector DBs 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「自定义文档切分」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 LangChain / RAG / Vector DBs 课中编写并运行代码吗？

能。每节 LangChain / RAG / Vector DBs 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 LangChain / RAG / Vector DBs