LLM Apps in Production (RAG + Vector DB + Caching) · 课时

处理复杂文档结构

实现从表格或嵌套章节等复杂文档中有效分块和检索信息的策略。

处理复杂文档结构 是 CoddyKit 上的免费 LLM Apps in Production (RAG + Vector DB + Caching) 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 LLM Apps in Production (RAG + Vector DB + Caching) 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 LLM Apps in Production (RAG + Vector DB + Caching) 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Beyond Simple Text: Complex Documents

When building RAG systems, we often deal with documents that aren't just plain, flowing text. Think about financial reports, scientific papers, or legal contracts.

These documents frequently contain tables, nested sections (like chapters and sub-chapters), and other intricate structures. Standard text chunking methods often struggle with these, breaking context and making retrieval less effective.

Tables: A Challenge for RAG

Tables are a prime example of complex structures. They present data in a structured, grid-like format where relationships between rows and columns are crucial.

Lost Context: A simple character-based chunker might split a table row, separating a value from its header, making the chunk meaningless.
Poor Embeddings: Without proper context, the generated embeddings for table fragments might not accurately represent the data.

Table-Aware Chunking Strategies

To effectively handle tables, we need specific strategies:

Extraction: Identify and extract tables as distinct entities.
Serialization: Convert tables into a more LLM-friendly text format, like Markdown or structured JSON, preserving their relationships.
Summarization: For very large tables, generate a concise summary to be embedded, linking back to the full table.

This ensures the LLM receives the full, meaningful context of the table.

Extracting Table Data (Python)

Here's a simple Python example showing how you might process a table represented as a string, converting it into a more structured list of rows.

import csv
import io

def process_table_string(table_str):
    # Use StringIO to treat the string as a file
    f = io.StringIO(table_str)
    reader = csv.reader(f, delimiter='|')
    
    rows = []
    for i, row in enumerate(reader):
        # Strip whitespace and filter empty strings
        cleaned_row = [item.strip() for item in row if item.strip()]
        if cleaned_row and i > 0: # Skip header line
            rows.append(cleaned_row)
    return rows

if __name__ == "__main__":
    data = """
    Name   | Age | City   
    -------|-----|--------
    Alice  | 30  | New York
    Bob    | 24  | London 
    Charlie| 35  | Paris  
    """
    
    processed_data = process_table_string(data)
    for row in processed_data:
        print(row)

Understanding Nested Documents

Documents often have a natural hierarchy. Think of a book with chapters, sections, and subsections. Each part builds on the previous one, and its meaning is often tied to its parent context.

Standard chunking might split a subsection from its main section's heading, making the retrieved chunk less informative or even confusing without the proper context.

Preserving Document Hierarchy

To handle nested structures effectively, we use hierarchical chunking:

Semantic Boundaries: Instead of fixed character counts, chunk based on logical divisions like headings (H1, H2, H3).
Parent Context: Include the title of the parent section in the child chunk. For example, a chunk from 'Section 2.1' might start with 'Chapter 2: Introduction - Section 2.1: Subtopic'.
Metadata: Store the full path or hierarchy level in the chunk's metadata.

Chunking by Sections (Python)

This Python example demonstrates a simple way to split a document into chunks based on markdown-style headings. Each chunk will contain a section's content.

def chunk_by_headings(document_text):
    lines = document_text.split('\n')
    chunks = []
    current_chunk = []
    current_heading = ""

    for line in lines:
        if line.startswith('# '): # Main heading
            if current_chunk:
                chunks.append({'heading': current_heading, 'content': '\n'.join(current_chunk).strip()})
            current_heading = line.strip()
            current_chunk = [line]
        elif line.startswith('## '):
            # Sub-heading, can be part of the current chunk, or signal a new sub-chunk
            # For simplicity, we'll just add it to the current chunk content here
            # More advanced logic might create nested chunks or separate entries
            current_chunk.append(line)
        else:
            current_chunk.append(line)
    
    if current_chunk:
        chunks.append({'heading': current_heading, 'content': '\n'.join(current_chunk).strip()})
    return chunks

if __name__ == "__main__":
    doc = """
# Chapter 1: Introduction
This is the introduction text.

## Section 1.1: Background
More details about the background.

# Chapter 2: Methods
Here we describe the methods used.

## Section 2.1: Data Collection
How data was collected.
"""
    
    document_chunks = chunk_by_headings(doc)
    for i, chunk in enumerate(document_chunks):
        print(f"--- Chunk {i+1} ---")
        print(f"Heading: {chunk['heading']}")
        print(f"Content snippet: {chunk['content'][:50]}...")
        print()

Metadata for Richer Context

Metadata is extra information attached to a chunk that describes it without being part of the chunk's main text. It's incredibly powerful for complex documents.

Document Title: Which source document does this chunk come from?
Page Number: Where in the original document was this found?
Parent Section/Chapter: What larger context does this chunk belong to?
Table ID: If it's a table, which table is it?

Metadata allows for targeted filtering during retrieval and provides valuable context to the LLM.

Beyond Single-Vector Retrieval

For highly complex content, simple text chunks might not be enough. Multi-vector retrieval is an advanced technique where you create different types of embeddings for the same content.

For example, you could have a small, concise summary of a table embedded for quick retrieval, and the full, detailed table content stored separately. The RAG system retrieves the summary, and if relevant, then fetches the full table to pass to the LLM.

Complex Document Check

You've learned about various strategies for handling complex document structures in RAG. Let's test your understanding.

Recap: Mastering Complex Docs

Congratulations! You've explored critical strategies for handling complex document structures in RAG.

We saw how tables can lose context with standard chunking and learned to extract and serialize them.
We discussed nested documents and the importance of hierarchical chunking to preserve relationships.
Finally, we highlighted the power of metadata to enrich chunks and enable more precise retrieval.

By applying these techniques, your RAG system can deliver more accurate and contextually relevant responses, even from the most intricate documents!

常见问题解答

「处理复杂文档结构」课时是免费的吗？

是的 — 「处理复杂文档结构」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 LLM Apps in Production (RAG + Vector DB + Caching) 课程的其余内容，请升级到 CoddyKit PRO。 LLM Apps in Production (RAG + Vector DB + Caching) 课程共包含 4 节课。

「处理复杂文档结构」这节课中我会学到什么？

实现从表格或嵌套章节等复杂文档中有效分块和检索信息的策略。你通过在浏览器中直接运行的动手代码来练习 LLM Apps in Production (RAG + Vector DB + Caching)，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 LLM Apps in Production (RAG + Vector DB + Caching) 需要有经验吗？

无需任何先前经验。CoddyKit 上的 LLM Apps in Production (RAG + Vector DB + Caching) 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「处理复杂文档结构」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 LLM Apps in Production (RAG + Vector DB + Caching) 课中编写并运行代码吗？

能。每节 LLM Apps in Production (RAG + Vector DB + Caching) 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 LLM Apps in Production (RAG + Vector DB + Caching)