0Pricing
LangChain / RAG / Vector DBs · 课时

开发自定义文档加载器

创建定制的文档加载器，从 LangChain 不直接支持的独特或专有数据源中摄取数据。

开发自定义文档加载器 是 CoddyKit 上的免费 LangChain / RAG / Vector DBs 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 LangChain / RAG / Vector DBs 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 LangChain / RAG / Vector DBs 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Custom Document Loaders?

LangChain offers many built-in document loaders for common formats like PDFs, web pages, and databases. But what if your data is unique?

Sometimes, you'll encounter:

  • Proprietary file formats
  • Internal APIs or data sources
  • Complex data structures needing custom parsing

This is where custom document loaders shine!

Meet LangChain's BaseLoader

To create your own loader, you'll inherit from LangChain's BaseLoader class. This is an abstract class, meaning it provides a template for what your loader needs to do.

The most important method you'll implement is load(). This method is responsible for fetching your data and transforming it into a list of Document objects.

The LangChain Document Object

All data processed by LangChain, especially for RAG, is standardized into Document objects. Each Document has two main parts:

  • page_content: The actual text content.
  • metadata: A dictionary of key-value pairs describing the document (e.g., source file, page number, author).

Your custom loader's job is to create these Document objects from your unique data.

Basic Custom Loader Structure

Let's start with a very simple custom loader that just returns a fixed text string as a document. This shows the basic structure of inheriting from BaseLoader and implementing load().

from langchain_core.documents import Document
from langchain_core.document_loaders import BaseLoader

class MySimpleTextLoader(BaseLoader):
    def load(self):
        content = "This is a custom text document from MySimpleTextLoader."
        doc = Document(page_content=content)
        return [doc]

# Example usage:
if __name__ == "__main__":
    loader = MySimpleTextLoader()
    documents = loader.load()
    for doc in documents:
        print(f"Content: {doc.page_content}")
        print(f"Metadata: {doc.metadata}")

Making Your Loader Dynamic

A fixed string loader isn't very useful! Real-world loaders need to take parameters, like a file path, a URL, or API credentials.

You can achieve this by adding an __init__ method to your custom loader class. This allows you to pass arguments when you create an instance of your loader.

Loading a 'Custom' Log File

Imagine you have a simple application log file (app.log) where each line is an event. Let's create a custom loader to read this file, treating each line as a separate document.

We'll create a dummy app.log file content directly in the code for simplicity.

from langchain_core.documents import Document
from langchain_core.document_loaders import BaseLoader

# Simulate a log file content
log_file_content = (
    "[INFO] User logged in: user123\n"
    "[ERROR] Database connection failed\n"
    "[DEBUG] Processing request for /api/data\n"
    "[INFO] Data retrieved successfully"
)

class CustomLogLoader(BaseLoader):
    def __init__(self, log_data: str):
        self.log_data = log_data.split('\n')

    def load(self):
        documents = []
        for line in self.log_data:
            if line.strip(): # Avoid empty lines
                doc = Document(page_content=line)
                documents.append(doc)
        return documents

# Example usage:
if __name__ == "__main__":
    loader = CustomLogLoader(log_file_content)
    documents = loader.load()
    for i, doc in enumerate(documents):
        print(f"Doc {i+1}: {doc.page_content[:40]}...")

Adding Rich Metadata

Metadata is incredibly useful! It helps the LLM understand the context of the text and can be used for filtering or improving retrieval. For our log file example, knowing the original log line number or the source file could be very helpful.

You can add any relevant information as key-value pairs to the metadata dictionary of a Document.

Log Loader with Metadata

Let's enhance our CustomLogLoader to include metadata like the original source and the line number for each log entry. This makes the retrieved information much richer!

from langchain_core.documents import Document
from langchain_core.document_loaders import BaseLoader

# Simulate a log file content
log_file_content = (
    "[INFO] User logged in: user123\n"
    "[ERROR] Database connection failed\n"
    "[DEBUG] Processing request for /api/data\n"
    "[INFO] Data retrieved successfully"
)

class CustomLogLoaderWithMeta(BaseLoader):
    def __init__(self, log_data: str, source_name: str = "app.log"):
        self.log_data = log_data.split('\n')
        self.source_name = source_name

    def load(self):
        documents = []
        for i, line in enumerate(self.log_data):
            if line.strip():
                metadata = {
                    "source": self.source_name,
                    "line_number": i + 1
                }
                doc = Document(page_content=line, metadata=metadata)
                documents.append(doc)
        return documents

# Example usage:
if __name__ == "__main__":
    loader = CustomLogLoaderWithMeta(log_file_content, "my_custom_app_logs")
    documents = loader.load()
    for i, doc in enumerate(documents):
        print(f"Doc {i+1}:")
        print(f"  Content: {doc.page_content[:40]}...")
        print(f"  Metadata: {doc.metadata}")

Integrating Custom Documents

Once your custom loader produces Document objects, they behave just like documents loaded by any other LangChain loader.

You can then pass them into subsequent steps of your RAG pipeline:

  • Text Splitting: Break large documents into smaller chunks.
  • Embeddings: Convert text chunks into numerical vectors.
  • Vector Stores: Store these embeddings for efficient similarity search.

Your custom data is now ready for advanced LLM applications!

Quick Check: Custom Loaders

You're building a custom document loader for LangChain. Which of the following statements is TRUE about the Document object you must return?

Recap: Custom Document Loaders

Great job! You've learned how to develop custom document loaders in LangChain.

  • You inherit from BaseLoader and implement the load() method.
  • Your loader converts unique data into a list of Document objects.
  • Document objects contain page_content and a flexible metadata dictionary.
  • Custom loaders are essential for integrating proprietary data sources into your RAG applications.

Next, we'll explore how to integrate custom embedding models!

常见问题解答

「开发自定义文档加载器」课时是免费的吗？

是的 — 「开发自定义文档加载器」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 LangChain / RAG / Vector DBs 课程的其余内容，请升级到 CoddyKit PRO。 LangChain / RAG / Vector DBs 课程共包含 4 节课。

「开发自定义文档加载器」这节课中我会学到什么？

创建定制的文档加载器，从 LangChain 不直接支持的独特或专有数据源中摄取数据。 你通过在浏览器中直接运行的动手代码来练习 LangChain / RAG / Vector DBs，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 LangChain / RAG / Vector DBs 需要有经验吗？

无需任何先前经验。CoddyKit 上的 LangChain / RAG / Vector DBs 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。

「开发自定义文档加载器」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 LangChain / RAG / Vector DBs 课中编写并运行代码吗？

能。每节 LangChain / RAG / Vector DBs 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 开发自定义文档加载器
  2. 集成自定义嵌入模型
  3. 使用自定义逻辑扩展检索链
  4. 构建自定义输出解析器
← 返回 LangChain / RAG / Vector DBs