문서 분할 사용자 지정
의미 기반 청킹과 코드 또는 특정 데이터 구조 처리를 포함한 고급 텍스트 분할 기법을 구현합니다.
문서 분할 사용자 지정은(는) CoddyKit의 무료 LangChain / RAG / Vector DBs 강의입니다. 이것은 4개 중 3번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 LangChain / RAG / Vector DBs 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. LangChain / RAG / Vector DBs 강의에는 총 4개의 강의가 포함되어 있습니다.
이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.
Why Customize Text Splitting?
When preparing documents for Retrieval Augmented Generation (RAG), how you split them into chunks is crucial. Basic text splitters are a good start, but they often fall short for complex or highly structured content.
Customizing your text splitting strategy allows you to maintain better contextual integrity, leading to more accurate retrievals and better LLM responses.
Tailoring Character Splitters
LangChain's CharacterTextSplitter is simple but powerful. You can customize it by providing specific separator characters. This is useful when your documents have unique delimiters you want to respect, like a specific tag or a unique line break pattern.
By defining your own separators, you can ensure logical breaks rather than arbitrary character counts.
from langchain.text_splitter import CharacterTextSplitter
class Main:
def run(self):
text = "Chapter 1: Intro.Section 1.1: Basics.Section 1.2: Advanced."
# Custom separator is "."
splitter = CharacterTextSplitter(
separator=".",
chunk_size=20,
chunk_overlap=0
)
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {chunk}")
if __name__ == "__main__":
Main().run()Refining Recursive Splitters
The RecursiveCharacterTextSplitter attempts to split text using a list of separators in order, trying to keep chunks as large as possible. You can customize this list to match your document's inherent structure.
For example, you might prioritize splitting by double newlines, then single newlines, then spaces, and finally characters.
from langchain.text_splitter import RecursiveCharacterTextSplitter
class Main:
def run(self):
text = "Hello there!\n\nThis is a paragraph.\nAnd this is another sentence."
# Custom list of separators
splitter = RecursiveCharacterTextSplitter(
separators=["\n\n", "\n", " ", ""],
chunk_size=40,
chunk_overlap=0
)
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {chunk}")
if __name__ == "__main__":
Main().run()Intro to Semantic Chunking
Instead of relying solely on character counts or delimiters, what if we could split text based on its meaning?
Semantic chunking aims to create chunks that represent complete, coherent ideas or topics. This method helps prevent important concepts from being arbitrarily split across different chunks, which often happens with fixed-size or simple character splitters.
How Semantic Chunking Works
Semantic chunking typically involves a few steps:
- Embed Sentences: Each sentence or a small unit of text is converted into a vector embedding.
- Measure Similarity: The semantic similarity between adjacent sentences or units is measured using their embeddings.
- Identify Breakpoints: Chunks are formed where semantic similarity drops significantly, indicating a topic change or a shift in discussion.
While LangChain doesn't have a single 'semantic splitter' out-of-the-box, it's a pattern you can build using embedding models and custom logic.
Specialized Code Splitters
Code has a unique structure, with functions, classes, comments, and specific syntax. Generic text splitters often break code in awkward places, making the resulting chunks hard to understand or use as context for an LLM.
LangChain provides specialized splitters for different programming languages. These splitters understand the syntax of a language and ensure that chunks are syntactically meaningful, like keeping a whole function or class together.
Python Code Splitter Demo
The RecursiveCharacterTextSplitter.from_language method allows you to specify a programming language. It then uses language-specific separators (like class definitions, function definitions, etc.) to create more intelligent chunks.
This ensures that code snippets passed to an LLM are more coherent.
from langchain.text_splitter import RecursiveCharacterTextSplitter, Language
class Main:
def run(self):
python_code = """
def calculate_sum(a, b):
# This function adds two numbers
return a + b
class MyCalculator:
def __init__(self):
self.result = 0
def add(self, num):
self.result += num
if __name__ == "__main__":
total = calculate_sum(5, 3)
print(f"Sum: {total}")
calc = MyCalculator()
calc.add(10)
print(f"Calc result: {calc.result}")
"""
# Initialize splitter for Python code
python_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON,
chunk_size=100, # Adjust chunk size to see more splits
chunk_overlap=0
)
docs = python_splitter.create_documents([python_code])
for i, doc in enumerate(docs):
print(f"--- Chunk {i+1} ---")
print(doc.page_content)
if __name__ == "__main__":
Main().run()Beyond Code: Other Structures
LangChain also offers specialized splitters for other structured formats, not just code:
MarkdownTextSplitter: Understands Markdown syntax (headings, code blocks, lists) to create logically grouped chunks.LatexTextSplitter: Recognizes LaTeX sections, chapters, and environments, preserving the document's academic structure.
These specialized splitters are invaluable for processing documents where the formatting itself conveys important structural information.
Creating Custom Splitter Logic
For truly unique document structures or proprietary data formats, you might need to implement your own splitting logic. LangChain allows you to:
- Subclass
TextSplitter: Create a new class that inherits fromTextSplitterand overrides its methods to define custom splitting rules. - Write a custom function: Develop a function that takes your text and returns a list of chunks based on your specific parsing requirements.
This approach offers maximum flexibility to handle complex regex patterns, custom delimiters, or nested structures unique to your dataset.
Check Your Understanding
You've learned about various ways to customize text splitting for different document types. Let's test your knowledge.
Custom Splitting Recap
In this lesson, we explored how to go beyond basic text splitting to handle diverse and complex document types more effectively:
- We customized Character and Recursive Character splitters with specific lists of separators.
- We introduced the concept of Semantic Chunking for meaning-based splits.
- We learned about Language-specific splitters for code (e.g., Python, Java) and other structured formats like Markdown and LaTeX.
- Finally, we discussed the power and flexibility of creating entirely custom splitting logic for unique data.
Mastering customized splitting strategies is a critical step in building accurate and robust RAG applications.
자주 묻는 질문
“문서 분할 사용자 지정” 강의는 무료인가요?
네 — “문서 분할 사용자 지정” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 LangChain / RAG / Vector DBs 강의 전체를 잠금 해제할 수 있습니다. LangChain / RAG / Vector DBs 강의에는 총 4개의 강의가 포함되어 있습니다.
“문서 분할 사용자 지정”에서 뭘 배우나요?
의미 기반 청킹과 코드 또는 특정 데이터 구조 처리를 포함한 고급 텍스트 분할 기법을 구현합니다. 브라우저에서 직접 실행하는 실습 코드로 LangChain / RAG / Vector DBs을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.
LangChain / RAG / Vector DBs을(를) 시작하는 데 경험이 필요한가요?
사전 경험은 필요하지 않습니다. CoddyKit의 LangChain / RAG / Vector DBs은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 3번째 강의입니다.
“문서 분할 사용자 지정” 강의는 얼마나 걸리나요?
대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.
이 LangChain / RAG / Vector DBs 강의에서 코드를 작성하고 실행할 수 있나요?
네. 모든 LangChain / RAG / Vector DBs 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.
이 강의의 모든 강의
- 다양한 문서 유형 로딩
- 텍스트 분할 전략 이해
- 문서 분할 사용자 지정
- 문서 메타데이터 처리와 필터링