LLM Apps in Production (RAG + Vector DB + Caching) · 강의

데이터 로딩과 텍스트 분할 기초

비정형 데이터를 불러오고 최적의 검색 성능을 위한 효과적인 텍스트 분할 전략을 적용하는 방법을 배웁니다.

데이터 로딩과 텍스트 분할 기초은(는) CoddyKit의 무료 LLM Apps in Production (RAG + Vector DB + Caching) 강의입니다. 이것은 4개 중 2번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 LLM Apps in Production (RAG + Vector DB + Caching) 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. LLM Apps in Production (RAG + Vector DB + Caching) 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

Loading Data for RAG

Welcome to Lesson 2! In Retrieval Augmented Generation (RAG), the first step is always to get your data ready. This means loading your information and preparing it for the Large Language Model (LLM).

Most real-world data is unstructured, meaning it doesn't fit neatly into rows and columns like a spreadsheet. Think of documents, web pages, or books.

Common Unstructured Data Sources

RAG systems can work with many types of unstructured data. Here are some common examples:

Text files (.txt): Simple, plain text documents.
PDFs (.pdf): Often contain text, images, and complex layouts.
Word Documents (.docx): Rich text with formatting.
Web Pages (.html): Content from websites.
Databases/APIs: Text extracted from various fields.

The goal is to extract the raw text content from these sources.

Basic Text File Loading

Let's start with the simplest form: loading a plain text file. In Python, you can easily read the entire content of a file into a string.

This example creates a small sample.txt and then reads its content.

import os

def load_text_file(filepath):
    with open(filepath, 'r', encoding='utf-8') as f:
        return f.read()

if __name__ == "__main__":
    # Create a dummy file for demonstration
    file_content = "This is the first line.\nThis is the second line.\nAnd a final line of text."
    with open("sample.txt", "w", encoding="utf-8") as f:
        f.write(file_content)
    
    # Load and print the content
    loaded_data = load_text_file("sample.txt")
    print("--- Loaded Content ---")
    print(loaded_data)

    # Clean up the dummy file
    os.remove("sample.txt")

Why Text Chunking is Essential

Once you've loaded your data, you can't usually send an entire book or long document directly to an LLM. Why not?

Context Window Limits: LLMs have a maximum amount of text they can process at once.
Cost: Sending very long texts is expensive, as you're typically charged per token.
Relevance: Shorter, focused pieces of text are often more relevant for retrieval.

This is where text chunking comes in.

Understanding the Context Window

The context window is like an LLM's short-term memory. It's the maximum number of tokens (words or sub-words) it can consider when generating a response.

If your input text is too long, it gets truncated.
The LLM only 'sees' what's in its context window.

Chunking breaks your big document into smaller, manageable pieces that fit within this window.

Basic Chunking: Fixed Size

The simplest chunking strategy is fixed-size chunking. You define a specific number of characters or tokens, and then split your document into chunks of that exact size.

For example, if you have a 1000-character document and a chunk size of 100, you'll get 10 chunks.

Pros: Easy to implement.
Cons: Can cut sentences or paragraphs in half, losing context.

Fixed-Size Chunking in Action

Here's a Python example demonstrating fixed-size chunking. Notice how the text is simply cut at regular intervals, which might sometimes break words or sentences.

def fixed_size_chunker(text, chunk_size):
    chunks = []
    for i in range(0, len(text), chunk_size):
        chunks.append(text[i : i + chunk_size])
    return chunks

if __name__ == "__main__":
    sample_text = "Large language models are powerful tools for text generation and understanding. However, they have limitations, especially with very long inputs due to their context window size."
    
    chunk_size = 40
    chunks = fixed_size_chunker(sample_text, chunk_size)
    
    print(f"Original text length: {len(sample_text)}")
    print(f"Chunk size: {chunk_size}")
    print("--- Chunks ---")
    for i, chunk in enumerate(chunks):
        print(f"Chunk {i+1} ({len(chunk)} chars): '{chunk}'")

Improving Context with Overlap

Fixed-size chunking can be problematic if important context is split across two chunks. To mitigate this, we use overlapping chunks.

With overlap, each new chunk starts a bit before the previous one ended. This ensures that some text appears in multiple chunks, preserving continuity.

A common overlap size is 10-20% of the chunk size.
It helps the LLM connect ideas even if they span chunk boundaries.

Overlapping Chunking Example

See how adding an overlap helps maintain context. The start of each new chunk includes some text from the end of the previous one.

def overlapping_chunker(text, chunk_size, overlap_size):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap_size
        if start < 0: # Handle cases where overlap > chunk_size initially
            start = 0
    return chunks

if __name__ == "__main__":
    text_data = "The quick brown fox jumps over the lazy dog. Dogs are mammals and often friendly animals."
    
    chunk_size = 30
    overlap_size = 10
    chunks = overlapping_chunker(text_data, chunk_size, overlap_size)
    
    print(f"Original text length: {len(text_data)}")
    print(f"Chunk size: {chunk_size}, Overlap size: {overlap_size}")
    print("--- Overlapping Chunks ---")
    for i, chunk in enumerate(chunks):
        print(f"Chunk {i+1} ({len(chunk)} chars): '{chunk}'")

Check Your Understanding

You've learned about loading data and basic chunking strategies. Now, let's test your knowledge!

Recap: Data Loading & Chunking

Great job! In this lesson, we covered the foundational steps of preparing data for RAG applications:

Data Loading: Extracting raw text from various unstructured sources like text files, PDFs, and web pages.
Text Chunking: The essential process of breaking down long documents into smaller, manageable pieces.
Context Window: Understanding the LLM's limitation on input text length (measured in tokens).
Chunking Strategies: Explored basic fixed-size chunking and the improved method of fixed-size chunking with overlap to preserve context.

Next, we'll see how these chunks are used to build a simple RAG pipeline!

자주 묻는 질문

“데이터 로딩과 텍스트 분할 기초” 강의는 무료인가요?

네 — “데이터 로딩과 텍스트 분할 기초” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 LLM Apps in Production (RAG + Vector DB + Caching) 강의 전체를 잠금 해제할 수 있습니다. LLM Apps in Production (RAG + Vector DB + Caching) 강의에는 총 4개의 강의가 포함되어 있습니다.

“데이터 로딩과 텍스트 분할 기초”에서 뭘 배우나요?

비정형 데이터를 불러오고 최적의 검색 성능을 위한 효과적인 텍스트 분할 전략을 적용하는 방법을 배웁니다. 브라우저에서 직접 실행하는 실습 코드로 LLM Apps in Production (RAG + Vector DB + Caching)을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

LLM Apps in Production (RAG + Vector DB + Caching)을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 LLM Apps in Production (RAG + Vector DB + Caching)은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 2번째 강의입니다.

“데이터 로딩과 텍스트 분할 기초” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 LLM Apps in Production (RAG + Vector DB + Caching) 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 LLM Apps in Production (RAG + Vector DB + Caching) 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← LLM Apps in Production (RAG + Vector DB + Caching)(으)로 돌아가기