テキスト分割と埋め込み
大きな文書を扱いやすいチャンクに分割し、セマンティック検索用の数値埋め込みを生成する手法を身につけます。
「テキスト分割と埋め込み」はCoddyKit上の無料AI Agents with LangChain & Autonomous Workflowsレッスンです。 これはレッスン2/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはAI Agents with LangChain & Autonomous Workflows学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 AI Agents with LangChain & Autonomous Workflowsコースには全4レッスンが含まれています。
このレッスンの一部はまだ翻訳されておらず、英語で表示されています。
Why Split & Embed Text?
When working with large documents, directly feeding them to a Large Language Model (LLM) often causes problems. LLMs have strict input limits, known as context windows.
This lesson teaches you how to prepare large texts for LLMs using two key techniques: text splitting and embeddings. These are essential for building advanced AI agents.
The Problem: Long Documents
Imagine you have a 100-page PDF document. If you try to ask an LLM a question about it, you can't just send the whole document.
- Context Window Limits: LLMs can only process a certain amount of text at once (e.g., 4,000 to 128,000 tokens).
- Cost: Longer inputs mean higher API costs.
- Relevance: Filling the context window with irrelevant information can make the LLM 'forget' the important parts.
Text splitting solves this by breaking documents into smaller, manageable chunks.
Introducing Text Splitters
LangChain provides various text splitters to divide documents efficiently. Their goal is to keep semantically related pieces of text together while respecting size limits.
Instead of just cutting at arbitrary character counts, smart splitters try to break text at logical points, like paragraphs or sentences.
A common and versatile splitter is the RecursiveCharacterTextSplitter.
Recursive Character Text Splitter
The RecursiveCharacterTextSplitter is a powerful tool. It attempts to split text using a list of characters, trying them in order until the chunks are small enough.
- It starts by trying to split by
\n\n(double newline for paragraphs). - If chunks are still too big, it tries
\n(single newline for lines). - Then spaces, and finally individual characters.
This recursive approach helps maintain semantic coherence.
Code: Basic Splitting Demo
Let's see how RecursiveCharacterTextSplitter works. We'll split a short story into chunks.
from langchain_text_splitters import RecursiveCharacterTextSplitter
story = (
"Alice was beginning to get very tired of sitting by her sister on the bank, "
"and of having nothing to do: once or twice she had peeped into the book her "
"sister was reading, but it had no pictures or conversations in it, 'and what "
"is the use of a book,' thought Alice 'without pictures or conversation?'"
"So she was considering in her own mind (as well as she could, for the hot "
"day made her feel very sleepy and stupid), whether the pleasure of making "
"a daisy-chain would be worth the trouble of getting up and picking the "
"daisies, when suddenly a White Rabbit with pink eyes ran close by her."
)
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=100,
chunk_overlap=0
)
chunks = text_splitter.split_text(story)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {chunk}\n")Chunk Size & Overlap Explained
Two crucial parameters for text splitting are chunk_size and chunk_overlap:
- Chunk Size: This is the maximum number of characters (or tokens, depending on the splitter) in each chunk. Choose a size that fits within your LLM's context window.
- Chunk Overlap: This specifies how many characters (or tokens) should overlap between consecutive chunks. Overlap helps preserve context across splits, ensuring that important information isn't lost at chunk boundaries.
Finding the right balance for these parameters is key to effective retrieval.
Code: Splitting with Overlap
Let's modify our previous example to use a chunk_overlap. Notice how parts of the text are repeated in adjacent chunks, providing continuity.
from langchain_text_splitters import RecursiveCharacterTextSplitter
story = (
"Alice was beginning to get very tired of sitting by her sister on the bank, "
"and of having nothing to do: once or twice she had peeped into the book her "
"sister was reading, but it had no pictures or conversations in it, 'and what "
"is the use of a book,' thought Alice 'without pictures or conversation?'"
"So she was considering in her own mind (as well as she could, for the hot "
"day made her feel very sleepy and stupid), whether the pleasure of making "
"a daisy-chain would be worth the trouble of getting up and picking the "
"daisies, when suddenly a White Rabbit with pink eyes ran close by her."
)
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=100,
chunk_overlap=20 # Added overlap
)
chunks = text_splitter.split_text(story)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {chunk}\n")What are Text Embeddings?
Once you've split your documents, how do you find the most relevant chunks for a user's query? This is where text embeddings come in.
- An embedding is a numerical representation (a vector) of text.
- Texts with similar meanings have embeddings that are 'closer' to each other in a high-dimensional space.
- This allows us to perform semantic search: finding chunks that are conceptually similar to a query, not just keyword matches.
Embeddings are the backbone of Retrieval Augmented Generation (RAG).
Generating Embeddings with LangChain
LangChain makes it easy to generate embeddings using various models. You interact with an Embeddings object, which abstracts away the underlying model details.
Popular embedding models include those from OpenAI, Hugging Face, Cohere, and many open-source options like `all-MiniLM-L6-v2`.
You typically initialize an embedding model and then call its embed_query() for a single text or embed_documents() for a list of chunks.
Code: Creating Embeddings
Here's how to generate an embedding for a simple text using OpenAI's embedding model. Remember, you'll need an OpenAI API key for this to run successfully.
import os
# Set your OpenAI API key as an environment variable
# os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"
from langchain_openai import OpenAIEmbeddings
# Initialize the embedding model
# Requires OPENAI_API_KEY env var or direct pass
embeddings_model = OpenAIEmbeddings()
text_to_embed = "The quick brown fox jumps over the lazy dog."
# Generate the embedding vector
embedding_vector = embeddings_model.embed_query(text_to_embed)
print(f"Original Text: '{text_to_embed}'")
print(f"Embedding Vector (first 5 values): {embedding_vector[:5]}...")
print(f"Vector Dimension: {len(embedding_vector)}")Quick Check: Splitting & Embeddings
Consider the following statements about text splitting and embeddings:
Recap: Splitting & Embedding for RAG
You've learned two fundamental techniques for handling large documents in AI agents:
- Text Splitting: Breaking large texts into smaller, manageable chunks using tools like
RecursiveCharacterTextSplitter, controlled bychunk_sizeandchunk_overlap. - Embeddings: Converting text chunks into numerical vectors using embedding models (e.g.,
OpenAIEmbeddings) to enable semantic similarity search.
These techniques are crucial for building effective Retrieval Augmented Generation (RAG) systems, allowing your agents to intelligently find and use relevant information from vast knowledge bases.
よくある質問
「テキスト分割と埋め込み」レッスンは無料ですか？
はい。「テキスト分割と埋め込み」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、AI Agents with LangChain & Autonomous Workflowsコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 AI Agents with LangChain & Autonomous Workflowsコースには全4レッスンが含まれています。
「テキスト分割と埋め込み」で何を学びますか？
大きな文書を扱いやすいチャンクに分割し、セマンティック検索用の数値埋め込みを生成する手法を身につけます。 ブラウザで直接実行するハンズオンコードでAI Agents with LangChain & Autonomous Workflowsを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。
AI Agents with LangChain & Autonomous Workflowsを始めるのに経験は必要ですか？
事前経験は必要ありません。CoddyKitのAI Agents with LangChain & Autonomous Workflowsは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン2/4です。
「テキスト分割と埋め込み」レッスンにはどのくらい時間がかかりますか？
ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。
このAI Agents with LangChain & Autonomous Workflowsレッスンでコードを書いて実行できますか？
はい。すべてのAI Agents with LangChain & Autonomous Workflowsレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。
このコースのすべてのレッスン
- ドキュメントローダーを解説
- テキスト分割と埋め込み
- 検索のためのベクトルストア
- Retrieverとコンテキスト圧縮