Document Loading, Splitting, and Embedding
PyPDFLoader, RecursiveCharacterTextSplitter, OpenAIEmbeddings, embedding strategies.
Document Loading, Splitting, and Embedding is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The RAG Ingestion Pipeline
Before an LLM can answer from your documents, you must load, split, and embed them. This lesson covers that ingestion pipeline, the foundation of every Retrieval-Augmented Generation system.
pip install langchain-community pypdf langchain-openaiLoading a PDF
PyPDFLoader reads a PDF and returns a list of Document objects, one per page. Each Document has page_content (the text) and metadata (source, page number).
from langchain_community.document_loaders import PyPDFLoader
loader = PyPDFLoader("handbook.pdf")
docs = loader.load()
print(len(docs), "pages")Why Split Documents?
Whole pages are often too large to embed or fit in a prompt. Splitting breaks text into smaller chunks that embed cleanly and let retrieval return only the relevant passage, not an entire page.
RecursiveCharacterTextSplitter
The recommended splitter is RecursiveCharacterTextSplitter. It tries to split on paragraphs first, then sentences, then words, keeping chunks semantically coherent rather than cutting mid-word.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)chunk_size and chunk_overlap
chunk_size sets the max characters per chunk; chunk_overlap repeats some text between neighbors so context is not lost at boundaries. A 1000/200 split is a common starting point.
chunks = splitter.split_documents(docs)
print(len(chunks), "chunks")
print(chunks[0].page_content[:120])Tuning the Chunk Size
Small chunks give precise retrieval but may miss surrounding context. Large chunks keep context but dilute relevance and cost more tokens. Overlap of 10-20% of chunk size is a good default to bridge boundaries.
What Are Embeddings?
An embedding turns text into a fixed-length vector of numbers that captures meaning. Similar texts produce nearby vectors. This is what lets us search by semantic similarity instead of keywords.
OpenAIEmbeddings
Create embeddings with OpenAIEmbeddings. The model text-embedding-3-small is cheap and effective. embed_query embeds one string; embed_documents embeds a list.
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector = embeddings.embed_query("How do I reset my password?")
print(len(vector)) # 1536Embedding a Batch
Embed all your chunks at once. Each chunk becomes a vector you will later store in a vector database for fast similarity search.
texts = [c.page_content for c in chunks]
vectors = embeddings.embed_documents(texts)
print(len(vectors), "vectors of dim", len(vectors[0]))Estimating Embedding Cost
Embeddings are billed per token. Estimate total tokens across all chunks, then multiply by the price per 1K tokens. Counting before embedding avoids surprise bills on large corpora.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
total = sum(len(enc.encode(c.page_content)) for c in chunks)
price_per_1k = 0.00002
print("tokens:", total, "cost $:", total / 1000 * price_per_1k)Pipeline Summary
The ingestion flow: load documents, split into overlapping chunks, embed each chunk into a vector, and (next lesson) store them. Good chunking and cost-awareness here determine the quality and price of the whole RAG system.
Quick Check
Test your ingestion knowledge.
Recap: Loading, Splitting, Embedding
You used PyPDFLoader to load documents, RecursiveCharacterTextSplitter with chunk_size and chunk_overlap to chunk them, and OpenAIEmbeddings to turn chunks into vectors. You also learned to estimate embedding cost per token before processing a large corpus.
Frequently asked questions
Is the “Document Loading, Splitting, and Embedding” lesson free?
Yes — the full text of “Document Loading, Splitting, and Embedding” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Document Loading, Splitting, and Embedding”?
PyPDFLoader, RecursiveCharacterTextSplitter, OpenAIEmbeddings, embedding strategies. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Document Loading, Splitting, and Embedding” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- LangChain Architecture and LCEL
- Document Loading, Splitting, and Embedding
- Vector Stores: Chroma and FAISS
- Building a RAG Q&A System End-to-End