为更好的嵌入拆分文本
学习如何将文档拆分为适合生成嵌入的文本块，理解文本块大小和重叠范围为何重要，并掌握最大化检索质量的策略。
为更好的嵌入拆分文本 是 CoddyKit 上的免费 Vector Databases: Pinecone, Weaviate & pgvector 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Vector Databases: Pinecone, Weaviate & pgvector 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Vector Databases: Pinecone, Weaviate & pgvector 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
Why Chunking Matters
Embedding models have a token limit and produce one vector per input. Feeding a whole document yields a vague, averaged vector. Chunking splits text into focused pieces so each vector captures a specific idea.
The Goldilocks Problem
Chunk size is a balance:
- Too large — diluted meaning, mixed topics in one vector
- Too small — fragments lose context, more vectors to store
Aim for chunks that hold one coherent thought.
Fixed-Size Chunking
The simplest method: split every N characters or tokens.
def chunk(text, size):
return [text[i:i+size] for i in range(0, len(text), size)]
print(chunk('abcdefghij', 4))The Overlap Trick
Fixed splits can cut a sentence in half, losing context at the boundary. Adding overlap repeats the last few tokens of one chunk at the start of the next so ideas spanning a boundary survive.
def chunk_overlap(text, size, overlap):
out = []
i = 0
while i < len(text):
out.append(text[i:i+size])
i += size - overlap
return out
print(chunk_overlap('abcdefghij', 4, 1))Sentence-Aware Chunking
Better than blind character splits: break on sentence boundaries, then group sentences up to a target size. Chunks end cleanly and read coherently.
Recursive Chunking
Recursive splitting tries large separators first (paragraphs), then smaller ones (sentences, then words) until chunks fit the size limit. It respects document structure while guaranteeing size.
Structure-Aware Chunking
For Markdown, code, or HTML, split along structural elements:
- Markdown headers and sections
- Code functions or classes
- HTML sections and lists
This keeps related content together.
Token Counting
Models limit by tokens, not characters. Estimate tokens before embedding so chunks fit the model window.
def approx_tokens(text):
return max(1, len(text) // 4)
print(approx_tokens('The quick brown fox jumps'))Matching Chunks to Queries
Think about how users query. If questions target short facts, smaller chunks improve precision. If questions need broad context, larger chunks help. Sometimes you store both granularities.
Adding Context to Chunks
A chunk taken out of context can be ambiguous. Prepend a short header (document title, section name) to each chunk before embedding so the vector knows where it came from.
Iterate and Measure
There is no universal best chunk size. Pick a starting point (often a few hundred tokens with modest overlap), then measure retrieval quality and adjust. Chunking is an experiment, not a fixed rule.
Quick Check
Test your understanding of chunk overlap.
Recap
You learned that chunking turns documents into focused, embeddable pieces. Balance chunk size, add overlap to preserve boundary context, prefer sentence-aware or recursive splitting, count tokens, enrich chunks with context headers, and iterate while measuring retrieval quality.
常见问题解答
「为更好的嵌入拆分文本」课时是免费的吗？
是的 — 「为更好的嵌入拆分文本」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Vector Databases: Pinecone, Weaviate & pgvector 课程的其余内容，请升级到 CoddyKit PRO。 Vector Databases: Pinecone, Weaviate & pgvector 课程共包含 4 节课。
「为更好的嵌入拆分文本」这节课中我会学到什么？
学习如何将文档拆分为适合生成嵌入的文本块，理解文本块大小和重叠范围为何重要，并掌握最大化检索质量的策略。 你通过在浏览器中直接运行的动手代码来练习 Vector Databases: Pinecone, Weaviate & pgvector，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Vector Databases: Pinecone, Weaviate & pgvector 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Vector Databases: Pinecone, Weaviate & pgvector 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「为更好的嵌入拆分文本」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Vector Databases: Pinecone, Weaviate & pgvector 课中编写并运行代码吗？
能。每节 Vector Databases: Pinecone, Weaviate & pgvector 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。
此课程中的所有课时
- 文本嵌入模型
- 使用嵌入应用程序接口
- 存储与更新嵌入
- 为更好的嵌入拆分文本