AI Prompt Engineering · Lesson

Chunking Strategies for Long Texts

Fixed-size, sentence-boundary, and semantic chunking approaches.

Chunking Strategies for Long Texts is a free AI Prompt Engineering lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Long Documents Are a Challenge

LLMs have a context window — a maximum number of tokens they can process at once. GPT-4 supports 128k tokens, Claude supports 200k, but many documents exceed even these limits. More importantly, research shows model accuracy often degrades on very long contexts. Chunking is the practice of splitting documents into smaller pieces before processing.

Fixed-Size Chunking

Fixed-size chunking splits text every N tokens (or characters) regardless of content boundaries. It is the simplest strategy and requires no semantic understanding.

Typical chunk size: 500–1000 tokens. Chunks are easy to index and retrieve but may split sentences mid-way, losing meaning at boundaries.

import tiktoken

def fixed_chunk(text, max_tokens=1000):
    enc = tiktoken.get_encoding('cl100k_base')
    tokens = enc.encode(text)
    chunks = []
    for i in range(0, len(tokens), max_tokens):
        chunk_tokens = tokens[i:i + max_tokens]
        chunks.append(enc.decode(chunk_tokens))
    return chunks

chunks = fixed_chunk(long_document)
print(f'Total chunks: {len(chunks)}')

Trade-offs of Fixed-Size Chunking

Advantages:

Simple to implement — no NLP needed
Predictable chunk sizes — easier to manage API costs
Fast processing

Disadvantages:

Cuts across sentence and paragraph boundaries
A sentence split across chunks loses context for retrieval
Poor for summarization — chunk may end mid-argument

Sentence-Boundary Chunking

Sentence-boundary chunking splits text at natural sentence or paragraph breaks. Each chunk ends at a full stop, ensuring no sentence is cut in half. This produces more readable, coherent chunks.

Use a sentence tokenizer (spaCy or NLTK) to detect boundaries, then group sentences until the chunk reaches the target size.

import spacy

nlp = spacy.load('en_core_web_sm')

def sentence_chunk(text, max_tokens=1000):
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents]
    chunks, current, count = [], [], 0
    for sent in sentences:
        word_count = len(sent.split())
        if count + word_count > max_tokens and current:
            chunks.append(' '.join(current))
            current, count = [], 0
        current.append(sent)
        count += word_count
    if current:
        chunks.append(' '.join(current))
    return chunks

Semantic Chunking

Semantic chunking goes further — it splits text when the topic shifts. By embedding adjacent sentences and measuring cosine similarity, we can detect when the discussion moves to a new subject.

When similarity drops below a threshold, insert a chunk boundary. This produces chunks that are topically cohesive, ideal for retrieval-augmented generation (RAG).

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

def semantic_chunk(sentences, threshold=0.75):
    embeddings = model.encode(sentences)
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(
            [embeddings[i-1]], [embeddings[i]]
        )[0][0]
        if sim < threshold:
            chunks.append(' '.join(current))
            current = []
        current.append(sentences[i])
    if current:
        chunks.append(' '.join(current))
    return chunks

Comparing the Three Strategies

Here is a side-by-side comparison to help you choose:

Fixed-size: fastest, least accurate boundaries — use for simple pipelines
Sentence-boundary: preserves sentence integrity — use when coherence matters
Semantic: best topical cohesion — use for RAG where precision retrieval is critical

In practice, semantic chunking adds latency due to embedding computation. Choose based on retrieval accuracy requirements.

Chunking for Retrieval Tasks

For retrieval-augmented generation (RAG), chunks become the unit of search. A user query is embedded, and the most similar chunks are retrieved and injected into the prompt.

Smaller chunks (200–400 tokens) improve retrieval precision — each chunk contains one focused idea. Larger chunks (800–1000 tokens) provide more context but may include irrelevant content alongside the relevant passage.

import openai
import numpy as np

client = openai.OpenAI(api_key='sk-...')

def embed(text):
    resp = client.embeddings.create(
        model='text-embedding-3-small',
        input=text
    )
    return np.array(resp.data[0].embedding)

def retrieve(query, chunks, top_k=3):
    q_emb = embed(query)
    scores = [(i, np.dot(q_emb, embed(c))) for i, c in enumerate(chunks)]
    scores.sort(key=lambda x: x[1], reverse=True)
    return [chunks[i] for i, _ in scores[:top_k]]

Chunking for Summarization Tasks

For summarization, chunks should be large enough to contain a complete argument or section. Sentence-boundary chunks of 600–800 tokens work well.

Each chunk is summarized independently (the map step), then the summaries are combined into a final summary (the reduce step). This is the classic map-reduce pattern, covered in the next lesson.

Overlap: Bridging Chunk Boundaries

One practical technique to reduce boundary errors: add overlap between chunks. The last 100–200 tokens of chunk N are repeated at the start of chunk N+1.

Overlap ensures that a sentence spanning a boundary appears fully in at least one chunk. This is especially important for retrieval — a query about a topic that straddles a boundary will match the overlapping chunk.

def fixed_chunk_with_overlap(text, max_tokens=1000, overlap=200):
    enc = tiktoken.get_encoding('cl100k_base')
    tokens = enc.encode(text)
    chunks = []
    step = max_tokens - overlap
    for i in range(0, len(tokens), step):
        chunk_tokens = tokens[i:i + max_tokens]
        chunks.append(enc.decode(chunk_tokens))
        if i + max_tokens >= len(tokens):
            break
    return chunks

Metadata Enrichment per Chunk

Each chunk should carry metadata so downstream steps can reference its origin:

source: filename or URL
page: page number
chunk_index: position in document
section: chapter or heading

This metadata is stored alongside the embedding in a vector DB (Pinecone, Chroma, Weaviate) and returned with retrieved chunks so the LLM can cite sources.

def chunk_with_metadata(text, source='doc.pdf', page=1):
    chunks = fixed_chunk_with_overlap(text)
    return [
        {
            'text': chunk,
            'metadata': {
                'source': source,
                'page': page,
                'chunk_index': i
            }
        }
        for i, chunk in enumerate(chunks)
    ]

Choosing the Right Strategy

A quick decision guide:

Speed is priority, content is prose: sentence-boundary chunking
Retrieval accuracy is critical: semantic chunking with small chunks (300 tokens)
Summarizing a book or report: sentence-boundary, larger chunks (800 tokens), map-reduce
Legal / technical documents: semantic chunking to keep clauses together

Always measure retrieval recall on a representative query set before committing to a strategy.

Knowledge Check

Which chunking strategy splits text when cosine similarity between adjacent sentence embeddings drops below a threshold?

Recap: Chunking Strategies

You have learned three core chunking strategies:

Fixed-size: split every N tokens — fast, simple, boundary-unaware
Sentence-boundary: split at natural sentence breaks — coherent, moderate complexity
Semantic: split on topic shifts via embedding similarity — most accurate, highest cost

Add overlap between chunks to reduce boundary errors, and always attach metadata (source, page, index) to enable citation and traceability. Next lesson covers the map-reduce summarization pattern.

Frequently asked questions

Is the “Chunking Strategies for Long Texts” lesson free?

Yes — the full text of “Chunking Strategies for Long Texts” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Chunking Strategies for Long Texts”?

Fixed-size, sentence-boundary, and semantic chunking approaches. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Chunking Strategies for Long Texts” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering