AI Engineering Academy · Lesson

Context Windows: Size and Implications

Learn what the context window is, how it constrains conversation length and document processing, and compare context sizes across GPT-4o, Claude, and Gemini.

Context Windows: Size and Implications is a free AI Engineering Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is the Context Window?

The context window is the maximum number of tokens an LLM can process in a single API call. It includes everything: the system prompt, all previous conversation turns, any documents you inject for RAG, and the space reserved for the model's response. If the total exceeds the context window, the API returns an error.

Think of the context window as the model's working memory. Unlike a human who can remember past conversations across sessions, an LLM has no persistent memory — it can only 'know' what is present in the current context window. When a conversation grows beyond the window, the oldest content must be removed, which can cause the model to lose track of important earlier context.

Context Window Sizes in 2025

Context windows have grown dramatically. In 2020, GPT-3 offered 4,096 tokens. By 2025, leading models offer:

GPT-4o and GPT-4o-mini: 128,000 tokens (~100,000 words)
Claude 3.5 Sonnet / Opus: 200,000 tokens
Gemini 1.5 Pro: 1,000,000 tokens (one million)
Gemini 1.5 Flash: 1,000,000 tokens

A 128K context window can hold approximately 300 pages of text, a complete novel, or an entire medium-sized codebase. Despite this, infinite context is not a solved problem: attention cost grows quadratically with sequence length, making very long contexts expensive and sometimes less accurate than shorter focused contexts.

The Lost in the Middle Problem

Research has found that LLMs do not attend equally to all parts of the context window. They tend to pay the most attention to content at the very beginning (primacy effect) and very end (recency effect) of the context, while content in the middle is processed less reliably.

This is called the lost in the middle problem. It has practical implications for RAG systems: if you concatenate 10 retrieved documents and the most relevant one ends up in the middle, the model may not use it effectively. Best practice is to place the most important context at the beginning or end of the injected documents, not in the middle.

Context vs Conversation: A Practical Example

In a chat application, the full conversation history is included in every API call. As a conversation grows, so does the token count. A conversation with 50 messages averaging 100 tokens each already uses 5,000 tokens just for history. Add a 2,000-token system prompt and 10,000 tokens of RAG context and you are at 17,000 tokens before the user even asks their next question.

import tiktoken

def estimate_conversation_tokens(messages, model='gpt-4o'):
    enc = tiktoken.encoding_for_model(model)
    total = 3  # priming
    for msg in messages:
        total += 4  # per-message overhead
        total += len(enc.encode(msg.get('content', '')))
    return total

# Simulate a growing conversation
conversation = [
    {'role': 'system', 'content': 'You are a helpful coding assistant. ' * 20},  # ~100 tokens
]

for i in range(1, 21):
    conversation.append({'role': 'user', 'content': f'Question {i}: How do I implement feature X?'})
    conversation.append({'role': 'assistant', 'content': 'Here is how to implement that feature...' * 5})
    if i % 5 == 0:
        tokens = estimate_conversation_tokens(conversation)
        print(f'After {i} exchanges: {tokens} tokens')

Effective Context vs Maximum Context

Having a large context window does not mean you should fill it completely. Research consistently shows that model accuracy declines as context fills up, especially for tasks requiring precise retrieval of specific facts from a long context. A focused, relevant 5,000-token context often produces better answers than an unfocused 50,000-token context.

This is the core argument for RAG over simply dumping all your documents into the context: a RAG system retrieves only the 2-5 most relevant chunks, keeping the context focused and the model's attention concentrated on what matters. Think of it like searching a book's index vs reading the entire book to answer one question.

Implications for Document Processing

Long context windows enable powerful document processing workflows that were impossible before. You can now send an entire 50-page PDF to GPT-4o and ask questions about it, have the model summarize and cross-reference multiple contracts simultaneously, or analyze an entire codebase for patterns and anti-patterns.

However, at ~$0.15 per million input tokens, processing a 100,000-token document per query costs approximately $0.015 per query. At 10,000 queries per day over a document that rarely changes, you are paying $150/day for redundant processing. This is why caching and pre-processing strategies matter enormously in production document-analysis systems.

Context Window and Max Tokens Relationship

The max_tokens parameter in the API limits the output length, not the total context. The total context window equals input tokens plus output tokens. If your context window is 128,000 tokens and your input uses 120,000 tokens, you only have 8,000 tokens left for the response regardless of what you set for max_tokens.

Always reserve sufficient output budget. For a conversational assistant, reserving 2,000-4,000 tokens for output is usually enough. For code generation or long-form content, you may need 8,000-16,000 tokens. Build your token budget calculation into your context assembly logic.

import tiktoken

def check_context_budget(
    messages,
    model='gpt-4o',
    max_context=128000,
    min_output_tokens=2000
):
    enc = tiktoken.encoding_for_model(model)
    input_tokens = sum(
        len(enc.encode(m.get('content', ''))) + 4
        for m in messages
    ) + 3

    available_output = max_context - input_tokens
    if available_output < min_output_tokens:
        raise ValueError(
            f'Not enough output budget: only {available_output} tokens '
            f'remaining, need at least {min_output_tokens}.'
        )
    return input_tokens, available_output

Choosing Models by Context Requirements

Context window size should be one of your key criteria when choosing a model. Match the model's context window to your actual use case:

Chat assistants with short sessions: 8K-16K is usually sufficient; use gpt-4o-mini for cost efficiency
Document Q&A over medium documents: 32K-128K; gpt-4o balances quality and cost well
Legal/contract analysis with hundreds of pages: 128K-200K; consider Claude for its long-context performance
Full codebase or book analysis: 500K-1M; Gemini 1.5 Pro is currently the leader

Paying for a 1M token context window when you only need 8K is expensive overkill. Right-size your model to your actual context requirements.

Context Caching to Reduce Costs

When you repeatedly query the same large document or system prompt across many requests, you are paying to tokenize and process the same content every time. OpenAI's prompt caching automatically discounts repeated prompt prefixes at 50% off the input token price when the same prefix exceeds 1,024 tokens.

To maximize cache hits, structure your messages so the stable content comes first: system prompt, then the large document or context, then the varying user question. This way the long stable prefix is cached and only the small varying query is processed at full price on each request.

When to Expand vs When to Summarize

Given a large context window, you have two strategies for handling growing conversations or large documents: expand (keep everything in context) or summarize (compress old content to save tokens). The right choice depends on your use case.

Choose expansion when: you need to reference specific facts from earlier in the conversation, you are analyzing a document that requires quoting specific sections, or summarization would lose critical nuance. Choose summarization when: the broad themes of earlier conversation matter more than specific wording, you are approaching the context limit, or the same context will be reused many times (making summarization a one-time cost).

Monitoring Context Length in Production

In production, track context length per request as a key metric. Sudden spikes in average context length can indicate a bug in your context assembly code, users pasting very long inputs, or a feedback loop where the model's long responses are being fed back into the context. Set alerts when context length exceeds 80% of the model's maximum.

Also track truncation events — when you have to cut context to fit within the window. Frequent truncation means you need a better context management strategy, a larger-context model, or a RAG-based approach to retrieve only relevant content rather than sending everything.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: the context window is the total token budget for input plus output in a single API call, the lost in the middle problem means content at the start and end of context is processed more reliably, and a focused small context often outperforms a large unfocused one, making RAG preferable to stuffing all documents in. Next up we explore how to calculate and predict API costs before sending requests.

Frequently asked questions

Is the “Context Windows: Size and Implications” lesson free?

Yes — the full text of “Context Windows: Size and Implications” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Context Windows: Size and Implications”?

Learn what the context window is, how it constrains conversation length and document processing, and compare context sizes across GPT-4o, Claude, and Gemini. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Context Windows: Size and Implications” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Engineering Academy