Ventanas de contexto: tamaño e implicaciones

The context window is the maximum number of tokens an LLM can process in a single API call. It includes everything: the system prompt, all previous conversation turns, any documents you inject for RAG, and the space reserved for the model's response. If the total exceeds the context window, the API returns an error.

Think of the context window as the model's working memory. Unlike a human who can remember past conversations across sessions, an LLM has no persistent memory — it can only 'know' what is present in the current context window. When a conversation grows beyond the window, the oldest content must be removed, which can cause the model to lose track of important earlier context.

Context windows have grown dramatically. In 2020, GPT-3 offered 4,096 tokens. By 2025, leading models offer:

• GPT-4o and GPT-4o-mini: 128,000 tokens (~100,000 words)
• Claude 3.5 Sonnet / Opus: 200,000 tokens
• Gemini 1.5 Pro: 1,000,000 tokens (one million)
• Gemini 1.5 Flash: 1,000,000 tokens

A 128K context window can hold approximately 300 pages of text, a complete novel, or an entire medium-sized codebase. Despite this, infinite context is not a solved problem: attention cost grows quadratically with sequence length, making very long contexts expensive and sometimes less accurate than shorter focused contexts.

Research has found that LLMs do not attend equally to all parts of the context window. They tend to pay the most attention to content at the very beginning (primacy effect) and very end (recency effect) of the context, while content in the middle is processed less reliably.

This is called the lost in the middle problem. It has practical implications for RAG systems: if you concatenate 10 retrieved documents and the most relevant one ends up in the middle, the model may not use it effectively. Best practice is to place the most important context at the beginning or end of the injected documents, not in the middle.

In a chat application, the full conversation history is included in every API call. As a conversation grows, so does the token count. A conversation with 50 messages averaging 100 tokens each already uses 5,000 tokens just for history. Add a 2,000-token system prompt and 10,000 tokens of RAG context and you are at 17,000 tokens before the user even asks their next question.

Having a large context window does not mean you should fill it completely. Research consistently shows that model accuracy declines as context fills up, especially for tasks requiring precise retrieval of specific facts from a long context. A focused, relevant 5,000-token context often produces better answers than an unfocused 50,000-token context.

This is the core argument for RAG over simply dumping all your documents into the context: a RAG system retrieves only the 2-5 most relevant chunks, keeping the context focused and the model's attention concentrated on what matters. Think of it like searching a book's index vs reading the entire book to answer one question.

Long context windows enable powerful document processing workflows that were impossible before. You can now send an entire 50-page PDF to GPT-4o and ask questions about it, have the model summarize and cross-reference multiple contracts simultaneously, or analyze an entire codebase for patterns and anti-patterns.

However, at ~$0.15 per million input tokens, processing a 100,000-token document per query costs approximately $0.015 per query. At 10,000 queries per day over a document that rarely changes, you are paying $150/day for redundant processing. This is why caching and pre-processing strategies matter enormously in production document-analysis systems.

The max_tokens parameter in the API limits the output length, not the total context. The total context window equals input tokens plus output tokens. If your context window is 128,000 tokens and your input uses 120,000 tokens, you only have 8,000 tokens left for the response regardless of what you set for max_tokens.

Always reserve sufficient output budget. For a conversational assistant, reserving 2,000-4,000 tokens for output is usually enough. For code generation or long-form content, you may need 8,000-16,000 tokens. Build your token budget calculation into your context assembly logic.

Context window size should be one of your key criteria when choosing a model. Match the model's context window to your actual use case:

• Chat assistants with short sessions: 8K-16K is usually sufficient; use gpt-4o-mini for cost efficiency
• Document Q&A over medium documents: 32K-128K; gpt-4o balances quality and cost well
• Legal/contract analysis with hundreds of pages: 128K-200K; consider Claude for its long-context performance
• Full codebase or book analysis: 500K-1M; Gemini 1.5 Pro is currently the leader

Paying for a 1M token context window when you only need 8K is expensive overkill. Right-size your model to your actual context requirements.

When you repeatedly query the same large document or system prompt across many requests, you are paying to tokenize and process the same content every time. OpenAI's prompt caching automatically discounts repeated prompt prefixes at 50% off the input token price when the same prefix exceeds 1,024 tokens.

To maximize cache hits, structure your messages so the stable content comes first: system prompt, then the large document or context, then the varying user question. This way the long stable prefix is cached and only the small varying query is processed at full price on each request.

Given a large context window, you have two strategies for handling growing conversations or large documents: expand (keep everything in context) or summarize (compress old content to save tokens). The right choice depends on your use case.

Choose expansion when: you need to reference specific facts from earlier in the conversation, you are analyzing a document that requires quoting specific sections, or summarization would lose critical nuance. Choose summarization when: the broad themes of earlier conversation matter more than specific wording, you are approaching the context limit, or the same context will be reused many times (making summarization a one-time cost).

In production, track context length per request as a key metric. Sudden spikes in average context length can indicate a bug in your context assembly code, users pasting very long inputs, or a feedback loop where the model's long responses are being fed back into the context. Set alerts when context length exceeds 80% of the model's maximum.

Also track truncation events — when you have to cut context to fit within the window. Frequent truncation means you need a better context management strategy, a larger-context model, or a RAG-based approach to retrieve only relevant content rather than sending everything.

Test your understanding of AI Engineering concepts from this lesson.

In this lesson you learned: the context window is the total token budget for input plus output in a single API call, the lost in the middle problem means content at the start and end of context is processed more reliably, and a focused small context often outperforms a large unfocused one, making RAG preferable to stuffing all documents in. Next up we explore how to calculate and predict API costs before sending requests.