AI Engineering Academy · 강의

컨텍스트 윈도우: 크기와 의미

컨텍스트 윈도우가 무엇인지, 대화 길이와 문서 처리에 어떤 제약을 주는지 배우고, GPT-4o, Claude, Gemini의 컨텍스트 크기를 비교합니다.

컨텍스트 윈도우: 크기와 의미은(는) CoddyKit의 무료 AI Engineering Academy 강의입니다. 이것은 4개 중 2번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 AI Engineering Academy 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. AI Engineering Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

What Is the Context Window?

The context window is the maximum number of tokens an LLM can process in a single API call. It includes everything: the system prompt, all previous conversation turns, any documents you inject for RAG, and the space reserved for the model's response. If the total exceeds the context window, the API returns an error.

Think of the context window as the model's working memory. Unlike a human who can remember past conversations across sessions, an LLM has no persistent memory — it can only 'know' what is present in the current context window. When a conversation grows beyond the window, the oldest content must be removed, which can cause the model to lose track of important earlier context.

Context Window Sizes in 2025

Context windows have grown dramatically. In 2020, GPT-3 offered 4,096 tokens. By 2025, leading models offer:

GPT-4o and GPT-4o-mini: 128,000 tokens (~100,000 words)
Claude 3.5 Sonnet / Opus: 200,000 tokens
Gemini 1.5 Pro: 1,000,000 tokens (one million)
Gemini 1.5 Flash: 1,000,000 tokens

A 128K context window can hold approximately 300 pages of text, a complete novel, or an entire medium-sized codebase. Despite this, infinite context is not a solved problem: attention cost grows quadratically with sequence length, making very long contexts expensive and sometimes less accurate than shorter focused contexts.

The Lost in the Middle Problem

Research has found that LLMs do not attend equally to all parts of the context window. They tend to pay the most attention to content at the very beginning (primacy effect) and very end (recency effect) of the context, while content in the middle is processed less reliably.

This is called the lost in the middle problem. It has practical implications for RAG systems: if you concatenate 10 retrieved documents and the most relevant one ends up in the middle, the model may not use it effectively. Best practice is to place the most important context at the beginning or end of the injected documents, not in the middle.

Context vs Conversation: A Practical Example

In a chat application, the full conversation history is included in every API call. As a conversation grows, so does the token count. A conversation with 50 messages averaging 100 tokens each already uses 5,000 tokens just for history. Add a 2,000-token system prompt and 10,000 tokens of RAG context and you are at 17,000 tokens before the user even asks their next question.

import tiktoken

def estimate_conversation_tokens(messages, model='gpt-4o'):
    enc = tiktoken.encoding_for_model(model)
    total = 3  # priming
    for msg in messages:
        total += 4  # per-message overhead
        total += len(enc.encode(msg.get('content', '')))
    return total

# Simulate a growing conversation
conversation = [
    {'role': 'system', 'content': 'You are a helpful coding assistant. ' * 20},  # ~100 tokens
]

for i in range(1, 21):
    conversation.append({'role': 'user', 'content': f'Question {i}: How do I implement feature X?'})
    conversation.append({'role': 'assistant', 'content': 'Here is how to implement that feature...' * 5})
    if i % 5 == 0:
        tokens = estimate_conversation_tokens(conversation)
        print(f'After {i} exchanges: {tokens} tokens')

Effective Context vs Maximum Context

Having a large context window does not mean you should fill it completely. Research consistently shows that model accuracy declines as context fills up, especially for tasks requiring precise retrieval of specific facts from a long context. A focused, relevant 5,000-token context often produces better answers than an unfocused 50,000-token context.

This is the core argument for RAG over simply dumping all your documents into the context: a RAG system retrieves only the 2-5 most relevant chunks, keeping the context focused and the model's attention concentrated on what matters. Think of it like searching a book's index vs reading the entire book to answer one question.

Implications for Document Processing

Long context windows enable powerful document processing workflows that were impossible before. You can now send an entire 50-page PDF to GPT-4o and ask questions about it, have the model summarize and cross-reference multiple contracts simultaneously, or analyze an entire codebase for patterns and anti-patterns.

However, at ~$0.15 per million input tokens, processing a 100,000-token document per query costs approximately $0.015 per query. At 10,000 queries per day over a document that rarely changes, you are paying $150/day for redundant processing. This is why caching and pre-processing strategies matter enormously in production document-analysis systems.

Context Window and Max Tokens Relationship

The max_tokens parameter in the API limits the output length, not the total context. The total context window equals input tokens plus output tokens. If your context window is 128,000 tokens and your input uses 120,000 tokens, you only have 8,000 tokens left for the response regardless of what you set for max_tokens.

Always reserve sufficient output budget. For a conversational assistant, reserving 2,000-4,000 tokens for output is usually enough. For code generation or long-form content, you may need 8,000-16,000 tokens. Build your token budget calculation into your context assembly logic.

import tiktoken

def check_context_budget(
    messages,
    model='gpt-4o',
    max_context=128000,
    min_output_tokens=2000
):
    enc = tiktoken.encoding_for_model(model)
    input_tokens = sum(
        len(enc.encode(m.get('content', ''))) + 4
        for m in messages
    ) + 3

    available_output = max_context - input_tokens
    if available_output < min_output_tokens:
        raise ValueError(
            f'Not enough output budget: only {available_output} tokens '
            f'remaining, need at least {min_output_tokens}.'
        )
    return input_tokens, available_output

Choosing Models by Context Requirements

Context window size should be one of your key criteria when choosing a model. Match the model's context window to your actual use case:

Chat assistants with short sessions: 8K-16K is usually sufficient; use gpt-4o-mini for cost efficiency
Document Q&A over medium documents: 32K-128K; gpt-4o balances quality and cost well
Legal/contract analysis with hundreds of pages: 128K-200K; consider Claude for its long-context performance
Full codebase or book analysis: 500K-1M; Gemini 1.5 Pro is currently the leader

Paying for a 1M token context window when you only need 8K is expensive overkill. Right-size your model to your actual context requirements.

Context Caching to Reduce Costs

When you repeatedly query the same large document or system prompt across many requests, you are paying to tokenize and process the same content every time. OpenAI's prompt caching automatically discounts repeated prompt prefixes at 50% off the input token price when the same prefix exceeds 1,024 tokens.

To maximize cache hits, structure your messages so the stable content comes first: system prompt, then the large document or context, then the varying user question. This way the long stable prefix is cached and only the small varying query is processed at full price on each request.

When to Expand vs When to Summarize

Given a large context window, you have two strategies for handling growing conversations or large documents: expand (keep everything in context) or summarize (compress old content to save tokens). The right choice depends on your use case.

Choose expansion when: you need to reference specific facts from earlier in the conversation, you are analyzing a document that requires quoting specific sections, or summarization would lose critical nuance. Choose summarization when: the broad themes of earlier conversation matter more than specific wording, you are approaching the context limit, or the same context will be reused many times (making summarization a one-time cost).

Monitoring Context Length in Production

In production, track context length per request as a key metric. Sudden spikes in average context length can indicate a bug in your context assembly code, users pasting very long inputs, or a feedback loop where the model's long responses are being fed back into the context. Set alerts when context length exceeds 80% of the model's maximum.

Also track truncation events — when you have to cut context to fit within the window. Frequent truncation means you need a better context management strategy, a larger-context model, or a RAG-based approach to retrieve only relevant content rather than sending everything.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: the context window is the total token budget for input plus output in a single API call, the lost in the middle problem means content at the start and end of context is processed more reliably, and a focused small context often outperforms a large unfocused one, making RAG preferable to stuffing all documents in. Next up we explore how to calculate and predict API costs before sending requests.

자주 묻는 질문

“컨텍스트 윈도우: 크기와 의미” 강의는 무료인가요?

네 — “컨텍스트 윈도우: 크기와 의미” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 AI Engineering Academy 강의 전체를 잠금 해제할 수 있습니다. AI Engineering Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

“컨텍스트 윈도우: 크기와 의미”에서 뭘 배우나요?

컨텍스트 윈도우가 무엇인지, 대화 길이와 문서 처리에 어떤 제약을 주는지 배우고, GPT-4o, Claude, Gemini의 컨텍스트 크기를 비교합니다. 브라우저에서 직접 실행하는 실습 코드로 AI Engineering Academy을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

AI Engineering Academy을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 AI Engineering Academy은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 2번째 강의입니다.

“컨텍스트 윈도우: 크기와 의미” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 AI Engineering Academy 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 AI Engineering Academy 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← AI Engineering Academy(으)로 돌아가기