0Pricing
AI Engineering Academy · レッスン

Context内に収めるための戦略

sliding windowメモリ、メッセージの要約、選択的なコンテキスト削減を実装し、token制限を超えずに長い会話を処理します。

「Context内に収めるための戦略」はCoddyKit上の無料AI Engineering Academyレッスンです。 これはレッスン4/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはAI Engineering Academy学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 AI Engineering Academyコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

The Growing Conversation Problem

Every message exchanged in a chat conversation grows the token count of the next API call. In a long customer support session or a multi-hour coding session, the accumulated conversation history can easily exceed 20,000-50,000 tokens — all of which you must send to the API every single turn, paying for previously processed tokens repeatedly.

Without a context management strategy, your application will either hit the context limit and crash, or you will end up silently truncating messages and the model will lose track of important earlier context. Every production LLM application needs an explicit strategy for managing context growth.

Strategy 1: Sliding Window (Last N Messages)

The simplest strategy is to keep only the last N messages in the context, discarding older ones. This is called a sliding window. It is easy to implement, predictable in cost, and sufficient for many use cases where recent context matters more than older exchanges.

import openai

client = openai.OpenAI()

class SlidingWindowConversation:
    def __init__(self, system_prompt, window_size=10):
        self.system = system_prompt
        self.window_size = window_size
        self.history = []

    def chat(self, user_message):
        self.history.append({'role': 'user', 'content': user_message})

        # Keep only the last N messages
        windowed = self.history[-self.window_size:]
        messages = [{'role': 'system', 'content': self.system}] + windowed

        response = client.chat.completions.create(
            model='gpt-4o-mini',
            messages=messages
        )
        reply = response.choices[0].message.content
        self.history.append({'role': 'assistant', 'content': reply})
        print(f'Context: {len(windowed)} messages ({len(self.history)} total)')
        return reply

conv = SlidingWindowConversation('You are a helpful assistant.', window_size=6)
print(conv.chat('Hello! My name is Alice.'))
print(conv.chat('What is the capital of France?'))
print(conv.chat('What is my name?'))  # Might forget if window is small

Strategy 2: Token-Aware Trimming

Rather than trimming by message count, token-aware trimming removes messages until the total token count is below a threshold. This is more precise because messages vary widely in length — a message count limit might include 10 short messages or 3 very long ones, with very different token costs.

import tiktoken

def trim_to_token_budget(
    messages, system_prompt, model='gpt-4o-mini', max_input_tokens=8000
):
    enc = tiktoken.encoding_for_model(model)

    def count(msgs):
        return sum(len(enc.encode(m.get('content',''))) + 4 for m in msgs) + 3

    # System prompt token count
    system_tokens = len(enc.encode(system_prompt)) + 4
    budget = max_input_tokens - system_tokens

    # Trim from the oldest messages until we fit
    trimmed = list(messages)
    while trimmed and count(trimmed) > budget:
        trimmed.pop(0)  # Remove oldest message

    removed = len(messages) - len(trimmed)
    if removed:
        print(f'Trimmed {removed} old messages to stay within {max_input_tokens} tokens')
    return trimmed

Strategy 3: Conversation Summarization

Summarization compresses old conversation turns into a concise summary while preserving the key facts, decisions, and context that the user mentioned. This is more sophisticated than sliding windows because it retains the essence of past exchanges rather than simply discarding them.

The pattern works as follows: when the conversation exceeds a threshold, send the oldest N turns to the model with a 'Summarize this conversation so far' prompt, replace those turns with a single system message containing the summary, and continue the conversation with the new compressed history.

import openai

client = openai.OpenAI()

def summarize_conversation(messages_to_summarize, model='gpt-4o-mini'):
    summary_prompt = [
        {'role': 'system', 'content': 'You summarize conversations. Be concise but preserve key facts, decisions, and any specific information the user shared (names, numbers, preferences).'},
        {'role': 'user', 'content': 'Summarize this conversation:\n' + '\n'.join(
            f"{m['role'].upper()}: {m['content']}" for m in messages_to_summarize
        )}
    ]
    resp = client.chat.completions.create(model=model, messages=summary_prompt, max_tokens=500)
    return resp.choices[0].message.content

def compress_history(history, keep_recent=4):
    if len(history) <= keep_recent:
        return history
    to_summarize = history[:-keep_recent]
    summary = summarize_conversation(to_summarize)
    summary_msg = {'role': 'system', 'content': f'Earlier conversation summary: {summary}'}
    return [summary_msg] + history[-keep_recent:]

print('Summarization strategy compresses old turns into a single summary message')

Strategy 4: Entity Memory

Entity memory extracts and maintains a structured representation of key facts mentioned in the conversation — user preferences, names, project details, decisions made — rather than storing raw message history. Before each turn, the stored facts are injected into the system prompt.

This is more token-efficient than full history because you only include what is semantically important, not every word of every turn. Entity memory works especially well for long-running assistants where users refer back to preferences and facts established much earlier in the conversation.

import openai
import json

client = openai.OpenAI()

def extract_entities(messages, model='gpt-4o-mini'):
    prompt = [
        {'role': 'system', 'content': 'Extract key facts from this conversation as JSON: {"user_name": null, "preferences": [], "key_facts": []}'},
        {'role': 'user', 'content': '\n'.join(f"{m['role']}: {m['content']}" for m in messages)}
    ]
    resp = client.chat.completions.create(
        model=model,
        messages=prompt,
        response_format={'type': 'json_object'}
    )
    return json.loads(resp.choices[0].message.content)

conversation = [
    {'role': 'user', 'content': 'Hi, I am Alice and I prefer Python over JavaScript.'},
    {'role': 'assistant', 'content': 'Great to meet you, Alice! Python is an excellent choice.'},
    {'role': 'user', 'content': 'I am building a REST API for my e-commerce startup.'}
]
entities = extract_entities(conversation)
print(json.dumps(entities, indent=2))

When to Apply Each Strategy

Choose your strategy based on your application's characteristics:

  • Sliding window: Simple chat apps where recent context is all that matters and users do not refer back to old exchanges. Cheapest to implement.
  • Token-aware trimming: Any production app where message lengths vary significantly. Always better than message-count trimming.
  • Summarization: Long-running assistants, customer support sessions, project management bots. Users expect the assistant to remember key facts from hours-old exchanges.
  • Entity memory: Personal assistants, user-preference-aware chatbots, tutoring systems where user profile matters throughout the session.

These strategies can also be combined: use entity memory for core facts plus a sliding window for recent conversation.

RAG as an Alternative to Context Stuffing

For knowledge-heavy applications, the best context management strategy is not to put documents in the context at all — instead, use RAG (Retrieval-Augmented Generation) to retrieve only the specific snippets that are relevant to the current query. This keeps context focused, costs less, and often produces better answers than injecting an entire document.

The key insight is that long context windows solve the can it fit? problem but not the will the model use it well? problem. Precise retrieval addresses both by giving the model only the information it actually needs for the current question.

Handling Max Context Errors Gracefully

Despite best efforts, you may encounter context limit errors in production, especially from user-generated content that is unexpectedly long. Always handle the context_length_exceeded error explicitly, rather than letting it bubble up as an unhandled exception to the user.

import openai
import tiktoken

client = openai.OpenAI()

def chat_with_context_guard(messages, model='gpt-4o-mini', max_tokens=100000):
    enc = tiktoken.encoding_for_model(model)
    total = sum(len(enc.encode(m.get('content',''))) + 4 for m in messages) + 3

    while total > max_tokens and len(messages) > 2:
        # Remove the second message (keep system prompt)
        removed = messages.pop(1)
        total -= len(enc.encode(removed.get('content',''))) + 4
        print(f'Trimmed a message. New total: {total} tokens')

    try:
        return client.chat.completions.create(model=model, messages=messages)
    except openai.BadRequestError as e:
        if 'context_length' in str(e):
            return {'error': 'Message history too long. Please start a new conversation.'}
        raise

Persisting Context Across Sessions

In a real application, users close and reopen the app. The server restarts. Context management must account for persistence across sessions, not just within a single session. This means storing conversation history in a database and loading it at the start of each session.

A sensible pattern: store the full raw history in PostgreSQL for auditing and fine-tuning purposes, but when loading context for a new API call, apply your summarization or trimming strategy to fit within the token budget. This way you never lose data but you do not pay for full history on every request.

Monitoring Context Growth in Production

Log the token count of every API call and track it over time by conversation ID. A healthy conversation should show gradual growth followed by a plateau (as summarization kicks in). A conversation that grows unboundedly to the context limit indicates your trimming logic is not working correctly.

Also monitor the average context length across all conversations. If it trends upward over time, it may indicate that your default system prompt is growing (from feature additions), that users are pasting longer inputs, or that RAG is returning progressively larger chunks. Each of these has a different fix.

Combining Strategies: A Production Pattern

A robust production context management system combines multiple strategies in layers:

  1. Before assembling context: check token budget and log it
  2. Apply entity extraction: inject a structured memory block with key facts
  3. Add recent history: include the last 6-10 messages verbatim
  4. Summarize older history: if older history exists, inject a running summary
  5. Guard check: if total still exceeds budget, trim oldest verbatim messages

This layered approach preserves the most important information (entity memory and recent context) while gracefully degrading older history into summaries, then into trimming only as a last resort.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: sliding windows and token-aware trimming are simple strategies that discard old context to stay within limits, summarization compresses old turns into a compact representation preserving key facts, and entity memory extracts structured facts for token-efficient long-term memory across a conversation. Next up we explore how to make LLMs return reliable JSON using JSON mode and structured outputs.

よくある質問

「Context内に収めるための戦略」レッスンは無料ですか？

はい。「Context内に収めるための戦略」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、AI Engineering Academyコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 AI Engineering Academyコースには全4レッスンが含まれています。

「Context内に収めるための戦略」で何を学びますか？

sliding windowメモリ、メッセージの要約、選択的なコンテキスト削減を実装し、token制限を超えずに長い会話を処理します。 ブラウザで直接実行するハンズオンコードでAI Engineering Academyを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

AI Engineering Academyを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのAI Engineering Academyは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン4/4です。

「Context内に収めるための戦略」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このAI Engineering Academyレッスンでコードを書いて実行できますか？

はい。すべてのAI Engineering Academyレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

  1. Tokenとは何か
  2. Context Window：サイズと影響
  3. APIコストの計算と予測
  4. Context内に収めるための戦略
← AI Engineering Academyに戻る