AI Engineering Academy · 课时

保持在上下文范围内的策略

实现滑动窗口记忆、消息摘要和选择性上下文裁剪，在不超过令牌限制的情况下处理较长对话。

保持在上下文范围内的策略 是 CoddyKit 上的免费 AI Engineering Academy 课时。这是第 4 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 AI Engineering Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Engineering Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

The Growing Conversation Problem

Every message exchanged in a chat conversation grows the token count of the next API call. In a long customer support session or a multi-hour coding session, the accumulated conversation history can easily exceed 20,000-50,000 tokens — all of which you must send to the API every single turn, paying for previously processed tokens repeatedly.

Without a context management strategy, your application will either hit the context limit and crash, or you will end up silently truncating messages and the model will lose track of important earlier context. Every production LLM application needs an explicit strategy for managing context growth.

Strategy 1: Sliding Window (Last N Messages)

The simplest strategy is to keep only the last N messages in the context, discarding older ones. This is called a sliding window. It is easy to implement, predictable in cost, and sufficient for many use cases where recent context matters more than older exchanges.

import openai

client = openai.OpenAI()

class SlidingWindowConversation:
    def __init__(self, system_prompt, window_size=10):
        self.system = system_prompt
        self.window_size = window_size
        self.history = []

    def chat(self, user_message):
        self.history.append({'role': 'user', 'content': user_message})

        # Keep only the last N messages
        windowed = self.history[-self.window_size:]
        messages = [{'role': 'system', 'content': self.system}] + windowed

        response = client.chat.completions.create(
            model='gpt-4o-mini',
            messages=messages
        )
        reply = response.choices[0].message.content
        self.history.append({'role': 'assistant', 'content': reply})
        print(f'Context: {len(windowed)} messages ({len(self.history)} total)')
        return reply

conv = SlidingWindowConversation('You are a helpful assistant.', window_size=6)
print(conv.chat('Hello! My name is Alice.'))
print(conv.chat('What is the capital of France?'))
print(conv.chat('What is my name?'))  # Might forget if window is small

Strategy 2: Token-Aware Trimming

Rather than trimming by message count, token-aware trimming removes messages until the total token count is below a threshold. This is more precise because messages vary widely in length — a message count limit might include 10 short messages or 3 very long ones, with very different token costs.

import tiktoken

def trim_to_token_budget(
    messages, system_prompt, model='gpt-4o-mini', max_input_tokens=8000
):
    enc = tiktoken.encoding_for_model(model)

    def count(msgs):
        return sum(len(enc.encode(m.get('content',''))) + 4 for m in msgs) + 3

    # System prompt token count
    system_tokens = len(enc.encode(system_prompt)) + 4
    budget = max_input_tokens - system_tokens

    # Trim from the oldest messages until we fit
    trimmed = list(messages)
    while trimmed and count(trimmed) > budget:
        trimmed.pop(0)  # Remove oldest message

    removed = len(messages) - len(trimmed)
    if removed:
        print(f'Trimmed {removed} old messages to stay within {max_input_tokens} tokens')
    return trimmed

Strategy 3: Conversation Summarization

Summarization compresses old conversation turns into a concise summary while preserving the key facts, decisions, and context that the user mentioned. This is more sophisticated than sliding windows because it retains the essence of past exchanges rather than simply discarding them.

The pattern works as follows: when the conversation exceeds a threshold, send the oldest N turns to the model with a 'Summarize this conversation so far' prompt, replace those turns with a single system message containing the summary, and continue the conversation with the new compressed history.

import openai

client = openai.OpenAI()

def summarize_conversation(messages_to_summarize, model='gpt-4o-mini'):
    summary_prompt = [
        {'role': 'system', 'content': 'You summarize conversations. Be concise but preserve key facts, decisions, and any specific information the user shared (names, numbers, preferences).'},
        {'role': 'user', 'content': 'Summarize this conversation:\n' + '\n'.join(
            f"{m['role'].upper()}: {m['content']}" for m in messages_to_summarize
        )}
    ]
    resp = client.chat.completions.create(model=model, messages=summary_prompt, max_tokens=500)
    return resp.choices[0].message.content

def compress_history(history, keep_recent=4):
    if len(history) <= keep_recent:
        return history
    to_summarize = history[:-keep_recent]
    summary = summarize_conversation(to_summarize)
    summary_msg = {'role': 'system', 'content': f'Earlier conversation summary: {summary}'}
    return [summary_msg] + history[-keep_recent:]

print('Summarization strategy compresses old turns into a single summary message')

Strategy 4: Entity Memory

Entity memory extracts and maintains a structured representation of key facts mentioned in the conversation — user preferences, names, project details, decisions made — rather than storing raw message history. Before each turn, the stored facts are injected into the system prompt.

This is more token-efficient than full history because you only include what is semantically important, not every word of every turn. Entity memory works especially well for long-running assistants where users refer back to preferences and facts established much earlier in the conversation.

import openai
import json

client = openai.OpenAI()

def extract_entities(messages, model='gpt-4o-mini'):
    prompt = [
        {'role': 'system', 'content': 'Extract key facts from this conversation as JSON: {"user_name": null, "preferences": [], "key_facts": []}'},
        {'role': 'user', 'content': '\n'.join(f"{m['role']}: {m['content']}" for m in messages)}
    ]
    resp = client.chat.completions.create(
        model=model,
        messages=prompt,
        response_format={'type': 'json_object'}
    )
    return json.loads(resp.choices[0].message.content)

conversation = [
    {'role': 'user', 'content': 'Hi, I am Alice and I prefer Python over JavaScript.'},
    {'role': 'assistant', 'content': 'Great to meet you, Alice! Python is an excellent choice.'},
    {'role': 'user', 'content': 'I am building a REST API for my e-commerce startup.'}
]
entities = extract_entities(conversation)
print(json.dumps(entities, indent=2))

When to Apply Each Strategy

Choose your strategy based on your application's characteristics:

Sliding window: Simple chat apps where recent context is all that matters and users do not refer back to old exchanges. Cheapest to implement.
Token-aware trimming: Any production app where message lengths vary significantly. Always better than message-count trimming.
Summarization: Long-running assistants, customer support sessions, project management bots. Users expect the assistant to remember key facts from hours-old exchanges.
Entity memory: Personal assistants, user-preference-aware chatbots, tutoring systems where user profile matters throughout the session.

These strategies can also be combined: use entity memory for core facts plus a sliding window for recent conversation.

RAG as an Alternative to Context Stuffing

For knowledge-heavy applications, the best context management strategy is not to put documents in the context at all — instead, use RAG (Retrieval-Augmented Generation) to retrieve only the specific snippets that are relevant to the current query. This keeps context focused, costs less, and often produces better answers than injecting an entire document.

The key insight is that long context windows solve the can it fit? problem but not the will the model use it well? problem. Precise retrieval addresses both by giving the model only the information it actually needs for the current question.

Handling Max Context Errors Gracefully

Despite best efforts, you may encounter context limit errors in production, especially from user-generated content that is unexpectedly long. Always handle the context_length_exceeded error explicitly, rather than letting it bubble up as an unhandled exception to the user.

import openai
import tiktoken

client = openai.OpenAI()

def chat_with_context_guard(messages, model='gpt-4o-mini', max_tokens=100000):
    enc = tiktoken.encoding_for_model(model)
    total = sum(len(enc.encode(m.get('content',''))) + 4 for m in messages) + 3

    while total > max_tokens and len(messages) > 2:
        # Remove the second message (keep system prompt)
        removed = messages.pop(1)
        total -= len(enc.encode(removed.get('content',''))) + 4
        print(f'Trimmed a message. New total: {total} tokens')

    try:
        return client.chat.completions.create(model=model, messages=messages)
    except openai.BadRequestError as e:
        if 'context_length' in str(e):
            return {'error': 'Message history too long. Please start a new conversation.'}
        raise

Persisting Context Across Sessions

In a real application, users close and reopen the app. The server restarts. Context management must account for persistence across sessions, not just within a single session. This means storing conversation history in a database and loading it at the start of each session.

A sensible pattern: store the full raw history in PostgreSQL for auditing and fine-tuning purposes, but when loading context for a new API call, apply your summarization or trimming strategy to fit within the token budget. This way you never lose data but you do not pay for full history on every request.

Monitoring Context Growth in Production

Log the token count of every API call and track it over time by conversation ID. A healthy conversation should show gradual growth followed by a plateau (as summarization kicks in). A conversation that grows unboundedly to the context limit indicates your trimming logic is not working correctly.

Also monitor the average context length across all conversations. If it trends upward over time, it may indicate that your default system prompt is growing (from feature additions), that users are pasting longer inputs, or that RAG is returning progressively larger chunks. Each of these has a different fix.

Combining Strategies: A Production Pattern

A robust production context management system combines multiple strategies in layers:

Before assembling context: check token budget and log it
Apply entity extraction: inject a structured memory block with key facts
Add recent history: include the last 6-10 messages verbatim
Summarize older history: if older history exists, inject a running summary
Guard check: if total still exceeds budget, trim oldest verbatim messages

This layered approach preserves the most important information (entity memory and recent context) while gracefully degrading older history into summaries, then into trimming only as a last resort.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: sliding windows and token-aware trimming are simple strategies that discard old context to stay within limits, summarization compresses old turns into a compact representation preserving key facts, and entity memory extracts structured facts for token-efficient long-term memory across a conversation. Next up we explore how to make LLMs return reliable JSON using JSON mode and structured outputs.

常见问题解答

「保持在上下文范围内的策略」课时是免费的吗？

是的 — 「保持在上下文范围内的策略」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Engineering Academy 课程的其余内容，请升级到 CoddyKit PRO。 AI Engineering Academy 课程共包含 4 节课。

「保持在上下文范围内的策略」这节课中我会学到什么？

实现滑动窗口记忆、消息摘要和选择性上下文裁剪，在不超过令牌限制的情况下处理较长对话。你通过在浏览器中直接运行的动手代码来练习 AI Engineering Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 AI Engineering Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 AI Engineering Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 4 节课，共 4 节。

「保持在上下文范围内的策略」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 AI Engineering Academy 课中编写并运行代码吗？

能。每节 AI Engineering Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 AI Engineering Academy