AI Engineering Academy · Lesson

Strategies for Staying Within Context

Implement sliding window memory, message summarization, and selective context trimming to handle long conversations without exceeding token limits.

Strategies for Staying Within Context is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Growing Conversation Problem

Every message exchanged in a chat conversation grows the token count of the next API call. In a long customer support session or a multi-hour coding session, the accumulated conversation history can easily exceed 20,000-50,000 tokens — all of which you must send to the API every single turn, paying for previously processed tokens repeatedly.

Without a context management strategy, your application will either hit the context limit and crash, or you will end up silently truncating messages and the model will lose track of important earlier context. Every production LLM application needs an explicit strategy for managing context growth.

Strategy 1: Sliding Window (Last N Messages)

The simplest strategy is to keep only the last N messages in the context, discarding older ones. This is called a sliding window. It is easy to implement, predictable in cost, and sufficient for many use cases where recent context matters more than older exchanges.

import openai

client = openai.OpenAI()

class SlidingWindowConversation:
    def __init__(self, system_prompt, window_size=10):
        self.system = system_prompt
        self.window_size = window_size
        self.history = []

    def chat(self, user_message):
        self.history.append({'role': 'user', 'content': user_message})

        # Keep only the last N messages
        windowed = self.history[-self.window_size:]
        messages = [{'role': 'system', 'content': self.system}] + windowed

        response = client.chat.completions.create(
            model='gpt-4o-mini',
            messages=messages
        )
        reply = response.choices[0].message.content
        self.history.append({'role': 'assistant', 'content': reply})
        print(f'Context: {len(windowed)} messages ({len(self.history)} total)')
        return reply

conv = SlidingWindowConversation('You are a helpful assistant.', window_size=6)
print(conv.chat('Hello! My name is Alice.'))
print(conv.chat('What is the capital of France?'))
print(conv.chat('What is my name?'))  # Might forget if window is small

Strategy 2: Token-Aware Trimming

Rather than trimming by message count, token-aware trimming removes messages until the total token count is below a threshold. This is more precise because messages vary widely in length — a message count limit might include 10 short messages or 3 very long ones, with very different token costs.

import tiktoken

def trim_to_token_budget(
    messages, system_prompt, model='gpt-4o-mini', max_input_tokens=8000
):
    enc = tiktoken.encoding_for_model(model)

    def count(msgs):
        return sum(len(enc.encode(m.get('content',''))) + 4 for m in msgs) + 3

    # System prompt token count
    system_tokens = len(enc.encode(system_prompt)) + 4
    budget = max_input_tokens - system_tokens

    # Trim from the oldest messages until we fit
    trimmed = list(messages)
    while trimmed and count(trimmed) > budget:
        trimmed.pop(0)  # Remove oldest message

    removed = len(messages) - len(trimmed)
    if removed:
        print(f'Trimmed {removed} old messages to stay within {max_input_tokens} tokens')
    return trimmed

Strategy 3: Conversation Summarization

Summarization compresses old conversation turns into a concise summary while preserving the key facts, decisions, and context that the user mentioned. This is more sophisticated than sliding windows because it retains the essence of past exchanges rather than simply discarding them.

The pattern works as follows: when the conversation exceeds a threshold, send the oldest N turns to the model with a 'Summarize this conversation so far' prompt, replace those turns with a single system message containing the summary, and continue the conversation with the new compressed history.

import openai

client = openai.OpenAI()

def summarize_conversation(messages_to_summarize, model='gpt-4o-mini'):
    summary_prompt = [
        {'role': 'system', 'content': 'You summarize conversations. Be concise but preserve key facts, decisions, and any specific information the user shared (names, numbers, preferences).'},
        {'role': 'user', 'content': 'Summarize this conversation:\n' + '\n'.join(
            f"{m['role'].upper()}: {m['content']}" for m in messages_to_summarize
        )}
    ]
    resp = client.chat.completions.create(model=model, messages=summary_prompt, max_tokens=500)
    return resp.choices[0].message.content

def compress_history(history, keep_recent=4):
    if len(history) <= keep_recent:
        return history
    to_summarize = history[:-keep_recent]
    summary = summarize_conversation(to_summarize)
    summary_msg = {'role': 'system', 'content': f'Earlier conversation summary: {summary}'}
    return [summary_msg] + history[-keep_recent:]

print('Summarization strategy compresses old turns into a single summary message')

Strategy 4: Entity Memory

Entity memory extracts and maintains a structured representation of key facts mentioned in the conversation — user preferences, names, project details, decisions made — rather than storing raw message history. Before each turn, the stored facts are injected into the system prompt.

This is more token-efficient than full history because you only include what is semantically important, not every word of every turn. Entity memory works especially well for long-running assistants where users refer back to preferences and facts established much earlier in the conversation.

import openai
import json

client = openai.OpenAI()

def extract_entities(messages, model='gpt-4o-mini'):
    prompt = [
        {'role': 'system', 'content': 'Extract key facts from this conversation as JSON: {"user_name": null, "preferences": [], "key_facts": []}'},
        {'role': 'user', 'content': '\n'.join(f"{m['role']}: {m['content']}" for m in messages)}
    ]
    resp = client.chat.completions.create(
        model=model,
        messages=prompt,
        response_format={'type': 'json_object'}
    )
    return json.loads(resp.choices[0].message.content)

conversation = [
    {'role': 'user', 'content': 'Hi, I am Alice and I prefer Python over JavaScript.'},
    {'role': 'assistant', 'content': 'Great to meet you, Alice! Python is an excellent choice.'},
    {'role': 'user', 'content': 'I am building a REST API for my e-commerce startup.'}
]
entities = extract_entities(conversation)
print(json.dumps(entities, indent=2))

When to Apply Each Strategy

Choose your strategy based on your application's characteristics:

Sliding window: Simple chat apps where recent context is all that matters and users do not refer back to old exchanges. Cheapest to implement.
Token-aware trimming: Any production app where message lengths vary significantly. Always better than message-count trimming.
Summarization: Long-running assistants, customer support sessions, project management bots. Users expect the assistant to remember key facts from hours-old exchanges.
Entity memory: Personal assistants, user-preference-aware chatbots, tutoring systems where user profile matters throughout the session.

These strategies can also be combined: use entity memory for core facts plus a sliding window for recent conversation.

RAG as an Alternative to Context Stuffing

For knowledge-heavy applications, the best context management strategy is not to put documents in the context at all — instead, use RAG (Retrieval-Augmented Generation) to retrieve only the specific snippets that are relevant to the current query. This keeps context focused, costs less, and often produces better answers than injecting an entire document.

The key insight is that long context windows solve the can it fit? problem but not the will the model use it well? problem. Precise retrieval addresses both by giving the model only the information it actually needs for the current question.

Handling Max Context Errors Gracefully

Despite best efforts, you may encounter context limit errors in production, especially from user-generated content that is unexpectedly long. Always handle the context_length_exceeded error explicitly, rather than letting it bubble up as an unhandled exception to the user.

import openai
import tiktoken

client = openai.OpenAI()

def chat_with_context_guard(messages, model='gpt-4o-mini', max_tokens=100000):
    enc = tiktoken.encoding_for_model(model)
    total = sum(len(enc.encode(m.get('content',''))) + 4 for m in messages) + 3

    while total > max_tokens and len(messages) > 2:
        # Remove the second message (keep system prompt)
        removed = messages.pop(1)
        total -= len(enc.encode(removed.get('content',''))) + 4
        print(f'Trimmed a message. New total: {total} tokens')

    try:
        return client.chat.completions.create(model=model, messages=messages)
    except openai.BadRequestError as e:
        if 'context_length' in str(e):
            return {'error': 'Message history too long. Please start a new conversation.'}
        raise

Persisting Context Across Sessions

In a real application, users close and reopen the app. The server restarts. Context management must account for persistence across sessions, not just within a single session. This means storing conversation history in a database and loading it at the start of each session.

A sensible pattern: store the full raw history in PostgreSQL for auditing and fine-tuning purposes, but when loading context for a new API call, apply your summarization or trimming strategy to fit within the token budget. This way you never lose data but you do not pay for full history on every request.

Monitoring Context Growth in Production

Log the token count of every API call and track it over time by conversation ID. A healthy conversation should show gradual growth followed by a plateau (as summarization kicks in). A conversation that grows unboundedly to the context limit indicates your trimming logic is not working correctly.

Also monitor the average context length across all conversations. If it trends upward over time, it may indicate that your default system prompt is growing (from feature additions), that users are pasting longer inputs, or that RAG is returning progressively larger chunks. Each of these has a different fix.

Combining Strategies: A Production Pattern

A robust production context management system combines multiple strategies in layers:

Before assembling context: check token budget and log it
Apply entity extraction: inject a structured memory block with key facts
Add recent history: include the last 6-10 messages verbatim
Summarize older history: if older history exists, inject a running summary
Guard check: if total still exceeds budget, trim oldest verbatim messages

This layered approach preserves the most important information (entity memory and recent context) while gracefully degrading older history into summaries, then into trimming only as a last resort.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: sliding windows and token-aware trimming are simple strategies that discard old context to stay within limits, summarization compresses old turns into a compact representation preserving key facts, and entity memory extracts structured facts for token-efficient long-term memory across a conversation. Next up we explore how to make LLMs return reliable JSON using JSON mode and structured outputs.

Frequently asked questions

Is the “Strategies for Staying Within Context” lesson free?

Yes — the full text of “Strategies for Staying Within Context” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Strategies for Staying Within Context”?

Implement sliding window memory, message summarization, and selective context trimming to handle long conversations without exceeding token limits. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Strategies for Staying Within Context” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Engineering Academy