0Pricing
AI Engineering Academy · Lesson

Summary Memory and Token-Aware Truncation

Use ConversationSummaryMemory to automatically summarize older turns, keeping the conversation condensed while preserving key facts the user mentioned earlier.

Summary Memory and Token-Aware Truncation is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Token Cost of Full History

Conversation Buffer Memory keeps every message ever exchanged, which quickly consumes your context window. A 100-turn conversation might use 20,000 tokens just for history, leaving little room for the actual response. Summary Memory solves this by replacing old turns with a compressed summary.

How Summary Memory Works

When total tokens exceed a threshold, ConversationSummaryMemory feeds the oldest conversation turns to an LLM and asks it to summarize the key points. The full turns are discarded and replaced by this compact summary. Future turns accumulate on top of the summary.

  • Old turns: replaced by summary
  • Recent turns: kept verbatim
  • Net result: meaningful compression with minimal information loss

LangChain ConversationSummaryMemory

LangChain provides ConversationSummaryMemory that automatically summarizes whenever the buffer grows too large. You pass an LLM to the memory object so it can call the model to generate summaries on demand.

from langchain.memory import ConversationSummaryMemory
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model='gpt-4o-mini')

memory = ConversationSummaryMemory(
    llm=llm,
    return_messages=True
)

# Add messages manually
memory.save_context(
    {'input': 'My name is Alice and I am building a RAG system.'},
    {'output': 'Great, I can help you build a RAG system, Alice.'}
)

print(memory.load_memory_variables({}))

Summary Buffer Memory: Best of Both

ConversationSummaryBufferMemory is a hybrid approach: it keeps the most recent turns verbatim for accuracy and summarizes only the older turns that exceed a max_token_limit. This gives you exact recall of recent context plus compressed recall of older context.

from langchain.memory import ConversationSummaryBufferMemory
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model='gpt-4o-mini')

memory = ConversationSummaryBufferMemory(
    llm=llm,
    max_token_limit=500,  # Summarize when older turns exceed 500 tokens
    return_messages=True
)

# As the conversation grows, old turns get summarized automatically
print('Moving summary:', memory.moving_summary_buffer)

Counting Tokens Before Truncation

To make smart truncation decisions, you need to know how many tokens your messages consume. The tiktoken library lets you count tokens for any OpenAI model. This lets you implement token-aware truncation that stays strictly under a budget.

import tiktoken

def count_tokens(messages: list, model: str = 'gpt-4o') -> int:
    encoding = tiktoken.encoding_for_model(model)
    total = 0
    for msg in messages:
        # Each message has overhead tokens for role framing
        total += 4
        total += len(encoding.encode(msg.get('content', '')))
    total += 2  # Reply primer
    return total

messages = [
    {'role': 'user', 'content': 'Explain RAG to me.'},
    {'role': 'assistant', 'content': 'RAG stands for Retrieval-Augmented Generation...'}
]
print('Token count:', count_tokens(messages))

Manual Token-Aware Truncation

Sometimes you want full control over truncation without LangChain's memory classes. A simple approach: keep all messages, then remove the oldest non-system messages one at a time until the total token count fits within your budget.

def truncate_to_budget(messages: list, budget: int, model: str = 'gpt-4o') -> list:
    '''Remove oldest non-system messages until under budget.'''
    import tiktoken
    encoding = tiktoken.encoding_for_model(model)

    def token_count(msgs):
        total = 2
        for m in msgs:
            total += 4 + len(encoding.encode(m.get('content', '')))
        return total

    result = list(messages)
    while token_count(result) > budget and len(result) > 1:
        # Never remove the system prompt at index 0
        if result[0]['role'] == 'system':
            del result[1]
        else:
            del result[0]
    return result

Generating the Summary Prompt

When you build your own summary memory, you craft a prompt asking the LLM to distill the key facts. The prompt should instruct the model to capture user preferences, named entities, and unresolved questions — the facts most likely to matter in future turns.

from openai import OpenAI

client = OpenAI()

def summarize_history(old_summary: str, new_turns: list) -> str:
    turns_text = '\n'.join(
        f'{m["role"].capitalize()}: {m["content"]}' for m in new_turns
    )
    prompt = f'''Current summary:\n{old_summary}\n\nNew conversation turns:\n{turns_text}\n\nWrite an updated summary that captures key facts, user preferences, and open questions.'''

    resp = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': prompt}]
    )
    return resp.choices[0].message.content

Integrating Summary Into the System Prompt

Once you have a summary, inject it into the system prompt so the LLM always has context about the conversation history. Prepend the summary as a brief context block before the main persona instructions.

def build_messages_with_summary(summary: str, recent_turns: list, user_input: str) -> list:
    system_content = (
        'You are a helpful AI assistant.'
        + ('\n\n[Conversation summary]\n' + summary if summary else '')
    )

    messages = [{'role': 'system', 'content': system_content}]
    messages.extend(recent_turns)
    messages.append({'role': 'user', 'content': user_input})
    return messages

Triggering Summarization Automatically

A common pattern is to trigger summarization when the conversation exceeds a token threshold — for example, when the running history crosses 2,000 tokens. At that point, you summarize all but the last two turns and replace them with the summary.

class SummaryBufferChat:
    def __init__(self, max_tokens=2000):
        self.summary = ''
        self.recent_turns = []
        self.max_tokens = max_tokens

    def chat(self, user_message: str) -> str:
        from openai import OpenAI
        client = OpenAI()

        messages = build_messages_with_summary(self.summary, self.recent_turns, user_message)
        resp = client.chat.completions.create(model='gpt-4o', messages=messages)
        reply = resp.choices[0].message.content

        self.recent_turns.append({'role': 'user', 'content': user_message})
        self.recent_turns.append({'role': 'assistant', 'content': reply})

        if count_tokens(self.recent_turns) > self.max_tokens:
            to_summarize = self.recent_turns[:-2]
            self.recent_turns = self.recent_turns[-2:]
            self.summary = summarize_history(self.summary, to_summarize)

        return reply

Preserving Named Entities in Summaries

Summary quality depends heavily on the summarization prompt. If users mention their name, location, preferences, or deadlines, your prompt must explicitly instruct the model to include those facts. Generic summaries often drop proper nouns that matter most for personalization.

  • Include: names, dates, technical choices made, open questions
  • Exclude: filler exchanges, acknowledgements, repeated pleasantries

Trade-offs: Summary vs. Window Memory

Choosing between summary and window memory depends on your use case. Window memory preserves exact wording, which matters for precise recall but wastes tokens on irrelevant context. Summary memory compresses aggressively but introduces an extra LLM call and may lose edge-case details. Most production chatbots use a hybrid: recent exact turns plus an older rolling summary.

Quick Check

Test your understanding of summary memory and token-aware truncation concepts.

Lesson Recap

In this lesson you learned: summary memory compresses old turns via an LLM call, ConversationSummaryBufferMemory hybrids exact recent turns with summarized older ones, and tiktoken enables token-aware truncation to keep conversations within budget. Next up we explore persisting chat history in Redis and PostgreSQL for durable, scalable storage.

Frequently asked questions

Is the “Summary Memory and Token-Aware Truncation” lesson free?

Yes — the full text of “Summary Memory and Token-Aware Truncation” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Summary Memory and Token-Aware Truncation”?

Use ConversationSummaryMemory to automatically summarize older turns, keeping the conversation condensed while preserving key facts the user mentioned earlier. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Summary Memory and Token-Aware Truncation” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Why Stateless LLMs Need External Memory
  2. Buffer and Window Memory
  3. Summary Memory and Token-Aware Truncation
  4. Persisting Chat History in Redis and PostgreSQL
← Back to AI Engineering Academy