AI Agents · Lesson

Short-Term Memory in the Context Window

Store conversation history in the messages array — the simplest memory there is, and why it has hard limits.

Short-Term Memory in the Context Window is a free AI Agents lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Memory Is Just a List

The "memory" of a chat-style LLM is the messages list you send on every call. The model is stateless — it knows nothing between requests.

"Short-term memory" = whatever is in that list right now.

Why It Works

You append each user message and each assistant response. By the time you ask the third question, the model sees:

messages = [
  {'role': 'system', 'content': 'You are helpful.'},
  {'role': 'user',   'content': 'My name is Alice.'},
  {'role': 'assistant', 'content': 'Nice to meet you, Alice.'},
  {'role': 'user',   'content': 'What is my name?'}
]
# Model replies: 'Alice'
for m in messages:
    print(f"{m['role']}: {m['content']}")
print("Model replies: Alice")

Context Window Limits

Every model has a hard context window — the max number of tokens in input + output combined.

gpt-4o-mini: 128k tokens
claude-sonnet-4-5: 200k tokens
gemini-1.5-pro: 2M tokens

Exceed it and the API errors out.

Token Cost Is Linear

You pay for every token, every turn. A 30-turn chat where each turn is 500 tokens costs you 15,000 input tokens on the LAST call alone — the whole history replays.

Long sessions get expensive fast.

Lost-In-The-Middle

Even with a 200k context, models pay LESS attention to content in the middle of a long prompt. Important info should go at the start (system) or end (most recent user message).

Sliding Window Truncation

The simplest memory management — keep the system message + last N turns:

MAX_TURNS = 20  # 10 user + 10 assistant

def trim(messages):
    system = messages[0]
    rest = messages[1:]
    return [system] + rest[-MAX_TURNS:]

demo_messages = [{'role': 'system', 'content': 'sys'}] + [
    {'role': 'user' if i % 2 == 0 else 'assistant', 'content': f'msg {i}'} for i in range(30)
]
trimmed = trim(demo_messages)
print(f"Original messages: {len(demo_messages)}")
print(f"Trimmed messages: {len(trimmed)}")

Token-Based Trimming

More precise: trim by token count, not turn count:

import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')

def trim_by_tokens(messages, max_tokens=8000):
    out = [messages[0]]   # keep system
    tokens_left = max_tokens - len(enc.encode(messages[0]['content']))
    # walk backwards from newest
    for m in reversed(messages[1:]):
        cost = len(enc.encode(m['content'])) + 4
        if cost > tokens_left:
            break
        out.insert(1, m)
        tokens_left -= cost
    return out

Keep Pairs Together

Truncating mid-pair leaves orphan tool calls. Always trim user+assistant turns together:

# Bad: cuts after assistant call but before tool result
# Good: trim whole turns (user + all responses)
print("Bad: cuts after assistant call but before tool result")
print("Good: trim whole turns (user + all responses)")

When Short-Term Is Not Enough

Short-term memory breaks when:

Conversation > 20 turns
User returns the next day expecting recall
Multiple users share the same agent

You need a different strategy — see the next lessons.

System Prompt Stays Pinned

The system message must always be the first item. Never trim it. It carries identity, rules, and tool descriptions.

Pinning Recent System Updates

If you append "remember the user prefers Celsius" as a system note, pin it at the front like the original system prompt:

SYSTEM_PROMPT = 'You are a helpful assistant.'
messages = [
    {'role': 'system', 'content': SYSTEM_PROMPT},
    {'role': 'system', 'content': 'USER PREFERENCE: Celsius units'},
    {'role': 'user', 'content': 'What is the weather in Paris?'},
    {'role': 'assistant', 'content': 'It is 21C and sunny.'},
]
print('Pinned system messages:')
for m in messages:
    if m['role'] == 'system':
        print(' -', m['content'])
print('Total messages:', len(messages))

Context Limit

What happens when you exceed the model's context window?

Recap

Short-term memory = the messages list. Manage it with sliding-window or token-based trimming, pin the system message, and accept the cost grows with turns.

Frequently asked questions

Is the “Short-Term Memory in the Context Window” lesson free?

Yes — the full text of “Short-Term Memory in the Context Window” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Short-Term Memory in the Context Window”?

Store conversation history in the messages array — the simplest memory there is, and why it has hard limits. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Short-Term Memory in the Context Window” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents