Why Long Contexts Don't Scale
Cost grows linearly, quality degrades, and 'lost-in-the-middle' makes the model forget content buried in long prompts.
Why Long Contexts Don't Scale is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Bigger Is Not Always Better
Modern models have huge context windows — 200k, 1M, even 2M tokens. It is tempting to "just dump everything in" instead of building real memory.
That approach fails in production for three reasons.
Reason 1: Cost
Every token in every call costs money. A 100k-token system prompt with 1000 turns = 100M tokens of input, totalling tens of dollars per session.
Prompt caching helps but does not eliminate the cost.
Reason 2: Latency
Processing 100k input tokens adds 1-3 seconds to time-to-first-token. For chat UX you want TTFT under 500ms.
Reason 3: Quality Drops
Models pay less attention to content in the middle of long prompts — the "lost-in-the-middle" effect documented by Liu et al. 2023.
Information at the start or end is recalled accurately; information in the middle is often missed.
A Concrete Test
Insert a unique fact in different positions of a long document and ask the model to retrieve it. Accuracy:
- Start of doc: 95%
- End of doc: 90%
- Middle: 50-70%
Long Context for Eval, Not Production
Long contexts are useful for one-off tasks (analyze this whole codebase) but unsuited for steady-state production agents.
For production, use retrieval to inject only the relevant 5% on each turn.
When Long Contexts Help
- Initial codebase analysis
- Whole-document Q&A in a single call
- Comparison of two long documents
One-shot tasks with no replay.
When Long Contexts Hurt
- Chatbots with hundreds of turns
- Multi-tenant agents shared across users
- Anywhere cost or latency matters
The Right Architecture
For long-running agents:
- Recent turns in the prompt (last 10-20)
- Older turns summarised into a running summary
- Facts vectorised into long-term memory
- Retrieve the top-K relevant memories per turn
Hybrid Approach
Combine techniques:
messages = [
{'role': 'system', 'content': PERSONA},
{'role': 'system', 'content': f'Conversation summary so far: {summary}'},
{'role': 'system', 'content': f'Relevant past facts: {retrieved_facts}'},
*last_n_turns,
]Compaction Triggers
Decide WHEN to compact:
- Every N turns (simple)
- When token count > threshold (better)
- When the model starts forgetting (best, but hard to detect)
Lost in the Middle
What is the "lost-in-the-middle" effect?
Recap
Don't solve memory by maxing out context. Use summaries + retrieval to surface what matters per turn.
Frequently asked questions
Is the “Why Long Contexts Don't Scale” lesson free?
Yes — the full text of “Why Long Contexts Don't Scale” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Why Long Contexts Don't Scale”?
Cost grows linearly, quality degrades, and 'lost-in-the-middle' makes the model forget content buried in long prompts. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Why Long Contexts Don't Scale” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Short-Term Memory in the Context Window
- Why Long Contexts Don't Scale
- Summarisation as Compression
- Simple Memory Stores (Key-Value)