Million-Token Context Windows

Million-token windows let you place entire codebases, document corpora, or multi-hour transcripts directly in context — no retrieval system required. The promise is in-context everything: the model reasons over the raw material rather than over a lossy summary.

• Whole-repository reasoning and refactors.
• Cross-document synthesis without chunking.
• Long-horizon conversations that retain early detail.

But capacity is not the same as effective use.

The advertised window is an upper bound, not a guarantee of uniform recall. Effective context — the span over which the model reliably retrieves and reasons — is usually smaller and uneven across positions.

Treat the window as a budget with non-uniform value: tokens near the edges are recalled better than tokens buried in the middle.

Attention cost grows with context length, and so does time-to-first-token. A million-token prompt can mean seconds of prefill latency and substantial per-call cost, even when the answer is short.

• Prefill dominates latency for huge prompts.
• Cost scales with input tokens regardless of output size.
• Stuffing context you do not need is paying to confuse the model.

More context can hurt accuracy. Irrelevant material dilutes attention and introduces distractors the model may latch onto. A focused 20k-token prompt often beats a noisy 500k-token one for a specific question.

The rule: include what helps the task, exclude what merely could be relevant.

Long context wins when the task needs global, cross-cutting reasoning that retrieval would fragment: 'find every place this invariant is violated across the repo', 'reconcile contradictions across all ten contracts'. Chunked retrieval can miss the connection that spans chunks.

Use full context for synthesis; use retrieval for needle lookups in enormous corpora.

Retrieval wins when the relevant fraction is tiny and stable. Fetching the 5 relevant pages out of 50,000 is cheaper, faster, and often more accurate than stuffing all 50,000 and hoping the model finds them.

• Frequent queries over a static corpus -> index once, retrieve cheaply.
• One-off global synthesis -> long context.

Many systems combine both: retrieve to narrow, then long-context to synthesize.

Because recall varies by position, where you place content is a design decision. Put the task instruction and the most critical material at the boundaries the model recalls best, and avoid burying must-use facts in the deep middle.

You manage position deliberately, not by accident of concatenation order.

Never assume the model used a buried fact. Probe it. Insert a known canary fact at a known depth and ask for it; measure recall across depths to characterize your model on your prompt shape before trusting it in production.

In long agentic sessions, do not let context grow unbounded. Periodically compact: summarize stale turns into a dense state object and drop the raw history. This preserves the signal while reclaiming budget and reducing dilution.

Accumulation is the default and the trap; compaction is the discipline.

The strongest designs blend strategies: retrieve a focused candidate set, place it with deliberate positioning, cache the stable prefix, and compact the conversation. Long context is one tool in a budget-management toolkit, not a replacement for engineering.

• Retrieve to shrink.
• Position to expose.
• Cache to cheapen.
• Compact to sustain.

Before reaching for the full window, ask:

• Does the task need global reasoning, or a needle? Needle -> retrieve.
• Is the relevant fraction small and stable? Yes -> retrieve/cache.
• Will latency/cost of a huge prefill be acceptable? No -> shrink.
• Have I verified recall at the depth I am relying on? If not, probe first.

Capacity is permission, not a plan.

You must answer one narrow factual question that lives on roughly 3 pages out of a static 40,000-page archive, and you will run this query thousands of times a day.

A huge window is a non-uniform budget, not free recall. Effective context is smaller than capacity, latency and cost scale with input, and irrelevant material dilutes accuracy. Use long context for global synthesis, retrieval for needles, and hybrids for the rest — positioning critical content at strong recall zones, caching stable prefixes, compacting long sessions, and always probing recall before you trust a buried fact.