Caching Long Prefixes

Every long prompt pays prefill cost to encode its tokens before generating. When the same long prefix repeats across many calls — a system prompt, a tool spec, a large reference document — prompt caching lets the provider reuse the already-computed attention state instead of recomputing it.

• Cache hits cut latency and input cost dramatically.
• The savings scale with prefix size and reuse frequency.

Caches key on an exact token-prefix match from the start of the prompt. The cached span runs from the beginning up to the first point of divergence. Change anything early and everything after it misses the cache.

Implication: the layout that maximizes hits puts the most stable content first and the most variable content last.

Arrange content from most to least stable: immutable system rules and tool definitions, then large stable reference material, then session-stable context, then the volatile per-request input and question last.

• Stable -> front (cacheable).
• Volatile -> back (the only part recomputed).

This single ordering principle drives most caching wins.

Some providers let you mark explicit cache breakpoints. Place them at the end of each stable segment so the system can cache up to that point. Mark the largest stable block (the reference corpus or long system prompt) as cacheable.

Without explicit markers, structure stability-ordering anyway so implicit prefix matching still helps.

Subtle, unintentional changes silently break caching: a timestamp in the system prompt, a per-request id injected early, reordered tool definitions, or non-deterministic JSON key order. Each shifts the prefix and forces a full recompute.

Audit your prefix for anything that varies between calls and move it after the cached region.

Caches expire after a provider-defined time-to-live, often refreshed on each hit. Cold misses recur if calls are too sparse. For bursty, high-frequency workloads caching pays off; for rare, scattered calls the cache may expire between uses.

Match your traffic pattern to the TTL, and consider keep-alive pings for valuable prefixes.

Caching typically charges a small premium to write a cache entry and a steep discount to read it. The economics favor reuse: one write amortized over many reads is a large net saving; a single use that only writes can cost slightly more.

• High reuse -> cache aggressively.
• One-shot prefixes -> caching may not pay.

Make your system prompt and tool specs deterministic and version-pinned. Sort tool definitions canonically, avoid embedding dynamic data, and change them only on deliberate releases. A stable system block becomes a long-lived, high-hit cache entry shared across all requests.

Treat the cached prefix as an artifact with a version, not a string you tweak casually.

In agent loops, the growing conversation is a natural cache: each turn extends a prefix the next turn reuses. Append new turns at the tail and never rewrite earlier turns, so the prior cache stays valid.

When you must compact, do it in a way that creates a new stable prefix for subsequent turns rather than mutating old ones repeatedly.

Instrument it. Most providers report cached vs uncached input tokens per call. Track hit rate, and if it is low, inspect the prefix for drift. A regression in hit rate usually points to a recently introduced dynamic value early in the prompt.

• Log cached_tokens / total_input_tokens.
• Alert when hit rate drops after a deploy.

To control cost on long prefixes: order content by stability, mark the largest stable block as a cache breakpoint, eliminate hidden drift, pin and version system/tool blocks, append-only in agent loops, and monitor hit rate. The goal is one big reusable prefix and a small volatile tail recomputed each call.

You reuse a 100k-token reference corpus across thousands of daily queries but your cache hit rate is near zero.

Prompt caching reuses the prefill of an exact, stable prefix, slashing latency and input cost when reuse is frequent. Order content most-stable-first, mark the big stable block as a breakpoint, and push all volatility to the tail. Eliminate hidden drift, pin and version system/tool blocks, append-only in agent loops, and monitor the hit rate so a regression flags a newly introduced early-varying value.