Caching Long Prefixes

Every long prompt pays prefill cost to encode its tokens before generating. When the same long prefix repeats across many calls — a system prompt, a tool spec, a large reference document — prompt caching lets the provider reuse the already-computed attention state instead of recomputing it.

• Cache hits cut latency and input cost dramatically.
• The savings scale with prefix size and reuse frequency.

Caches key on an exact token-prefix match from the start of the prompt. The cached span runs from the beginning up to the first point of divergence. Change anything early and everything after it misses the cache.

Implication: the layout that maximizes hits puts the most stable content first and the most variable content last.