OpenAI Prompt Prefix Caching
Leverage OpenAI's automatic prompt caching that discounts repeated long system prompt prefixes at 50 percent off, and structure your prompts to maximize cache hit rates.
OpenAI Prompt Prefix Caching is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is Prompt Prefix Caching?
Prompt prefix caching is a server-side optimization built into OpenAI's API that automatically discounts tokens in the prompt prefix that were seen in a recent prior request. Unlike application-level caching that returns a stored response, prompt prefix caching still calls the model — but at a 50 percent reduced input token price for the cached prefix portion. It reduces cost without sacrificing fresh generation.
How Prefix Caching Works Under the Hood
Modern LLMs represent prompts as KV (key-value) caches in GPU memory. Processing a prompt means computing attention keys and values for every token. If the first N tokens of two consecutive requests are identical, OpenAI can reuse the KV cache from the first request, skipping the expensive compute for those tokens. The API does this automatically and transparently — you just pay the lower cached token rate when it applies.
# No code changes needed to enable prefix caching!
# It is automatic on supported models.
# The API response shows you how many tokens were cached:
# response.usage.prompt_tokens_details.cached_tokens
# Example response usage:
# ChatCompletionUsage(
# prompt_tokens=2048,
# completion_tokens=256,
# total_tokens=2304,
# prompt_tokens_details=PromptTokensDetails(
# cached_tokens=1984, # these tokens were served from KV cache
# audio_tokens=0,
# )
# )Checking Cache Hit in the Response
After each API call, inspect response.usage.prompt_tokens_details.cached_tokens to see how many input tokens were served from the KV cache. If cached_tokens > 0, you paid the 50 percent discount rate for those tokens. Logging this value lets you track your actual cache efficiency and compute the cost savings from prefix caching over time.
from openai import OpenAI
client = OpenAI()
SYSTEM_PROMPT = 'You are an expert AI engineer assistant. ' * 100 # long system prompt
def call_with_cache_check(user_message: str):
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': SYSTEM_PROMPT},
{'role': 'user', 'content': user_message},
],
)
usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens if usage.prompt_tokens_details else 0
print(f'Total prompt tokens: {usage.prompt_tokens}')
print(f'Cached tokens: {cached} ({100*cached//usage.prompt_tokens}%)')
return response.choices[0].message.contentThe Prefix Must Be Exactly Identical
Prefix caching only applies when the first N tokens are byte-for-byte identical to a recent prior request. Even a single character change in the system prompt invalidates the cache. OpenAI caches in 128-token chunks — the cache applies to complete chunks that match exactly. This means the portion of your prompt that varies per request should come after the long, stable prefix to maximize cached tokens.
# Optimal structure for prefix caching:
# [LONG STABLE SYSTEM PROMPT] [CACHED DOCUMENTS] [USER QUERY]
# ↑ ↑ ↑
# always same always same varies per request
# → cached at 50% → cached at 50% → not cached, full price
# BAD structure (defeats prefix caching):
# [USER QUERY] [CACHED DOCUMENTS] [LONG STABLE SYSTEM PROMPT]
# ↑ ↑
# changes every request never cached because
# so prefix never matches it comes after the queryStructuring Prompts for Maximum Cache Efficiency
To maximize cache hit rates, structure your prompts so the stable portions come first. In a RAG system: (1) system prompt with instructions and persona, (2) retrieved documents that change only when the query changes significantly, (3) conversation history, (4) user query at the very end. The system prompt alone — often 500-2000 tokens — will typically be cached, saving 25-50 percent of input token costs.
def build_rag_prompt_for_caching(
system_prompt: str,
retrieved_docs: list[str],
conversation_history: list[dict],
user_query: str,
) -> list[dict]:
# Order: stable → semi-stable → variable
context_block = '\n\n'.join(
f'[Document {i+1}]\n{doc}' for i, doc in enumerate(retrieved_docs)
)
return [
# 1. Stable system prompt (always cached after first request)
{'role': 'system', 'content': system_prompt},
# 2. Context injection as a user message (cached when same docs retrieved)
{'role': 'user', 'content': f'Context documents:\n{context_block}'},
{'role': 'assistant', 'content': 'I have read the documents.'},
# 3. Conversation history (semi-stable)
*conversation_history,
# 4. Current user query (always different → never cached prefix)
{'role': 'user', 'content': user_query},
]Cache Duration and Eviction
OpenAI's KV cache is maintained in-memory on GPU and has an eviction policy. Prefixes that have not been reused within approximately 5-10 minutes are evicted as other requests take up GPU memory. This means prefix caching benefits are largest for high-throughput applications with frequent requests sharing the same prefix. Low-traffic applications may see few cache hits because the prefix is evicted between spaced-out requests.
Supported Models and Pricing
As of 2025, prompt prefix caching is available on GPT-4o, GPT-4o-mini, o1, and o3-mini models. The cached token price is 50 percent of the standard input token price for most models. The minimum cacheable prefix length is 1024 tokens — shorter prefixes receive no discount. Always check the OpenAI pricing page for the current rates, as pricing evolves as the feature matures.
# Rough pricing reference (verify at platform.openai.com/pricing)
PRICING = {
'gpt-4o': {
'input_per_1M': 2.50,
'cached_input_per_1M': 1.25, # 50% off
'output_per_1M': 10.00,
},
'gpt-4o-mini': {
'input_per_1M': 0.15,
'cached_input_per_1M': 0.075, # 50% off
'output_per_1M': 0.60,
},
}
def estimate_cost_with_caching(prompt_tokens, cached_tokens, output_tokens, model):
p = PRICING[model]
uncached = (prompt_tokens - cached_tokens) * p['input_per_1M'] / 1_000_000
cached_cost = cached_tokens * p['cached_input_per_1M'] / 1_000_000
output_cost = output_tokens * p['output_per_1M'] / 1_000_000
return uncached + cached_cost + output_costAnthropic Prompt Caching
Anthropic offers a similar feature called prompt caching for Claude models, but requires explicit opt-in by marking cache breakpoints in the prompt with a cache_control field. Unlike OpenAI's automatic caching, you explicitly mark which portions of the prompt should be cached (up to 4 cache breakpoints per request). Cached tokens cost 10 percent of the standard input price and are stored for 5 minutes.
import anthropic
client = anthropic.Anthropic()
LONG_DOCUMENT = 'This is a very long reference document...' * 500 # 2000+ tokens
response = client.messages.create(
model='claude-sonnet-4-5',
max_tokens=1024,
system=[
{
'type': 'text',
'text': 'You are a helpful assistant.',
},
{
'type': 'text',
'text': LONG_DOCUMENT,
'cache_control': {'type': 'ephemeral'}, # mark for caching
}
],
messages=[{'role': 'user', 'content': 'Summarize the document.'}],
)
print(response.usage.cache_read_input_tokens) # tokens served from cache
print(response.usage.cache_creation_input_tokens) # tokens written to cacheCombining Prefix Caching with Application Caching
Prefix caching and application-level caching are complementary. Prefix caching reduces the cost of each API call but still calls the LLM. Application-level exact and semantic caches eliminate API calls entirely for repeated queries. Use prefix caching for all requests to reduce per-call input cost, and layer application-level caching on top to eliminate calls entirely for frequently repeated queries. Together, they can reduce AI infrastructure costs by 60-80 percent.
# Three-layer cost optimization stack
#
# Layer 1: Exact cache (Redis, hash-based)
# → Eliminates 100% of API cost for identical requests
# → Miss rate: ~60-80% (most queries are unique)
#
# Layer 2: Semantic cache (vector similarity)
# → Eliminates 100% of API cost for semantically similar requests
# → Miss rate: ~40-60% of remaining queries
#
# Layer 3: OpenAI prefix caching (automatic)
# → Reduces input token cost by 50% for long stable prefixes
# → Applies to ALL remaining API calls that escape layers 1 and 2
#
# Combined effect: 60-80% cost reduction in FAQ/support applicationsMeasuring Your Cache Efficiency
Track cache efficiency ratio as a composite metric: total tokens at full price divided by total tokens actually billed. This accounts for all caching layers. Log cached_tokens from every API response and sum them weekly. A system achieving 50 percent cached tokens across all API calls effectively halves its input token costs with zero application code changes needed for prefix caching.
from dataclasses import dataclass, field
from typing import ClassVar
@dataclass
class CachingMetrics:
total_prompt_tokens: int = 0
total_cached_tokens: int = 0
app_cache_hits: int = 0
app_cache_misses: int = 0
@property
def prefix_cache_ratio(self) -> float:
if self.total_prompt_tokens == 0:
return 0
return self.total_cached_tokens / self.total_prompt_tokens
@property
def app_cache_hit_rate(self) -> float:
total = self.app_cache_hits + self.app_cache_misses
return self.app_cache_hits / total if total > 0 else 0
def report(self):
print(f'App cache hit rate: {self.app_cache_hit_rate:.1%}')
print(f'Prefix cache ratio: {self.prefix_cache_ratio:.1%}')
savings_multiplier = (1 - self.app_cache_hit_rate) * (1 - 0.5 * self.prefix_cache_ratio)
print(f'Effective cost vs no-cache: {savings_multiplier:.1%}')When Prefix Caching Doesn't Help
Prefix caching provides no benefit for: (1) short prompts under 1024 tokens (minimum cacheable length), (2) highly variable prefixes where the system prompt changes per user or request, (3) low-traffic applications where the KV cache is evicted between requests, or (4) when you are already paying the minimum token rate. In these cases, focus optimization effort on application-level semantic caching instead.
Quick Check
Test your understanding of OpenAI prompt prefix caching from this lesson.
Lesson Recap
In this lesson you learned: OpenAI prompt prefix caching automatically discounts cached input tokens at 50 percent when the prompt prefix matches a recent prior request, stable content must come first in your message structure (system prompt, documents, then user query) to maximize cached tokens, and Anthropic requires explicit cache_control markers for a similar feature on Claude. Combine with application-level caching for maximum cost reduction. Next up we look at batching, model routing, and cost dashboards to complete our optimization toolkit.
Frequently asked questions
Is the “OpenAI Prompt Prefix Caching” lesson free?
Yes — the full text of “OpenAI Prompt Prefix Caching” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “OpenAI Prompt Prefix Caching”?
Leverage OpenAI's automatic prompt caching that discounts repeated long system prompt prefixes at 50 percent off, and structure your prompts to maximize cache hit rates. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “OpenAI Prompt Prefix Caching” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Exact Caching with Redis
- Semantic Caching with Embeddings
- OpenAI Prompt Prefix Caching
- Batching, Model Routing, and Cost Dashboards