0Pricing
AI Engineering Academy · 课时

OpenAI 提示前缀缓存

利用 OpenAI 的自动提示缓存功能，让重复出现的长系统提示前缀享受五折优惠，并组织提示以最大限度提高缓存命中率。

OpenAI 提示前缀缓存 是 CoddyKit 上的免费 AI Engineering Academy 课时。 这是第 3 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Engineering Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Engineering Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What Is Prompt Prefix Caching?

Prompt prefix caching is a server-side optimization built into OpenAI's API that automatically discounts tokens in the prompt prefix that were seen in a recent prior request. Unlike application-level caching that returns a stored response, prompt prefix caching still calls the model — but at a 50 percent reduced input token price for the cached prefix portion. It reduces cost without sacrificing fresh generation.

How Prefix Caching Works Under the Hood

Modern LLMs represent prompts as KV (key-value) caches in GPU memory. Processing a prompt means computing attention keys and values for every token. If the first N tokens of two consecutive requests are identical, OpenAI can reuse the KV cache from the first request, skipping the expensive compute for those tokens. The API does this automatically and transparently — you just pay the lower cached token rate when it applies.

# No code changes needed to enable prefix caching!
# It is automatic on supported models.

# The API response shows you how many tokens were cached:
# response.usage.prompt_tokens_details.cached_tokens

# Example response usage:
# ChatCompletionUsage(
#   prompt_tokens=2048,
#   completion_tokens=256,
#   total_tokens=2304,
#   prompt_tokens_details=PromptTokensDetails(
#     cached_tokens=1984,   # these tokens were served from KV cache
#     audio_tokens=0,
#   )
# )

Checking Cache Hit in the Response

After each API call, inspect response.usage.prompt_tokens_details.cached_tokens to see how many input tokens were served from the KV cache. If cached_tokens > 0, you paid the 50 percent discount rate for those tokens. Logging this value lets you track your actual cache efficiency and compute the cost savings from prefix caching over time.

from openai import OpenAI

client = OpenAI()

SYSTEM_PROMPT = 'You are an expert AI engineer assistant. ' * 100  # long system prompt

def call_with_cache_check(user_message: str):
    response = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': user_message},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens if usage.prompt_tokens_details else 0
    print(f'Total prompt tokens: {usage.prompt_tokens}')
    print(f'Cached tokens: {cached} ({100*cached//usage.prompt_tokens}%)')
    return response.choices[0].message.content

The Prefix Must Be Exactly Identical

Prefix caching only applies when the first N tokens are byte-for-byte identical to a recent prior request. Even a single character change in the system prompt invalidates the cache. OpenAI caches in 128-token chunks — the cache applies to complete chunks that match exactly. This means the portion of your prompt that varies per request should come after the long, stable prefix to maximize cached tokens.

# Optimal structure for prefix caching:
# [LONG STABLE SYSTEM PROMPT] [CACHED DOCUMENTS] [USER QUERY]
#         ↑                              ↑                ↑
#    always same               always same         varies per request
#    → cached at 50%           → cached at 50%     → not cached, full price

# BAD structure (defeats prefix caching):
# [USER QUERY] [CACHED DOCUMENTS] [LONG STABLE SYSTEM PROMPT]
#       ↑                                       ↑
#  changes every request                 never cached because
#  so prefix never matches               it comes after the query

Structuring Prompts for Maximum Cache Efficiency

To maximize cache hit rates, structure your prompts so the stable portions come first. In a RAG system: (1) system prompt with instructions and persona, (2) retrieved documents that change only when the query changes significantly, (3) conversation history, (4) user query at the very end. The system prompt alone — often 500-2000 tokens — will typically be cached, saving 25-50 percent of input token costs.

def build_rag_prompt_for_caching(
    system_prompt: str,
    retrieved_docs: list[str],
    conversation_history: list[dict],
    user_query: str,
) -> list[dict]:
    # Order: stable → semi-stable → variable
    context_block = '\n\n'.join(
        f'[Document {i+1}]\n{doc}' for i, doc in enumerate(retrieved_docs)
    )
    return [
        # 1. Stable system prompt (always cached after first request)
        {'role': 'system', 'content': system_prompt},
        # 2. Context injection as a user message (cached when same docs retrieved)
        {'role': 'user', 'content': f'Context documents:\n{context_block}'},
        {'role': 'assistant', 'content': 'I have read the documents.'},
        # 3. Conversation history (semi-stable)
        *conversation_history,
        # 4. Current user query (always different → never cached prefix)
        {'role': 'user', 'content': user_query},
    ]

Cache Duration and Eviction

OpenAI's KV cache is maintained in-memory on GPU and has an eviction policy. Prefixes that have not been reused within approximately 5-10 minutes are evicted as other requests take up GPU memory. This means prefix caching benefits are largest for high-throughput applications with frequent requests sharing the same prefix. Low-traffic applications may see few cache hits because the prefix is evicted between spaced-out requests.

Supported Models and Pricing

As of 2025, prompt prefix caching is available on GPT-4o, GPT-4o-mini, o1, and o3-mini models. The cached token price is 50 percent of the standard input token price for most models. The minimum cacheable prefix length is 1024 tokens — shorter prefixes receive no discount. Always check the OpenAI pricing page for the current rates, as pricing evolves as the feature matures.

# Rough pricing reference (verify at platform.openai.com/pricing)
PRICING = {
    'gpt-4o': {
        'input_per_1M': 2.50,
        'cached_input_per_1M': 1.25,   # 50% off
        'output_per_1M': 10.00,
    },
    'gpt-4o-mini': {
        'input_per_1M': 0.15,
        'cached_input_per_1M': 0.075,  # 50% off
        'output_per_1M': 0.60,
    },
}

def estimate_cost_with_caching(prompt_tokens, cached_tokens, output_tokens, model):
    p = PRICING[model]
    uncached = (prompt_tokens - cached_tokens) * p['input_per_1M'] / 1_000_000
    cached_cost = cached_tokens * p['cached_input_per_1M'] / 1_000_000
    output_cost = output_tokens * p['output_per_1M'] / 1_000_000
    return uncached + cached_cost + output_cost

Anthropic Prompt Caching

Anthropic offers a similar feature called prompt caching for Claude models, but requires explicit opt-in by marking cache breakpoints in the prompt with a cache_control field. Unlike OpenAI's automatic caching, you explicitly mark which portions of the prompt should be cached (up to 4 cache breakpoints per request). Cached tokens cost 10 percent of the standard input price and are stored for 5 minutes.

import anthropic

client = anthropic.Anthropic()

LONG_DOCUMENT = 'This is a very long reference document...' * 500  # 2000+ tokens

response = client.messages.create(
    model='claude-sonnet-4-5',
    max_tokens=1024,
    system=[
        {
            'type': 'text',
            'text': 'You are a helpful assistant.',
        },
        {
            'type': 'text',
            'text': LONG_DOCUMENT,
            'cache_control': {'type': 'ephemeral'},  # mark for caching
        }
    ],
    messages=[{'role': 'user', 'content': 'Summarize the document.'}],
)
print(response.usage.cache_read_input_tokens)   # tokens served from cache
print(response.usage.cache_creation_input_tokens)  # tokens written to cache

Combining Prefix Caching with Application Caching

Prefix caching and application-level caching are complementary. Prefix caching reduces the cost of each API call but still calls the LLM. Application-level exact and semantic caches eliminate API calls entirely for repeated queries. Use prefix caching for all requests to reduce per-call input cost, and layer application-level caching on top to eliminate calls entirely for frequently repeated queries. Together, they can reduce AI infrastructure costs by 60-80 percent.

# Three-layer cost optimization stack
#
# Layer 1: Exact cache (Redis, hash-based)
#   → Eliminates 100% of API cost for identical requests
#   → Miss rate: ~60-80% (most queries are unique)
#
# Layer 2: Semantic cache (vector similarity)
#   → Eliminates 100% of API cost for semantically similar requests
#   → Miss rate: ~40-60% of remaining queries
#
# Layer 3: OpenAI prefix caching (automatic)
#   → Reduces input token cost by 50% for long stable prefixes
#   → Applies to ALL remaining API calls that escape layers 1 and 2
#
# Combined effect: 60-80% cost reduction in FAQ/support applications

Measuring Your Cache Efficiency

Track cache efficiency ratio as a composite metric: total tokens at full price divided by total tokens actually billed. This accounts for all caching layers. Log cached_tokens from every API response and sum them weekly. A system achieving 50 percent cached tokens across all API calls effectively halves its input token costs with zero application code changes needed for prefix caching.

from dataclasses import dataclass, field
from typing import ClassVar

@dataclass
class CachingMetrics:
    total_prompt_tokens: int = 0
    total_cached_tokens: int = 0
    app_cache_hits: int = 0
    app_cache_misses: int = 0

    @property
    def prefix_cache_ratio(self) -> float:
        if self.total_prompt_tokens == 0:
            return 0
        return self.total_cached_tokens / self.total_prompt_tokens

    @property
    def app_cache_hit_rate(self) -> float:
        total = self.app_cache_hits + self.app_cache_misses
        return self.app_cache_hits / total if total > 0 else 0

    def report(self):
        print(f'App cache hit rate: {self.app_cache_hit_rate:.1%}')
        print(f'Prefix cache ratio: {self.prefix_cache_ratio:.1%}')
        savings_multiplier = (1 - self.app_cache_hit_rate) * (1 - 0.5 * self.prefix_cache_ratio)
        print(f'Effective cost vs no-cache: {savings_multiplier:.1%}')

When Prefix Caching Doesn't Help

Prefix caching provides no benefit for: (1) short prompts under 1024 tokens (minimum cacheable length), (2) highly variable prefixes where the system prompt changes per user or request, (3) low-traffic applications where the KV cache is evicted between requests, or (4) when you are already paying the minimum token rate. In these cases, focus optimization effort on application-level semantic caching instead.

Quick Check

Test your understanding of OpenAI prompt prefix caching from this lesson.

Lesson Recap

In this lesson you learned: OpenAI prompt prefix caching automatically discounts cached input tokens at 50 percent when the prompt prefix matches a recent prior request, stable content must come first in your message structure (system prompt, documents, then user query) to maximize cached tokens, and Anthropic requires explicit cache_control markers for a similar feature on Claude. Combine with application-level caching for maximum cost reduction. Next up we look at batching, model routing, and cost dashboards to complete our optimization toolkit.

常见问题解答

「OpenAI 提示前缀缓存」课时是免费的吗？

是的 — 「OpenAI 提示前缀缓存」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Engineering Academy 课程的其余内容，请升级到 CoddyKit PRO。 AI Engineering Academy 课程共包含 4 节课。

「OpenAI 提示前缀缓存」这节课中我会学到什么？

利用 OpenAI 的自动提示缓存功能，让重复出现的长系统提示前缀享受五折优惠，并组织提示以最大限度提高缓存命中率。 你通过在浏览器中直接运行的动手代码来练习 AI Engineering Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 AI Engineering Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 AI Engineering Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 3 节课，共 4 节。

「OpenAI 提示前缀缓存」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 AI Engineering Academy 课中编写并运行代码吗？

能。每节 AI Engineering Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 使用 Redis 实现精确缓存
  2. 使用嵌入实现语义缓存
  3. OpenAI 提示前缀缓存
  4. 批处理、模型路由与成本仪表板
← 返回 AI Engineering Academy