Exact Caching with Redis
Cache LLM responses by hashing the complete prompt and storing the result in Redis with a TTL, serving identical requests instantly without any API call.
Exact Caching with Redis is a free AI Engineering Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Cache LLM Responses?
LLM API calls are expensive: a single GPT-4o request can cost $0.005-$0.15 depending on token count. In many applications, a significant fraction of incoming queries are identical or near-identical to previous ones — think FAQ bots, customer support systems, or code review tools where users ask the same questions repeatedly. Caching can eliminate 20-50 percent of API calls in these use cases, directly cutting costs and reducing latency.
Exact Cache: Cache Key Design
An exact cache stores LLM responses keyed by a deterministic hash of the input. The cache key must capture every input that affects the output: the messages array, the model name, temperature, and any other parameters that change the response. Missing any of these from the key causes cache collisions where a cached response is served for a different effective request.
import hashlib
import json
def make_cache_key(messages: list[dict], model: str, temperature: float) -> str:
# Create a canonical, order-stable representation
key_data = {
'model': model,
'temperature': temperature,
'messages': messages, # list order matters
}
# Serialize to JSON with sorted keys for determinism
serialized = json.dumps(key_data, sort_keys=True, ensure_ascii=False)
# Hash to a fixed-length key safe for Redis
return 'llm_cache:' + hashlib.sha256(serialized.encode()).hexdigest()Connecting to Redis
Redis is the standard choice for LLM response caching due to its sub-millisecond read latency and built-in TTL support. Use the redis-py library for synchronous access or aioredis (now merged into redis-py as redis.asyncio) for async access in FastAPI applications. Store the Redis connection as a singleton to avoid connection pool exhaustion.
import redis
import redis.asyncio as aioredis
# Synchronous Redis client
r = redis.Redis(
host='localhost',
port=6379,
db=0,
decode_responses=True, # return str instead of bytes
)
# Async Redis client (for FastAPI)
async_r = aioredis.Redis(
host='localhost',
port=6379,
db=0,
decode_responses=True,
)
# Test connection
print(r.ping()) # True if Redis is runningCache-Aside Pattern Implementation
The cache-aside pattern is the standard caching strategy for LLM APIs. On each request: (1) compute the cache key, (2) check Redis for a cached response, (3) if found (cache hit) return it immediately, (4) if not found (cache miss) call the LLM API, (5) store the response in Redis with a TTL, (6) return the response. This pattern keeps caching logic external to the LLM call itself.
import json
from openai import OpenAI
client = OpenAI()
def cached_completion(
messages: list[dict],
model: str = 'gpt-4o-mini',
temperature: float = 0.7,
ttl_seconds: int = 3600,
) -> str:
cache_key = make_cache_key(messages, model, temperature)
# Cache hit?
cached = r.get(cache_key)
if cached is not None:
print('[CACHE HIT]')
return json.loads(cached)
# Cache miss: call API
print('[CACHE MISS]')
response = client.chat.completions.create(
model=model,
messages=messages,
temperature=temperature,
)
result = response.choices[0].message.content
# Store in cache with TTL
r.setex(cache_key, ttl_seconds, json.dumps(result))
return resultAsync Cache-Aside for FastAPI
In an async FastAPI application, use the async Redis client so cache lookups do not block the event loop. The pattern is identical to the synchronous version but uses await for all Redis operations. This keeps the caching layer fully non-blocking and compatible with the async LLM client.
from openai import AsyncOpenAI
import redis.asyncio as aioredis
import json
async_client = AsyncOpenAI()
async_r = aioredis.Redis(host='localhost', port=6379, decode_responses=True)
async def async_cached_completion(
messages: list[dict],
model: str = 'gpt-4o-mini',
temperature: float = 0.0,
ttl: int = 86400,
) -> str:
key = make_cache_key(messages, model, temperature)
cached = await async_r.get(key)
if cached:
return json.loads(cached)
response = await async_client.chat.completions.create(
model=model, messages=messages, temperature=temperature
)
result = response.choices[0].message.content
await async_r.setex(key, ttl, json.dumps(result))
return resultChoosing the Right TTL
The TTL (Time To Live) controls how long cached responses remain valid. For factual Q&A with stable knowledge bases, long TTLs (24-72 hours) maximize cache hit rates. For responses that should reflect the latest data (news summarization, live prices), short TTLs (5-15 minutes) or no caching at all are appropriate. For creative tasks with non-zero temperature, caching may produce stale responses — consider caching only for temperature=0.
# TTL strategy by use case
TTL_STRATEGY = {
'faq_answering': 86400 * 7, # 7 days — stable facts
'code_explanation': 86400, # 1 day — code rarely changes
'document_summarization': 3600 * 6, # 6 hours
'news_analysis': 300, # 5 minutes — stale quickly
'creative_writing': 0, # 0 = don't cache (non-deterministic)
}
def get_ttl_for_use_case(use_case: str) -> int:
return TTL_STRATEGY.get(use_case, 3600) # default 1 hourCaching Metrics and Monitoring
Track cache hit rate as a primary cost-reduction metric. A cache hit rate of 30 percent means 30 percent of API calls are avoided. Store hit and miss counts in Redis itself using INCR commands on separate counters. Expose a /metrics endpoint in your FastAPI app that reports current hit rate, total requests, and estimated cost savings to quantify the ROI of caching.
CACHE_HITS_KEY = 'llm_cache_metrics:hits'
CACHE_MISSES_KEY = 'llm_cache_metrics:misses'
async def async_cached_completion_instrumented(messages, model, temperature=0.0):
key = make_cache_key(messages, model, temperature)
cached = await async_r.get(key)
if cached:
await async_r.incr(CACHE_HITS_KEY)
return json.loads(cached)
await async_r.incr(CACHE_MISSES_KEY)
response = await async_client.chat.completions.create(
model=model, messages=messages, temperature=temperature
)
result = response.choices[0].message.content
await async_r.setex(key, 3600, json.dumps(result))
return result
async def get_cache_stats():
hits = int(await async_r.get(CACHE_HITS_KEY) or 0)
misses = int(await async_r.get(CACHE_MISSES_KEY) or 0)
total = hits + misses
return {'hit_rate': hits / total if total > 0 else 0, 'total': total}Cache Invalidation Strategies
Exact cache invalidation is straightforward because keys are deterministic. To invalidate a specific entry, recompute its key and call r.delete(key). To invalidate all entries for a specific prompt pattern, use Redis key prefixes with a wildcard scan. To invalidate the entire cache on a major knowledge base update, call r.flushdb() (use with caution — this deletes all keys in the database).
async def invalidate_cache_entry(messages, model, temperature):
key = make_cache_key(messages, model, temperature)
deleted = await async_r.delete(key)
print(f'Deleted {deleted} cache entries')
async def invalidate_all_llm_cache():
# Scan for all keys with prefix 'llm_cache:'
keys_to_delete = []
async for key in async_r.scan_iter(match='llm_cache:*'):
keys_to_delete.append(key)
if keys_to_delete:
await async_r.delete(*keys_to_delete)
print(f'Invalidated {len(keys_to_delete)} cache entries')Serializing Complex Responses
If your application caches entire API response objects (not just the text content), serialize them carefully. The full ChatCompletion object includes token usage, model version, and finish reason — useful for logging and cost tracking. Use the SDK's .model_dump_json() method to serialize Pydantic response objects to JSON strings, and reconstruct them with ChatCompletion.model_validate_json() on cache retrieval.
from openai.types.chat import ChatCompletion
async def cached_completion_full_response(
messages, model='gpt-4o-mini', temperature=0.0
):
key = make_cache_key(messages, model, temperature) + ':full'
cached = await async_r.get(key)
if cached:
return ChatCompletion.model_validate_json(cached) # reconstruct object
response = await async_client.chat.completions.create(
model=model, messages=messages, temperature=temperature
)
# Serialize Pydantic model to JSON
await async_r.setex(key, 3600, response.model_dump_json())
return responseCaching and Non-Determinism
Exact caching only makes sense for deterministic or near-deterministic requests. At temperature=0 and top_p=1.0, most LLMs produce the same output for the same input (though not guaranteed due to floating-point non-determinism). At higher temperatures, cached responses become stale as the model would have produced different outputs. Always cache at temperature=0 or document clearly in your cache key that responses may vary.
Redis Cluster and Production Setup
For production deployments with high cache volumes, use Redis Cluster for horizontal sharding across multiple nodes, or a managed Redis service like AWS ElastiCache or Redis Cloud. Set a maxmemory policy (typically allkeys-lru to evict the least recently used entries when memory is full) to prevent Redis from running out of memory and automatically manage cache size.
# Redis configuration for production LLM caching
# In redis.conf:
# maxmemory 2gb
# maxmemory-policy allkeys-lru
# Connection with retry and connection pool
import redis
from redis.retry import Retry
from redis.backoff import ExponentialBackoff
retry = Retry(ExponentialBackoff(base=0.1), 3)
production_redis = redis.Redis(
host='your-redis-host.cache.amazonaws.com',
port=6379,
ssl=True,
decode_responses=True,
max_connections=50,
retry=retry,
retry_on_error=[redis.ConnectionError, redis.TimeoutError],
)Quick Check
Test your understanding of exact LLM response caching with Redis from this lesson.
Lesson Recap
In this lesson you learned: exact caching hashes all LLM inputs to produce a deterministic cache key, the cache-aside pattern checks Redis before calling the API and stores results after a miss, and TTL selection should reflect how frequently your content changes — longer for stable knowledge, shorter for dynamic data. Monitor cache hit rate as a primary cost-reduction metric. Next up we build semantic caching for similar but non-identical queries.
Frequently asked questions
Is the “Exact Caching with Redis” lesson free?
Yes — the full text of “Exact Caching with Redis” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Exact Caching with Redis”?
Cache LLM responses by hashing the complete prompt and storing the result in Redis with a TTL, serving identical requests instantly without any API call. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Exact Caching with Redis” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.