AI Engineering Academy · Lezione

Caching esatto con Redis

Caching esatto con Redis è una lezione AI Engineering Academy gratuita su CoddyKit. Questa è la lezione 1 di 4. Puoi leggere la lezione completa qui gratuitamente — poi esercitati direttamente nel browser con un editor di codice integrato e un tutor IA disponibile 24/7. Fa parte del percorso di apprendimento AI Engineering Academy, e i tuoi progressi si sincronizzano tra il web e l'app CoddyKit. Il corso AI Engineering Academy include 4 lezioni in totale.

Parti di questa lezione non sono ancora state tradotte e vengono mostrate in inglese.

Why Cache LLM Responses?

LLM API calls are expensive: a single GPT-4o request can cost $0.005-$0.15 depending on token count. In many applications, a significant fraction of incoming queries are identical or near-identical to previous ones — think FAQ bots, customer support systems, or code review tools where users ask the same questions repeatedly. Caching can eliminate 20-50 percent of API calls in these use cases, directly cutting costs and reducing latency.

Exact Cache: Cache Key Design

An exact cache stores LLM responses keyed by a deterministic hash of the input. The cache key must capture every input that affects the output: the messages array, the model name, temperature, and any other parameters that change the response. Missing any of these from the key causes cache collisions where a cached response is served for a different effective request.

import hashlib
import json

def make_cache_key(messages: list[dict], model: str, temperature: float) -> str:
    # Create a canonical, order-stable representation
    key_data = {
        'model': model,
        'temperature': temperature,
        'messages': messages,  # list order matters
    }
    # Serialize to JSON with sorted keys for determinism
    serialized = json.dumps(key_data, sort_keys=True, ensure_ascii=False)
    # Hash to a fixed-length key safe for Redis
    return 'llm_cache:' + hashlib.sha256(serialized.encode()).hexdigest()

Connecting to Redis

Redis is the standard choice for LLM response caching due to its sub-millisecond read latency and built-in TTL support. Use the redis-py library for synchronous access or aioredis (now merged into redis-py as redis.asyncio) for async access in FastAPI applications. Store the Redis connection as a singleton to avoid connection pool exhaustion.

import redis
import redis.asyncio as aioredis

# Synchronous Redis client
r = redis.Redis(
    host='localhost',
    port=6379,
    db=0,
    decode_responses=True,  # return str instead of bytes
)

# Async Redis client (for FastAPI)
async_r = aioredis.Redis(
    host='localhost',
    port=6379,
    db=0,
    decode_responses=True,
)

# Test connection
print(r.ping())  # True if Redis is running

Cache-Aside Pattern Implementation

The cache-aside pattern is the standard caching strategy for LLM APIs. On each request: (1) compute the cache key, (2) check Redis for a cached response, (3) if found (cache hit) return it immediately, (4) if not found (cache miss) call the LLM API, (5) store the response in Redis with a TTL, (6) return the response. This pattern keeps caching logic external to the LLM call itself.

import json
from openai import OpenAI

client = OpenAI()

def cached_completion(
    messages: list[dict],
    model: str = 'gpt-4o-mini',
    temperature: float = 0.7,
    ttl_seconds: int = 3600,
) -> str:
    cache_key = make_cache_key(messages, model, temperature)

    # Cache hit?
    cached = r.get(cache_key)
    if cached is not None:
        print('[CACHE HIT]')
        return json.loads(cached)

    # Cache miss: call API
    print('[CACHE MISS]')
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
    )
    result = response.choices[0].message.content

    # Store in cache with TTL
    r.setex(cache_key, ttl_seconds, json.dumps(result))
    return result

Async Cache-Aside for FastAPI

In an async FastAPI application, use the async Redis client so cache lookups do not block the event loop. The pattern is identical to the synchronous version but uses await for all Redis operations. This keeps the caching layer fully non-blocking and compatible with the async LLM client.

from openai import AsyncOpenAI
import redis.asyncio as aioredis
import json

async_client = AsyncOpenAI()
async_r = aioredis.Redis(host='localhost', port=6379, decode_responses=True)

async def async_cached_completion(
    messages: list[dict],
    model: str = 'gpt-4o-mini',
    temperature: float = 0.0,
    ttl: int = 86400,
) -> str:
    key = make_cache_key(messages, model, temperature)

    cached = await async_r.get(key)
    if cached:
        return json.loads(cached)

    response = await async_client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    result = response.choices[0].message.content
    await async_r.setex(key, ttl, json.dumps(result))
    return result

Choosing the Right TTL

The TTL (Time To Live) controls how long cached responses remain valid. For factual Q&A with stable knowledge bases, long TTLs (24-72 hours) maximize cache hit rates. For responses that should reflect the latest data (news summarization, live prices), short TTLs (5-15 minutes) or no caching at all are appropriate. For creative tasks with non-zero temperature, caching may produce stale responses — consider caching only for temperature=0.

# TTL strategy by use case
TTL_STRATEGY = {
    'faq_answering':          86400 * 7,   # 7 days — stable facts
    'code_explanation':       86400,        # 1 day — code rarely changes
    'document_summarization': 3600 * 6,    # 6 hours
    'news_analysis':          300,          # 5 minutes — stale quickly
    'creative_writing':       0,            # 0 = don't cache (non-deterministic)
}

def get_ttl_for_use_case(use_case: str) -> int:
    return TTL_STRATEGY.get(use_case, 3600)  # default 1 hour

Caching Metrics and Monitoring

Track cache hit rate as a primary cost-reduction metric. A cache hit rate of 30 percent means 30 percent of API calls are avoided. Store hit and miss counts in Redis itself using INCR commands on separate counters. Expose a /metrics endpoint in your FastAPI app that reports current hit rate, total requests, and estimated cost savings to quantify the ROI of caching.

CACHE_HITS_KEY = 'llm_cache_metrics:hits'
CACHE_MISSES_KEY = 'llm_cache_metrics:misses'

async def async_cached_completion_instrumented(messages, model, temperature=0.0):
    key = make_cache_key(messages, model, temperature)
    cached = await async_r.get(key)

    if cached:
        await async_r.incr(CACHE_HITS_KEY)
        return json.loads(cached)

    await async_r.incr(CACHE_MISSES_KEY)
    response = await async_client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    result = response.choices[0].message.content
    await async_r.setex(key, 3600, json.dumps(result))
    return result

async def get_cache_stats():
    hits = int(await async_r.get(CACHE_HITS_KEY) or 0)
    misses = int(await async_r.get(CACHE_MISSES_KEY) or 0)
    total = hits + misses
    return {'hit_rate': hits / total if total > 0 else 0, 'total': total}

Cache Invalidation Strategies

Exact cache invalidation is straightforward because keys are deterministic. To invalidate a specific entry, recompute its key and call r.delete(key). To invalidate all entries for a specific prompt pattern, use Redis key prefixes with a wildcard scan. To invalidate the entire cache on a major knowledge base update, call r.flushdb() (use with caution — this deletes all keys in the database).

async def invalidate_cache_entry(messages, model, temperature):
    key = make_cache_key(messages, model, temperature)
    deleted = await async_r.delete(key)
    print(f'Deleted {deleted} cache entries')

async def invalidate_all_llm_cache():
    # Scan for all keys with prefix 'llm_cache:'
    keys_to_delete = []
    async for key in async_r.scan_iter(match='llm_cache:*'):
        keys_to_delete.append(key)
    if keys_to_delete:
        await async_r.delete(*keys_to_delete)
    print(f'Invalidated {len(keys_to_delete)} cache entries')

Serializing Complex Responses

If your application caches entire API response objects (not just the text content), serialize them carefully. The full ChatCompletion object includes token usage, model version, and finish reason — useful for logging and cost tracking. Use the SDK's .model_dump_json() method to serialize Pydantic response objects to JSON strings, and reconstruct them with ChatCompletion.model_validate_json() on cache retrieval.

from openai.types.chat import ChatCompletion

async def cached_completion_full_response(
    messages, model='gpt-4o-mini', temperature=0.0
):
    key = make_cache_key(messages, model, temperature) + ':full'
    cached = await async_r.get(key)

    if cached:
        return ChatCompletion.model_validate_json(cached)  # reconstruct object

    response = await async_client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    # Serialize Pydantic model to JSON
    await async_r.setex(key, 3600, response.model_dump_json())
    return response

Caching and Non-Determinism

Exact caching only makes sense for deterministic or near-deterministic requests. At temperature=0 and top_p=1.0, most LLMs produce the same output for the same input (though not guaranteed due to floating-point non-determinism). At higher temperatures, cached responses become stale as the model would have produced different outputs. Always cache at temperature=0 or document clearly in your cache key that responses may vary.

Redis Cluster and Production Setup

For production deployments with high cache volumes, use Redis Cluster for horizontal sharding across multiple nodes, or a managed Redis service like AWS ElastiCache or Redis Cloud. Set a maxmemory policy (typically allkeys-lru to evict the least recently used entries when memory is full) to prevent Redis from running out of memory and automatically manage cache size.

# Redis configuration for production LLM caching
# In redis.conf:
# maxmemory 2gb
# maxmemory-policy allkeys-lru

# Connection with retry and connection pool
import redis
from redis.retry import Retry
from redis.backoff import ExponentialBackoff

retry = Retry(ExponentialBackoff(base=0.1), 3)
production_redis = redis.Redis(
    host='your-redis-host.cache.amazonaws.com',
    port=6379,
    ssl=True,
    decode_responses=True,
    max_connections=50,
    retry=retry,
    retry_on_error=[redis.ConnectionError, redis.TimeoutError],
)

Quick Check

Test your understanding of exact LLM response caching with Redis from this lesson.

Lesson Recap

In this lesson you learned: exact caching hashes all LLM inputs to produce a deterministic cache key, the cache-aside pattern checks Redis before calling the API and stores results after a miss, and TTL selection should reflect how frequently your content changes — longer for stable knowledge, shorter for dynamic data. Monitor cache hit rate as a primary cost-reduction metric. Next up we build semantic caching for similar but non-identical queries.

Domande Frequenti

La lezione «Caching esatto con Redis» è gratuita?

Sì — il testo completo di «Caching esatto con Redis» è gratuito qui sul web. Per esercitarvi in modo interattivo (un editor di codice integrato e un tutor IA 24/7) e sbloccare il resto del corso AI Engineering Academy, passa a CoddyKit PRO. Il corso AI Engineering Academy include 4 lezioni in totale.

Cosa imparerò in «Caching esatto con Redis»?

Memorizzi nella cache le risposte LLM calcolando l'hash del prompt completo e salvando il risultato in Redis con un TTL, così da servire istantaneamente le richieste identiche senza effettuare chiama… Eserciti AI Engineering Academy con codice pratico che esegui direttamente nel browser, e un tutor IA 24/7 risponde alle tue domande mentre lavori sulla lezione.

Ho bisogno di esperienza per iniziare AI Engineering Academy?

Non è richiesta alcuna esperienza precedente. AI Engineering Academy su CoddyKit è strutturato per principianti e studenti avanzati, quindi puoi iniziare da qui o dall'inizio e procedere al tuo ritmo. Questa è la lezione 1 di 4.

Quanto tempo richiede la lezione «Caching esatto con Redis»?

La maggior parte delle lezioni CoddyKit richiede circa 5–10 minuti. Ogni lezione è breve e interattiva, quindi fai progressi costanti e riprendi esattamente da dove hai lasciato su web e app.

Posso scrivere ed eseguire codice in questa lezione AI Engineering Academy?

Sì. Ogni lezione AI Engineering Academy include un editor di codice integrato, quindi scrivi ed esegui codice reale direttamente nel tuo browser e ricevi feedback istantaneo dall'IA — nessuna configurazione locale necessaria.

Tutte le lezioni di questo corso

Caching esatto con Redis
Caching semantico con gli embedding
Caching dei prefissi dei prompt con OpenAI
Batching, routing dei modelli e dashboard dei costi

← Torna a AI Engineering Academy