AI Engineering Academy · Ders

Redis ile Tam Önbellekleme

Tam istemi özetleyerek LLM yanıtlarını önbelleğe alın, sonucu Redis'te TTL ile saklayın ve API çağrısı yapmadan aynı istekleri anında sunun.

Redis ile Tam Önbellekleme, CoddyKit'te ücretsiz bir AI Engineering Academy dersidir. Bu, 4 dersinin 1. dersidir. Aşağıdan dersin tamamını ücretsiz okuyabilir, sonra tarayıcıda yerleşik kod editörü ve 7/24 yapay zeka koçu ile uygulamalı olarak pratik yapabilirsin. Bu, AI Engineering Academy öğrenme yolunun bir parçasıdır ve ilerlemeniz web ve CoddyKit uygulaması arasında senkronize olur. AI Engineering Academy kursu toplamda 4 dersten oluşur.

Bu dersin bazı bölümleri henüz çevrilmemiş olup İngilizce olarak gösterilmektedir.

Why Cache LLM Responses?

LLM API calls are expensive: a single GPT-4o request can cost $0.005-$0.15 depending on token count. In many applications, a significant fraction of incoming queries are identical or near-identical to previous ones — think FAQ bots, customer support systems, or code review tools where users ask the same questions repeatedly. Caching can eliminate 20-50 percent of API calls in these use cases, directly cutting costs and reducing latency.

Exact Cache: Cache Key Design

An exact cache stores LLM responses keyed by a deterministic hash of the input. The cache key must capture every input that affects the output: the messages array, the model name, temperature, and any other parameters that change the response. Missing any of these from the key causes cache collisions where a cached response is served for a different effective request.

import hashlib
import json

def make_cache_key(messages: list[dict], model: str, temperature: float) -> str:
    # Create a canonical, order-stable representation
    key_data = {
        'model': model,
        'temperature': temperature,
        'messages': messages,  # list order matters
    }
    # Serialize to JSON with sorted keys for determinism
    serialized = json.dumps(key_data, sort_keys=True, ensure_ascii=False)
    # Hash to a fixed-length key safe for Redis
    return 'llm_cache:' + hashlib.sha256(serialized.encode()).hexdigest()

Connecting to Redis

Redis is the standard choice for LLM response caching due to its sub-millisecond read latency and built-in TTL support. Use the redis-py library for synchronous access or aioredis (now merged into redis-py as redis.asyncio) for async access in FastAPI applications. Store the Redis connection as a singleton to avoid connection pool exhaustion.

import redis
import redis.asyncio as aioredis

# Synchronous Redis client
r = redis.Redis(
    host='localhost',
    port=6379,
    db=0,
    decode_responses=True,  # return str instead of bytes
)

# Async Redis client (for FastAPI)
async_r = aioredis.Redis(
    host='localhost',
    port=6379,
    db=0,
    decode_responses=True,
)

# Test connection
print(r.ping())  # True if Redis is running

Cache-Aside Pattern Implementation

The cache-aside pattern is the standard caching strategy for LLM APIs. On each request: (1) compute the cache key, (2) check Redis for a cached response, (3) if found (cache hit) return it immediately, (4) if not found (cache miss) call the LLM API, (5) store the response in Redis with a TTL, (6) return the response. This pattern keeps caching logic external to the LLM call itself.

import json
from openai import OpenAI

client = OpenAI()

def cached_completion(
    messages: list[dict],
    model: str = 'gpt-4o-mini',
    temperature: float = 0.7,
    ttl_seconds: int = 3600,
) -> str:
    cache_key = make_cache_key(messages, model, temperature)

    # Cache hit?
    cached = r.get(cache_key)
    if cached is not None:
        print('[CACHE HIT]')
        return json.loads(cached)

    # Cache miss: call API
    print('[CACHE MISS]')
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
    )
    result = response.choices[0].message.content

    # Store in cache with TTL
    r.setex(cache_key, ttl_seconds, json.dumps(result))
    return result

Async Cache-Aside for FastAPI

In an async FastAPI application, use the async Redis client so cache lookups do not block the event loop. The pattern is identical to the synchronous version but uses await for all Redis operations. This keeps the caching layer fully non-blocking and compatible with the async LLM client.

from openai import AsyncOpenAI
import redis.asyncio as aioredis
import json

async_client = AsyncOpenAI()
async_r = aioredis.Redis(host='localhost', port=6379, decode_responses=True)

async def async_cached_completion(
    messages: list[dict],
    model: str = 'gpt-4o-mini',
    temperature: float = 0.0,
    ttl: int = 86400,
) -> str:
    key = make_cache_key(messages, model, temperature)

    cached = await async_r.get(key)
    if cached:
        return json.loads(cached)

    response = await async_client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    result = response.choices[0].message.content
    await async_r.setex(key, ttl, json.dumps(result))
    return result

Choosing the Right TTL

The TTL (Time To Live) controls how long cached responses remain valid. For factual Q&A with stable knowledge bases, long TTLs (24-72 hours) maximize cache hit rates. For responses that should reflect the latest data (news summarization, live prices), short TTLs (5-15 minutes) or no caching at all are appropriate. For creative tasks with non-zero temperature, caching may produce stale responses — consider caching only for temperature=0.

# TTL strategy by use case
TTL_STRATEGY = {
    'faq_answering':          86400 * 7,   # 7 days — stable facts
    'code_explanation':       86400,        # 1 day — code rarely changes
    'document_summarization': 3600 * 6,    # 6 hours
    'news_analysis':          300,          # 5 minutes — stale quickly
    'creative_writing':       0,            # 0 = don't cache (non-deterministic)
}

def get_ttl_for_use_case(use_case: str) -> int:
    return TTL_STRATEGY.get(use_case, 3600)  # default 1 hour

Caching Metrics and Monitoring

Track cache hit rate as a primary cost-reduction metric. A cache hit rate of 30 percent means 30 percent of API calls are avoided. Store hit and miss counts in Redis itself using INCR commands on separate counters. Expose a /metrics endpoint in your FastAPI app that reports current hit rate, total requests, and estimated cost savings to quantify the ROI of caching.

CACHE_HITS_KEY = 'llm_cache_metrics:hits'
CACHE_MISSES_KEY = 'llm_cache_metrics:misses'

async def async_cached_completion_instrumented(messages, model, temperature=0.0):
    key = make_cache_key(messages, model, temperature)
    cached = await async_r.get(key)

    if cached:
        await async_r.incr(CACHE_HITS_KEY)
        return json.loads(cached)

    await async_r.incr(CACHE_MISSES_KEY)
    response = await async_client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    result = response.choices[0].message.content
    await async_r.setex(key, 3600, json.dumps(result))
    return result

async def get_cache_stats():
    hits = int(await async_r.get(CACHE_HITS_KEY) or 0)
    misses = int(await async_r.get(CACHE_MISSES_KEY) or 0)
    total = hits + misses
    return {'hit_rate': hits / total if total > 0 else 0, 'total': total}

Cache Invalidation Strategies

Exact cache invalidation is straightforward because keys are deterministic. To invalidate a specific entry, recompute its key and call r.delete(key). To invalidate all entries for a specific prompt pattern, use Redis key prefixes with a wildcard scan. To invalidate the entire cache on a major knowledge base update, call r.flushdb() (use with caution — this deletes all keys in the database).

async def invalidate_cache_entry(messages, model, temperature):
    key = make_cache_key(messages, model, temperature)
    deleted = await async_r.delete(key)
    print(f'Deleted {deleted} cache entries')

async def invalidate_all_llm_cache():
    # Scan for all keys with prefix 'llm_cache:'
    keys_to_delete = []
    async for key in async_r.scan_iter(match='llm_cache:*'):
        keys_to_delete.append(key)
    if keys_to_delete:
        await async_r.delete(*keys_to_delete)
    print(f'Invalidated {len(keys_to_delete)} cache entries')

Serializing Complex Responses

If your application caches entire API response objects (not just the text content), serialize them carefully. The full ChatCompletion object includes token usage, model version, and finish reason — useful for logging and cost tracking. Use the SDK's .model_dump_json() method to serialize Pydantic response objects to JSON strings, and reconstruct them with ChatCompletion.model_validate_json() on cache retrieval.

from openai.types.chat import ChatCompletion

async def cached_completion_full_response(
    messages, model='gpt-4o-mini', temperature=0.0
):
    key = make_cache_key(messages, model, temperature) + ':full'
    cached = await async_r.get(key)

    if cached:
        return ChatCompletion.model_validate_json(cached)  # reconstruct object

    response = await async_client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    # Serialize Pydantic model to JSON
    await async_r.setex(key, 3600, response.model_dump_json())
    return response

Caching and Non-Determinism

Exact caching only makes sense for deterministic or near-deterministic requests. At temperature=0 and top_p=1.0, most LLMs produce the same output for the same input (though not guaranteed due to floating-point non-determinism). At higher temperatures, cached responses become stale as the model would have produced different outputs. Always cache at temperature=0 or document clearly in your cache key that responses may vary.

Redis Cluster and Production Setup

For production deployments with high cache volumes, use Redis Cluster for horizontal sharding across multiple nodes, or a managed Redis service like AWS ElastiCache or Redis Cloud. Set a maxmemory policy (typically allkeys-lru to evict the least recently used entries when memory is full) to prevent Redis from running out of memory and automatically manage cache size.

# Redis configuration for production LLM caching
# In redis.conf:
# maxmemory 2gb
# maxmemory-policy allkeys-lru

# Connection with retry and connection pool
import redis
from redis.retry import Retry
from redis.backoff import ExponentialBackoff

retry = Retry(ExponentialBackoff(base=0.1), 3)
production_redis = redis.Redis(
    host='your-redis-host.cache.amazonaws.com',
    port=6379,
    ssl=True,
    decode_responses=True,
    max_connections=50,
    retry=retry,
    retry_on_error=[redis.ConnectionError, redis.TimeoutError],
)

Quick Check

Test your understanding of exact LLM response caching with Redis from this lesson.

Lesson Recap

In this lesson you learned: exact caching hashes all LLM inputs to produce a deterministic cache key, the cache-aside pattern checks Redis before calling the API and stores results after a miss, and TTL selection should reflect how frequently your content changes — longer for stable knowledge, shorter for dynamic data. Monitor cache hit rate as a primary cost-reduction metric. Next up we build semantic caching for similar but non-identical queries.

Sıkça Sorulan Sorular

“Redis ile Tam Önbellekleme” dersi ücretsiz mi?

Evet — “Redis ile Tam Önbellekleme” dersin tüm metni burada web'de ücretsiz olarak okunabilir. Etkileşimli olarak pratik yapmak (yerleşik kod editörü ve 7/24 yapay zeka koçu) ve AI Engineering Academy kursunun geri kalanını açmak için CoddyKit PRO'ya yükselt. AI Engineering Academy kursu toplamda 4 dersten oluşur.

“Redis ile Tam Önbellekleme” dersinde ne öğreneceğim?

Tam istemi özetleyerek LLM yanıtlarını önbelleğe alın, sonucu Redis'te TTL ile saklayın ve API çağrısı yapmadan aynı istekleri anında sunun. AI Engineering Academy ile uygulamalı kodu tarayıcıda doğrudan çalıştırarak pratik yaparsın ve 7/24 yapay zeka koçu dersi çalışırken sorularını yanıtlar.

AI Engineering Academy öğrenmeye başlamak için deneyim gerekli mi?

Önceden deneyim gerekmez. CoddyKit'te AI Engineering Academy, başlangıçtan ileri seviyeye kadar yapılandırıldığı için buradan başlayabilir veya başından başlayıp kendi hızında ilerleme yapabilirsin. Bu, 4 dersinin 1. dersidir.

“Redis ile Tam Önbellekleme” dersi ne kadar sürer?

Çoğu CoddyKit dersi yaklaşık 5–10 dakika sürer. Her biri kısa ve etkileşimli olduğu için sabit ilerleme yaparsın ve web ile uygulama arasında tam olarak bıraktığın yerden devam edebilirsin.

Bu AI Engineering Academy dersinde kod yazıp çalıştırabilir miyim?

Evet. Her AI Engineering Academy dersi yerleşik bir kod editörü içerir, bu sayede tarayıcıda gerçek kod yazıp çalıştırabilir ve anlık yapay zeka geri bildirimi alırsın — yerel kurulum gerekli değildir.

Bu kursun tüm dersleri

Redis ile Tam Önbellekleme
Gömme Tabanlı Anlamsal Önbellekleme
OpenAI İstem Öneki Önbellekleme
Toplu İşleme, Model Yönlendirme ve Maliyet Panoları

← AI Engineering Academy Sayfasına Dön