Semantic Caching for LLM Responses
Learn how semantic caching reuses answers for similar queries by matching on meaning rather than exact text, cutting LLM cost and latency dramatically.
Semantic Caching for LLM Responses is a free LLM Apps in Production (RAG + Vector DB + Caching) lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the LLM Apps in Production (RAG + Vector DB + Caching) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Beyond Exact-Match Caching
A normal cache only hits when the key is byte-identical. But 'What is your refund policy?' and 'How do refunds work?' mean the same thing yet miss an exact cache.
Semantic caching matches on meaning, so paraphrases reuse the same answer.
How It Works
The flow:
- Embed the incoming query into a vector
- Search the cache for a near-by stored query
- If similarity exceeds a threshold, return the cached answer
- Otherwise call the LLM and store the new pair
Embedding the Query
Each query is converted to a vector by an embedding model. Similar meanings produce nearby vectors.
def embed(text):
return [len(text), text.count('refund'), text.count('?')]
print(embed('How do refunds work?'))Cosine Similarity
Similarity between query vectors is usually measured with cosine similarity.
import math
def cosine(a, b):
dot = sum(x*y for x, y in zip(a, b))
na = math.sqrt(sum(x*x for x in a))
nb = math.sqrt(sum(y*y for y in b))
return dot / (na * nb)
print(round(cosine([1,2,1],[1,2,0]), 3))Choosing the Threshold
The similarity threshold is the key tuning knob:
- Too low -> false hits, wrong answers served
- Too high -> few hits, little savings
Tune it on real traffic and err conservative for high-stakes domains.
A Minimal Semantic Cache
Putting embedding, similarity, and a threshold together.
cache = []
THRESH = 0.95
def get(query, qvec):
for stored_vec, ans in cache:
if cosine(qvec, stored_vec) >= THRESH:
return ans
return None
def cosine(a, b):
return 1.0 if a == b else 0.0
cache.append(([1,0], 'Refunds take 5 days'))
print(get('q', [1,0]))When NOT to Cache
Semantic caching is wrong for queries whose answer depends on changing or personal state:
- 'What is my account balance?'
- 'What is today's weather?'
- Anything user-specific or time-sensitive
Cache only stable, general knowledge.
Scoping the Cache
To avoid leaking one user's data to another, scope cache keys by tenant, language, and any relevant context. A global cache for personalized answers is a privacy bug.
Eviction and Freshness
Cached answers go stale when source data changes. Add TTLs and invalidate entries when underlying documents update, so the cache does not serve outdated answers.
Measuring Savings
Track hit rate, cost saved, and latency improvement. A 40 percent semantic hit rate can roughly translate into a 40 percent reduction in LLM spend for cacheable traffic.
Production Stack
In production, store query embeddings in a vector DB or Redis with vector search, set a tuned threshold, scope by tenant, apply TTLs, and monitor hit rate. Combine with exact caching for the best coverage.
Quick Check
Test your understanding of semantic caching.
Recap
You learned that semantic caching reuses answers for paraphrased queries by embedding them and matching via cosine similarity above a tuned threshold. Cache only stable knowledge, scope by tenant for privacy, apply TTLs for freshness, and monitor hit rate to quantify savings.
Frequently asked questions
Is the “Semantic Caching for LLM Responses” lesson free?
Yes — the full text of “Semantic Caching for LLM Responses” is free to read here on the web, and the LLM Apps in Production (RAG + Vector DB + Caching) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the LLM Apps in Production (RAG + Vector DB + Caching) course, upgrade to CoddyKit PRO.
What will I learn in “Semantic Caching for LLM Responses”?
Learn how semantic caching reuses answers for similar queries by matching on meaning rather than exact text, cutting LLM cost and latency dramatically. You practise LLM Apps in Production (RAG + Vector DB + Caching) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start LLM Apps in Production (RAG + Vector DB + Caching)?
No prior experience is required. LLM Apps in Production (RAG + Vector DB + Caching) on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Semantic Caching for LLM Responses” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this LLM Apps in Production (RAG + Vector DB + Caching) lesson?
Yes. Every LLM Apps in Production (RAG + Vector DB + Caching) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Distributed Caching with Redis/Memcached
- Session Management and Context Persistence
- Advanced Cache Invalidation Strategies
- Semantic Caching for LLM Responses