Preventing Cache Stampedes and Thundering Herds
Mitigate stampedes with request coalescing, jittered TTLs, and probabilistic early expiration.
Preventing Cache Stampedes and Thundering Herds is a free Node.js Backend Development Bootcamp lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Node.js Backend Development Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is a Cache Stampede?
A cache stampede (also called a thundering herd) happens when a popular cached value expires and many concurrent requests miss the cache at the same instant. They all rush to the database to recompute the same value.
- One key expires → 5,000 in-flight requests all see a miss
- 5,000 identical queries hit the database simultaneously
- The DB saturates, latency spikes, and sometimes the whole system falls over
The irony: the cache exists to protect the database, but the moment of expiry becomes the most dangerous moment of all.
Seeing the Problem in Code
Here is a classic naive cache-aside read. It works fine under low traffic, but under heavy concurrency every miss spawns its own loadFromDb call.
Notice there is nothing stopping 1,000 callers from running loadFromDb at once for the same key.
async function getUser(redis, db, id) {
const key = 'user:' + id;
const cached = await redis.get(key);
if (cached !== null) {
return JSON.parse(cached);
}
// STAMPEDE RISK: every concurrent miss runs this
const user = await db.loadUser(id);
await redis.set(key, JSON.stringify(user), 'EX', 60);
return user;
}Strategy 1: Request Coalescing
Request coalescing (a.k.a. single-flight or in-flight de-duplication) means: when many callers want the same key at the same time, only one of them actually computes the value. The rest await the same promise.
- Keep an in-memory map of
key → pending promise - First caller starts the work and stores its promise
- Concurrent callers find the pending promise and await it
- When it settles, delete the entry
This collapses N database calls into 1 per process.
Implementing Single-Flight
A minimal, dependency-free single-flight helper. Every concurrent caller for the same key shares one underlying promise. The finally block clears the entry so the next round recomputes.
This snippet is fully self-contained and demonstrates the coalescing behavior with simulated slow work.
function createSingleFlight() {
const inFlight = new Map();
return function run(key, fn) {
if (inFlight.has(key)) {
return inFlight.get(key);
}
const p = Promise.resolve()
.then(fn)
.finally(() => inFlight.delete(key));
inFlight.set(key, p);
return p;
};
}
async function main() {
const flight = createSingleFlight();
let dbCalls = 0;
const load = () => new Promise(res => {
dbCalls++;
setTimeout(() => res('value-' + dbCalls), 50);
});
// 5 concurrent callers, same key
const results = await Promise.all(
Array.from({ length: 5 }, () => flight('user:42', load))
);
console.log('results:', results);
console.log('actual db calls:', dbCalls);
}
main();Coalescing's Limit: It's Per-Process
In-memory single-flight only de-duplicates within one Node.js process. If you run 20 instances behind a load balancer, you can still get up to 20 simultaneous DB calls per expired key — one per process.
- Great for cutting N requests → 1 inside each instance
- Not enough alone for large horizontal fleets
- For cross-process protection you need a distributed lock in Redis
Combine coalescing (cheap, local) with distributed locking or probabilistic expiration (cluster-wide) for full coverage.
Strategy 2: Distributed Lock
A distributed lock lets a single instance (across the whole fleet) win the right to recompute. Use Redis SET key value NX PX ttl: it sets the key only if it does not exist, atomically.
- Winner recomputes and repopulates the cache
- Losers either wait-and-retry the cache, or serve stale data
- Always set a TTL on the lock so a crashed winner cannot deadlock the key forever
This is the cross-process complement to in-process coalescing.
async function getWithLock(redis, db, id) {
const key = 'user:' + id;
const cached = await redis.get(key);
if (cached !== null) return JSON.parse(cached);
const lockKey = 'lock:' + key;
const token = Math.random().toString(36).slice(2);
// NX = only if absent, PX = lock TTL in ms
const won = await redis.set(lockKey, token, 'NX', 'PX', 5000);
if (won === 'OK') {
try {
const user = await db.loadUser(id);
await redis.set(key, JSON.stringify(user), 'EX', 60);
return user;
} finally {
// release only if we still own the lock
if (await redis.get(lockKey) === token) await redis.del(lockKey);
}
}
// Lost the race: briefly wait, then read the now-fresh cache
await new Promise(r => setTimeout(r, 50));
const retry = await redis.get(key);
return retry !== null ? JSON.parse(retry) : db.loadUser(id);
}Strategy 3: Jittered TTLs
If you warm 10,000 keys in a loop with the same TTL, they all expire at the same second — a synchronized stampede across many keys at once. TTL jitter spreads expirations out by adding a small random offset to each TTL.
- Base TTL 300s → actual TTL 270–330s, randomized per key
- Expirations scatter across a window instead of clustering on one tick
- Cheap, zero coordination, and it composes with every other strategy
Always jitter TTLs when you bulk-populate or refresh many related keys.
// Add +/- jitterPct random spread around a base TTL
function jitteredTtl(baseSeconds, jitterPct = 0.1) {
const spread = baseSeconds * jitterPct;
const offset = (Math.random() * 2 - 1) * spread; // -spread..+spread
return Math.max(1, Math.round(baseSeconds + offset));
}
// Demo: 5 keys warmed together get different lifetimes
for (let i = 0; i < 5; i++) {
console.log('key' + i + ' ttl =', jitteredTtl(300));
}Strategy 4: Probabilistic Early Expiration
Probabilistic early expiration (the XFetch algorithm) refreshes a key before it actually expires, with a probability that grows as expiry nears. So a single lucky request recomputes early while the old value is still served to everyone else.
The classic rule recomputes when:
now - delta * beta * ln(random()) ≥ expiry
Here delta is how long the last recompute took, and beta (default 1) tunes aggressiveness. Slow-to-compute keys (large delta) start refreshing earlier, which is exactly what you want.
XFetch in Code
To use XFetch you store, alongside the value, the recompute duration (delta) and the absolute expiry time. On each read you roll the probabilistic check. Most callers serve the cached value; occasionally one refreshes ahead of expiry.
This standalone demo shows that as we approach expiry, the early-refresh probability climbs toward 1.
function shouldRecompute(deltaMs, expiryMs, now, beta = 1) {
// XFetch: earlier refresh as we near expiry, scaled by recompute cost
const xfetch = now - deltaMs * beta * Math.log(Math.random());
return xfetch >= expiryMs;
}
const now = Date.now();
const delta = 200; // last recompute took 200ms
const expiry = now + 1000; // value expires in 1s
let refreshes = 0;
for (let i = 0; i < 1000; i++) {
// sample 'now' uniformly across the key's lifetime
const t = now + Math.random() * 1000;
if (shouldRecompute(delta, expiry, t)) refreshes++;
}
console.log('early refreshes out of 1000 reads:', refreshes);Combining the Strategies
These techniques are complementary layers, not competitors. A production cache read often stacks several:
- Coalescing — collapse duplicate work inside each process
- Distributed lock — one recompute across the whole fleet
- Jittered TTL — desynchronize bulk expirations
- Probabilistic early expiry — refresh hot keys before they ever miss
Start with jitter + coalescing (cheap, no coordination). Add a lock or XFetch for your hottest, most expensive keys.
Stale-While-Revalidate
A practical pattern that ties it together: stale-while-revalidate (SWR). Keep two lifetimes — a short fresh window and a longer stale window. While stale, immediately serve the old value and kick off a background refresh (de-duplicated by single-flight).
- Users almost never wait on a cold recompute
- The refresh runs once, off the request's critical path
- Pair with jitter so stale windows do not all end at once
SWR turns a hard miss (everyone waits) into a soft miss (one background refresh, everyone served instantly).
async function swrGet(redis, db, id, freshSec = 60, staleSec = 600) {
const key = 'user:' + id;
const raw = await redis.get(key);
if (raw) {
const { value, storedAt } = JSON.parse(raw);
const ageSec = (Date.now() - storedAt) / 1000;
if (ageSec > freshSec) {
// stale but usable: refresh in background, serve now
refreshInBackground(redis, db, id, key, staleSec);
}
return value;
}
return refreshInBackground(redis, db, id, key, staleSec);
}Quick Check
You run 30 Node.js instances behind a load balancer. A single extremely hot key expires and you must guarantee the database receives at most one recompute query for it. Which approach achieves this?
Recap
You learned how to defend against cache stampedes and thundering herds in Node.js:
- Request coalescing collapses concurrent duplicate work to one promise — but only per process.
- Distributed locks (
SET NX PX) extend that guarantee across the whole fleet; always give the lock a TTL. - Jittered TTLs desynchronize bulk expirations so many keys never expire on the same tick.
- Probabilistic early expiration (XFetch) refreshes hot, expensive keys before they ever miss.
- Stale-while-revalidate serves old data instantly and refreshes once in the background.
Layer them: jitter + coalescing as a baseline, then locks or XFetch for your hottest keys.
Frequently asked questions
Is the “Preventing Cache Stampedes and Thundering Herds” lesson free?
Yes — the full text of “Preventing Cache Stampedes and Thundering Herds” is free to read here on the web, and the Node.js Backend Development Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Node.js Backend Development Bootcamp course, upgrade to CoddyKit PRO.
What will I learn in “Preventing Cache Stampedes and Thundering Herds”?
Mitigate stampedes with request coalescing, jittered TTLs, and probabilistic early expiration. You practise Node.js Backend Development Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Node.js Backend Development Bootcamp?
No prior experience is required. Node.js Backend Development Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Preventing Cache Stampedes and Thundering Herds” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Node.js Backend Development Bootcamp lesson?
Yes. Every Node.js Backend Development Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Cache-Aside, Write-Through, and TTL Strategies
- Distributed Locks and the Redlock Algorithm
- Pub/Sub, Streams, and Rate Limiting with Redis
- Preventing Cache Stampedes and Thundering Herds