Thundering Herd & Cache Stampede

When a popular cache entry expires, every concurrent request may miss at once and hammer the database — the thundering herd problem. At scale this can take down your database in seconds. A related pattern, cache stampede, happens when many requests simultaneously try to regenerate the same expired entry.

Solutions exist at every level. Locking (or mutex): only one request regenerates the value; others wait or get the stale value. Request coalescing (singleflight): duplicate in-flight requests for the same key share one backend call. Jittered TTLs: instead of every key expiring at exactly 300 seconds, expire between 270–330 seconds so expirations spread over time. Probabilistic early expiration: each request has a small chance of refreshing the cache before TTL expires, spreading regeneration load gradually.

Before

Stampede — 1000 concurrent misses hit DB

1// Cache expires at exactly 12:00:002// 1000 requests arrive at 12:00:013// All 1000 query the database simultaneously4async function getProduct(id) {5  const cached = await redis.get(`product:${id}`);6  if (cached) return JSON.parse(cached);7 8  // No lock — every miss hits DB9  const product = await db.query('SELECT ...', [id]);10  await redis.setex(`product:${id}`, 300, JSON.stringify(product));11  return product;12}

After

Singleflight — one DB call, others wait

1const inflight = new Map();2 3async function getProduct(id) {4  const cached = await redis.get(`product:${id}`);5  if (cached) return JSON.parse(cached);6 7  const key = `product:${id}`;8  if (inflight.has(key)) return inflight.get(key);9 10  const promise = db.query('SELECT ...', [id])11    .then(async (product) => {12      const ttl = 270 + Math.random() * 60; // jittered13      await redis.setex(key, ttl, JSON.stringify(product));14      inflight.delete(key);15      return product;16    });17 18  inflight.set(key, promise);19  return promise;20}

Exercise

Add singleflight (in-flight promise map) or a lock around miss→DB→set for one hot key, plus TTL jitter. Describe how many DB queries you expect under 100 concurrent misses.