Hot keys and thundering herds
When one key draws the traffic, or expires under load, and the three fixes that work.
5 minute lesson
One key, too much traffic.
Traffic is rarely uniform. A celebrity's profile or a flash-sale product can draw a big share of all reads, and every one of those reads lands on the same cache node. That is a hot key: one node's CPU or network saturates while the rest of the cluster idles.
- Node A92%CPU · hot key
- Node B18%CPU
- Node C21%CPU
- hot key
- A single cache key that receives a large share of all requests, overloading the node that holds it.
- thundering herd
- Many requests miss at once and all hit the source together, typically when a hot key expires. Also called a cache stampede.
- single-flight
- Collapsing concurrent requests for the same key into one rebuild whose result they all share.
Key terms
When a hot key expires
- t = 0 s
The hot key is cached. 5,000 reads/sec hit the cache; the database sees almost nothing.
- Cachehit5,000 reads/s
- Database~0 qpsidle
- 0 s · cached
- 60 s · expires
- 60.2 s · rebuilt
hits
- t = 60 s
The entry expires. The rebuild query takes 200 ms, and in that window ~1,000 requests miss and all of them query the database.
- Cachemisskey expired
- Database1,000identical queries
- 0 s · cached
- 60 s · expires
- 60.2 s · rebuilt
hits, then herd
- t = 60.2 s
The first rebuild lands and reads hit again. If the burst slowed the database down, the window gets longer and the herd gets bigger.
- Cachehitrebuilt
- Databaserecoveringafter the spike
- 0 s · cached
- 60 s · expires
- 60.2 s · rebuilt
hits, then herd
Three ways out
- Single-flight. Let one request rebuild the key; the rest wait for its result or get the stale value.
- Jittered TTLs. Add randomness (say ±10%) so keys cached together don't expire together.
- Refresh early. Rebuild hot keys in the background before they expire, so readers never miss.
Only one worker should rebuild product:42. It takes a lock that expires on its own after 5 seconds, in case the worker dies.
Complete the Redis command.
SET lock:product:42 worker-7 {{1}} {{2}} 5000Show the answer
- 1: NX
- 2: PX
NX sets the key only if it doesn't exist, so exactly one worker wins the lock. PX 5000 expires it after 5,000 milliseconds, so a crashed worker can't hold it forever. EX 5000 would mean 5,000 seconds, and XX only sets keys that already exist.
Take a rebuild lock with SET key value NX PX ttl: one atomic command, one winner, automatic expiry.
The value (worker-7) identifies the owner, so the worker can release only its own lock: compare the value before deleting, atomically, for example in a small Lua script.
One product key gets 40% of all reads. The Redis node that holds it is at 100% CPU while the other nodes idle.
What helps most?
- Add more nodes to the Redis cluster
- Keep copies under several keys, or a small in-process cache in front of Redis
- Give the key a shorter TTL
- Move the products table to a bigger database server
Show the answer
- Keep copies under several keys, or a small in-process cache in front of Redis
A key lives on one node, so more nodes don't spread its reads. Copies under suffixed keys (product:42#1 … #4) land on different nodes, and an in-process cache answers most reads before they reach Redis at all.
A hot key is one node's problem. Copy it across nodes or cache it in-process.
Why do jittered TTLs help against stampedes?
Show the answer
Keys cached at the same moment would otherwise expire at the same moment. Random jitter spreads the expiries, so rebuilds trickle in instead of arriving as one spike.
Recap
- A hot key overloads one cache node. Copy it under several keys or cache it in-process.
- When a hot key expires, every concurrent miss hits the database: a thundering herd.
- Single-flight, jittered TTLs and early refresh keep the herd away.