Why cache at all?
Latency, load and the hit ratio: what a cache buys you and what it costs.
4 minute lesson
A cache trades memory for time.
A cache keeps a copy of data somewhere faster to reach than its source: process memory, a Redis node next door, a CDN edge near the user. You pay in memory and in a new problem (stale copies). You get back latency and database load.
- Appreadproduct:42
- Cache< 1 msin memory
- Database5–50 msonly on a miss
- cache hit
- The requested key is in the cache, so the source is never touched.
- cache miss
- The key isn't in the cache; the value has to come from the slower source.
- hit ratio
- Hits divided by all lookups. 0.95 means 95% of reads never reach the source.
Key terms
The hit ratio does the heavy lifting
Every read pays for the cache lookup; a miss also pays for the database. So the average read costs cache + miss ratio × database. Step through three hit ratios with a 1 ms cache and a 20 ms query.
- 50% hits
Half the reads miss. The database still sees 50% of the traffic, and the average read takes ~11 ms.
- Hit ratio50%
- Avg read~11 mscache 1 ms · DB 20 ms
- DB load50%of all reads
- 90% hits
The average drops to ~3 ms, and the database sees a tenth of the reads.
- Hit ratio90%
- Avg read~3 mscache 1 ms · DB 20 ms
- DB load10%of all reads
- 99% hits
~1.2 ms. Going from 90% to 99% cuts database reads by another 10×.
- Hit ratio99%
- Avg read~1.2 mscache 1 ms · DB 20 ms
- DB load1%of all reads
Order these from fastest to slowest.
- Random 4 KB read from an SSD
- L1 cache reference
- Packet round trip, California to the Netherlands
- Main memory reference
- Disk seek on a spinning HDD
Show the answer
- L1 cache reference
- Main memory reference
- Random 4 KB read from an SSD
- Disk seek on a spinning HDD
- Packet round trip, California to the Netherlands
Roughly: L1 ~1 ns, RAM ~100 ns, SSD random read ~100 µs, HDD seek ~10 ms, California to the Netherlands and back ~150 ms. RAM is about a thousand times faster than an SSD read, and that one round trip costs as much as ~1,500 SSD reads. That gap is why a cache in RAM pays off.
Nanoseconds for CPU caches and RAM, microseconds for SSDs, milliseconds for disks and networks.
What a cache costs you
- Staleness. The copy can lag the source. You decide how much lag is acceptable.
- Memory. RAM costs more than disk, so you cache the hot set, not everything.
- Cold starts. An empty cache sends every read to the database at once.
- A new failure mode. If the cache goes down, can the database take the full load?
A cache answers in 1 ms and the database in 20 ms. 80% of reads hit the cache.
About how long does the average read take?
- About 1 ms
- About 5 ms
- About 16 ms
- About 21 ms
Show the answer
- About 5 ms
Every read pays the 1 ms lookup, and the 20% that miss also pay the 20 ms database read: 1 + 0.2 × 20 = 5 ms.
Average read ≈ cache time + miss ratio × database time.
The hit ratio goes from 90% to 99%. What happens to database read load?
Show the answer
It drops 10×: misses go from 10% of reads to 1%. The last few points of hit ratio matter most.
Recap
- A cache trades memory and some staleness for latency and database load.
- Average read ≈ cache time + miss ratio × database time.
- Going from 90% to 99% hits cuts database reads 10×.