Hot partitions, rebalancing and cross-shard queries
Living with a sharded database: one shard that gets too much, moving data while the system runs, and work that spans shards.
6 minute lesson
Even data is not even load
A good shard key spreads the rows evenly. It cannot spread the attention. If one customer is a hundred times busier than the others, the shard that holds that customer is a hot partition: it runs at its limit while the others idle.
- Shard 110% of requests
- Shard 280% of requests
- Shard 310% of requests
Cooling a hot partition
- Give it a shard of its own. Move the one busy customer to a dedicated machine, sized for it.
- Split the key further. Add a second part to the key, such as the month, so that one customer's rows spread over several partitions.
- Take reads away. Put a cache or read replicas in front of the hot shard.
Each of these treats one known hot spot. Finding them is the first step: measure the load per shard and per key, not just the total.
A tool is sharded by company id over 8 shards. One company causes 60% of all requests, and its shard is overloaded. The team doubles the number of shards to 16.
What changes for the overloaded shard?
- Little: that company is still on one single shard
- A lot: its load is now halved between two shards
- A lot: its load is spread over all sixteen shards
- Nothing: new shards never take over any of the data
Show the answer
- Little: that company is still on one single shard
All rows of one company share one key value and stay together. Doubling the shards moves some smaller companies away, but the big one remains whole on one machine and brings 60% of the traffic with it. A hot key needs its own remedy: a dedicated shard, a finer key, or a cache in front.
Adding shards spreads keys, not the load of one key. A hot key stays on one shard.
Moving data while the system runs
- Imbalance
A fourth shard is added to three. It is empty, and the old ones are full. Nothing has improved yet.
- Shards 1 to 3full
- Shard 4empty
- Migrate
A background process moves ranges of data to the new shard, a piece at a time. That is rebalancing. The copying uses disk and network, so the busy shards get slower while it runs.
- Shards 1 to 3send ranges
- Shard 4receives them
- Switch
When a range has arrived, the routing is updated, and requests for it go to the new shard. The old copy is deleted. This repeats until the shards are even.
- Shards 1 to 4even
A cluster is at 95% of its capacity in the middle of the busiest day of the year. The team adds a shard to relieve it.
What happens first?
- The migration adds load, and things get worse
- The new shard takes over a fair share at once
- The cluster pauses until the data has moved
- The old shards delete a part of their data
Show the answer
- The migration adds load, and things get worse
The new shard is empty, and it helps only once data has been moved to it. Moving data means reading it from the busy shards and sending it over the network, on top of the traffic they can barely handle. Relief comes hours later. Capacity has to be added while there is still room to pay for the migration.
Rebalancing costs capacity before it gives any. Add shards well before the limit.
When work spans shards
- A query without the keyAsks every shardkeep these rare or serve them from another store
- A join across shardsDone by the applicationor avoided by keeping related rows on one shard
- A transaction across shardsNo single commitneeds a protocol of its own, taught in a later track
What happens when a shard is added to a running cluster, and what does it cost?
Show the answer
The new shard starts empty. A background process moves ranges of data to it from the others, switches the routing for each range once it has arrived and deletes the old copy. While that runs, the copying competes with normal traffic for disk and network. So shards are added before the cluster is at its limit, and migrations are scheduled for quiet hours.
Recap
- A hot partition gets far more traffic than its share. More shards do not cool it.
- Rebalancing moves data in the background, and slows the cluster while it runs. Add capacity early.
- Queries, joins and transactions across shards are costly. Keep what belongs together on one shard.