Replication lag and read-your-writes
Followers are always a little behind. What users notice of that, and how to hide it. Builds on Consistency models.
5 minute lesson
A follower lives in the past
With asynchronous replication a follower applies each change a little after the leader. That delay is the replication lag. Usually it is a few milliseconds. Under heavy writes, or with a slow network, it can grow to seconds or minutes, and every read from that follower is that old.
- Normallymilliseconds
- Under loadseconds to minutes
- replication lag
- How far a follower is behind its leader, in time or in changes not yet applied.
- follower
- A replica that receives the leader's writes and can answer reads. Also called a standby or a read replica.
Key terms
The change that seems to vanish
- Save
Ana changes her display name and presses Save. The write goes to the leader and is confirmed.
- Anasaves
- Leadernew name
- Reload
The page reloads. This read is sent to a follower that is half a second behind. It answers with the old name.
- Anareloads
- Followerold name
- Confusion
Ana sees her old name and concludes that the save failed. She saves again, or files a bug. The data was never wrong. She was shown the past.
- The datacorrect
- What Ana seeslooks like a failure
A user saves a new address. The write is confirmed. The page reloads from a follower and shows the old address.
What happened to the new address?
- It is on the leader and has not reached the follower
- It was lost between the leader and the follower
- It was rolled back when the page was reloaded
- It was rejected by the follower as out of date
Show the answer
- It is on the leader and has not reached the follower
The write is safely on the leader. The follower will apply it in a moment and is simply not there yet. Nothing was lost or undone. The reload came too early for the replica it was sent to. Read the same row from the leader, or from the follower a second later, and the new address is there.
A stale read from a follower is a delay, not a loss. The write is on the leader.
Show people their own changes
Other users will not notice that Ana's name arrives half a second late. Ana will. The guarantee she needs is read-your-writes: whoever changed something sees the change at once. There are three common ways to give it.
- Read your own things from the leader. A user's own profile comes from the leader, and everyone else's from followers.
- Read from the leader for a while after writing. For a few seconds after a save, that user's reads go to the leader.
- Wait for the follower. Remember how far the leader's log had got at the save, and read only from a follower that has reached that point.
A social app reads from followers. Users complain that their own edited profile shows old data for a moment. The team wants to keep the leader lightly loaded.
Which reads should move to the leader?
- Each user's reads of their own profile
- All reads of all profiles by anyone
- Reads of the most popular profiles
- No reads, as followers will catch up
Show the answer
- Each user's reads of their own profile
Only the person who made a change is waiting to see it. A stranger looking at the same profile cannot tell that it is half a second behind. Sending a user's reads of their own data to the leader gives everyone read-your-writes, and it moves only a small share of the traffic.
Read-your-writes needs the leader only for a user's own data, not for everyone's.
Keep an eye on the lag
- Measure the lag of every follower, and alert when it grows.
- Stop sending reads to a follower that has fallen far behind.
- Keep one user on one follower. Jumping between followers can show newer data first and older data after, as if time ran backwards.
A user saves a change and then sees the old value. Why, and how do you prevent it?
Show the answer
The save went to the leader, and the next read went to a follower that had not applied the change yet. To prevent it, give that user read-your-writes: read data they can change from the leader, or send their reads to the leader for a short time after a write, or read only from a follower that has caught up to the write.
Recap
- A follower is behind the leader by its replication lag: usually milliseconds, sometimes far more.
- Reading from a follower just after writing can show the old value.
- Give users read-your-writes, and watch the lag of every follower.