Database replication

DataTier 3 · scale vocabularyPhase 5 · It got popular

Keeping live copies of the database so reads can be spread across copies and the system survives one database failing.

If all your data lives on one database and it fails, the product is down and the data may be lost. Replication exists to keep live copies of the database on other machines, so reads can be spread across them and a failure does not mean an outage or lost data.

One database is both a bottleneck (all reads and writes hit it) and a single point of failure (if it dies, everything stops). Replication keeps synchronized copies. Writes go to a primary, which streams changes to replicas; the many read requests are spread across the replicas.

If the primary fails, a replica can be promoted to take over. So replication buys both more read capacity and resilience.

A junior PM, stuck

A customer's support ticket says the order she just placed vanished from her order list, then reappeared about two minutes later. Support wants to know if we lost her order or double-created it. I cannot tell whether this is a real data bug or something normal, and I have to answer.

Nothing was lost. Our database is not one machine: writes go to a primary, and reads come from copies called replicas that trail the primary by a moment. When a read lands on a replica that has not caught up yet, brand-new data looks missing for a beat. Reading this is quick. Here are the three calls from her session, the write and two reads, and you can see the order appear, miss, then appear again.

One order across a write and two reads: present, missing, then present again
The order is placed (write to the primary)
1.1
201
{ "order_id": "o_5514", "status": "confirmed" }
300ms later, the order list (read from a replica)
1.1
200
{ "orders": [ ] }
2 seconds later, the same read
1.1
200
{ "orders": [ { "order_id": "o_5514" } ] }

Click a step to see the lines it points at.

Reading the empty list as a lost order. The write returned 201 and the item reappears seconds later; nothing was lost, a read just hit a replica that had not caught up.
Suspecting a double-create. The order kept the same order_id o_5514 throughout; a vanish-and-return is one row seen at two moments, not two rows.
Escalating it as a P0 data bug. Brief lag right after a write is expected behavior in a replicated database; treating every reappearing item as corruption burns engineering time on a non-issue.
Missing that the read where freshness matters should hit the primary. If a screen must show a just-written record immediately, the fix is routing that specific read to the primary, not accepting flaky lists.

Tell support: "We did not lose or duplicate her order. The write saved to the primary, and a read a fraction of a second later hit a replica that had not caught up yet, so it looked missing for a moment, standard replication lag. If the just-placed order needs to show instantly, we route that one read to the primary." You told lag apart from data loss from the responses alone.

It's how read-heavy products scale and stay up; useful context when reliability and capacity come up.
Replication lag explains rare "my change didn't stick" reports right after saving; knowing it exists helps triage.

"Reads go to replicas; the report's just hitting replication lag."

Appears in Phase 5, It got popular.