Adding a read replica felt like free performance. We pointed the heavy read traffic at it, the primary breathed easier, and everyone was happy for about a day. Then the support tickets started: users saved something, immediately reloaded, and their change was gone.
Nothing was gone. The write went to the primary, the reload read from a replica that hadn't caught up yet, and the user experienced our replication lag as a bug. That's the trap with read replicas — they trade freshness for capacity, and you have to decide, per read, whether that trade is acceptable.
Lag is normal; surprise is the problem
A replica is always at least a little behind. Under load, or during a big write, or when a long query on the replica holds things up, that gap grows. The mistake isn't having lag — it's routing a read that can't tolerate staleness to a replica anyway.
The first thing I did was actually measure it, in seconds, and alert on it, so lag stopped being an invisible variable.
-- Lag in seconds, measured on the replica
SELECT now() - pg_last_xact_replay_timestamp() AS replica_lag;
-- On the primary: how far behind each replica is, in bytes
SELECT application_name,
pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind
FROM pg_stat_replication;Route reads by their tolerance
The real fix was classifying reads. A read that immediately follows a write by the same user — read-your-own-writes — goes to the primary. Analytics, search, list views where a second of staleness is invisible — those go to the replica. Most read traffic turned out to tolerate lag just fine; only a small, identifiable set didn't.
Once routing matched tolerance, the replica did its job and the "disappearing data" reports stopped.
When you truly need fresh, ask for it
For the cases that must be current, PostgreSQL gives you tools instead of guesswork. You can route to the primary, or use a lag threshold to fall back when a replica drifts too far. The point is to make the freshness requirement explicit in code rather than hoping the replica happens to be caught up.
- Measure replica lag in seconds and alert on it.
- Send read-your-own-writes to the primary.
- Send lag-tolerant reads (analytics, lists, search) to replicas.
- Fall back to the primary when a replica exceeds a lag threshold.
What I keep coming back to
None of this is clever. Name the failure mode, capture the before number, change one thing, and compare the same signal afterward. The boring version of database work is the version that actually holds up at 2 AM.