MySQL Semi-Sync Replication: When 1ms Network RTT Halves Your Commit Throughput
Enabled semi-synchronous replication to guarantee zero data loss and watched commit latency quadruple? Here is the deep architecture of MySQL binary log commits, AFTER_SYNC vs AFTER_COMMIT, and cross-AZ network latency.
Our infrastructure team decided to enable semi-synchronous replication on our production AWS RDS MySQL fleet to eliminate any risk of data loss during multi-availability-zone failovers.
Within twenty minutes of enabling the setting, our checkout API’s p95 transaction latency climbed from 14ms to 58ms, and overall write throughput collapsed by nearly 50%. The network ping between our primary AZ and secondary AZ was only 1.2 milliseconds.
We could not fathom how a 1.2ms network round trip could produce a fourfold latency penalty on transactional writes until we traced the exact internal execution path of MySQL’s binary log commit pipeline.
Asynchronous vs. Semi-Synchronous mechanics
In standard asynchronous MySQL replication, the primary writes changes to the binary log and immediately returns success to the client. Background binlog dump threads read the log and stream events to replicas at their own pace. If the primary crashes before an event reaches the replica, data is lost.
Semi-synchronous replication introduces an acknowledgment barrier: the primary writes the transaction, sends the binlog event to at least one semi-sync replica, and blocks until the replica acknowledges that it has written the event to its relay log.
This guarantee ensures zero data loss, but it puts network round-trip time directly into the critical execution path of every single COMMIT statement.
AFTER_SYNC vs. AFTER_COMMIT: The phantom read dilemma
MySQL supports two semi-sync modes via rpl_semi_sync_master_wait_point:
1. AFTER_COMMIT: The primary commits the transaction to the InnoDB storage engine first, then sends the binlog to the replica and waits for an ACK before responding to the client. This had a fatal flaw: other concurrent clients on the primary could read the committed changes before the replica received them. If the primary crashed during the wait, clients had seen data that never made it to the replica.
2. AFTER_SYNC (default since MySQL 5.7): The primary writes to the binlog and flushes it, waits for the replica ACK, and only then commits the changes to InnoDB. This prevents phantom reads, but it keeps InnoDB transaction locks held for the entire duration of the network round trip.
The serialization multiplier effect
A 1.2ms network RTT sounds negligible. But because transaction locks remain held during that 1.2ms window, any concurrent transaction attempting to modify the same rows or adjacent gaps must wait.
Furthermore, MySQL’s binary log group commit optimization must pause to accumulate transactions. If your application issues thousands of small, single-row updates sequentially, those 1.2ms delays stack up linearly, destroying thread concurrency.
-- Measuring semi-sync wait statistics and average latency
SHOW STATUS LIKE 'Rpl_semi_sync_master_wait_sessions';
SHOW STATUS LIKE 'Rpl_semi_sync_master_tx_avg_wait_time';
SHOW STATUS LIKE 'Rpl_semi_sync_master_status';The silent timeout trap
A critical operational hazard in semi-sync replication is rpl_semi_sync_master_timeout (default 10,000ms). If a replica fails to acknowledge a transaction within this timeout, MySQL silently downgrades replication to asynchronous mode and continues accepting writes.
Your monitoring must alert whenever Rpl_semi_sync_master_status flips from ON to OFF. If replication downgrades silently without alerting, an unexpected failover will result in the very split-brain data loss that semi-sync was deployed to prevent.
The practical standard
High-level database architecture is not about drawn boxes on an infrastructure diagram. It is about how the engine manages shared resources under concurrency — memory, latches, write-ahead logs, and lock tables. When things break at 2 AM, the fix is rarely adding another replica or throwing more CPU at the host. The fix is understanding the underlying resource bottleneck, measuring the exact wait event, and applying the architectural constraint that makes the system predictable.