The most reliable incident I ever inherited fired every weekday at 18:05. The settlement batch started on the source at six, and by five past, the read replicas were forty minutes behind. Dashboards went stale, support tickets arrived, and every morning the replicas had caught up again, so half the team believed nothing was really wrong. The source was writing with two hundred connections. The replicas were applying with one thread each. It was never a capacity problem; it was a serialization problem, and the fix was not bigger hardware but teaching the replica to apply in parallel and then measuring lag with something better than a counter that lies.
What follows is what that incident and its successors taught me: how MySQL's parallel applier actually decides what may run concurrently, which settings live on the source versus the replica, why Seconds_Behind_Master cannot be trusted, and the performance_schema views that tell the truth.
The bottleneck is one thread
On the source, hundreds of connections execute transactions concurrently, and InnoDB's whole design exists to let them. On a classic replica, an IO thread writes the relay log and a single SQL applier thread replays every transaction serially, in commit order. The moment the source's committed-transaction throughput exceeds what one thread can apply, a queue forms, and replication lag is simply the depth of that queue expressed in time.
Two details make it worse than it looks. First, the single applier thread has no concurrency to hide latency behind, so the same transaction often applies slower on the replica than it committed on the source, even on identical hardware. Second, lag peaks exactly when writes peak, which is exactly when the business is watching the dashboards the replica serves. A single-threaded applier does not fail at random; it fails on schedule, during your busiest hour.
What replica_parallel_type actually parallelizes
Setting replica_parallel_workers to a nonzero count spins up that many applier workers plus a coordinator thread that hands out transactions. The variable replica_parallel_type decides the partitioning rule, and this choice is where most attempts succeed or fail.
The DATABASE rule, the ancient default, parallelizes transactions that touched different schemas. If you run a multi-tenant application with hundreds of databases on one instance, it works beautifully. If you run the overwhelmingly common single-schema application, every transaction lands in the same bucket and you have paid for workers that never receive work. LOGICAL_CLOCK instead uses the commit history the source stamps into the binlog: transactions that were part of the same commit group, meaning they committed together without seeing each other's effects, may apply in parallel on the replica, with sequence numbers preserving order where it matters. Since MySQL 8.0.27 the defaults finally reflect this: four workers, LOGICAL_CLOCK, and replica_preserve_commit_order enabled out of the box. Before that release, a stock replica was single-threaded unless somebody knew to ask.
WRITESET: letting the source admit more parallelism
LOGICAL_CLOCK is only as good as the dependency information the source stamps into the binlog, and on MySQL 8.0 that is controlled on the source by binlog_transaction_dependency_tracking, a knob deprecated in 8.0.35 and removed in 8.4, where writeset-style tracking is simply the fixed behavior. The 8.0 default, COMMIT_ORDER, is conservative: only transactions that committed in the same group are marked parallel-safe. WRITESET changes the game. The source hashes the rows each transaction changed and stamps transactions as independent whenever their write sets provably do not overlap, which qualifies far more transactions for concurrent apply and keeps the replica's workers actually fed. WRITESET_SESSION adds one more constraint, serializing transactions from the same session, which matters if applications read their own writes from the replica. On 8.4 there is nothing left to set here, which is the direction I wanted anyway.
Three prerequisites shape whether WRITESET helps you. It requires row-based binary logging. It can only track transactions on tables with primary keys or unique keys; writes to keyless tables fall back to commit-order conservatism, which is one more entry on the long list of reasons every table needs a primary key. And on 8.0, binlog_transaction_dependency_history_size bounds how much recent write history the source remembers when computing independence, so on very write-heavy sources a bigger history finds more parallelism at a memory cost; it shares the tracking knob's lifecycle, deprecated in 8.0.35 and gone in 8.4. The setup work that makes performance_schema trustworthy enough to observe all of this is covered in Performance Schema with low overhead.
Seconds_Behind_Master lies
SHOW REPLICA STATUS, formerly SHOW SLAVE STATUS, reports Seconds_Behind_Source, formerly Seconds_Behind_Master, and that number lies in at least three ways. It is computed from the timestamp of the event currently being applied, and every event in a transaction carries the source's commit timestamp, so while a ten-minute batch transaction is mid-apply the number is really reporting that one transaction's age rather than the depth of any queue behind it, and whenever the applier is between events it reads zero even if the receiver thread is quietly minutes behind on a slow network. It goes NULL or stale when the IO thread stops, even though the data keeps aging in exactly that situation. And under parallel apply it reflects the coordinator's watermark, not what the workers have actually finished. Monitoring that pages on this counter alone will sleep through real lag and wake you for ghosts.
The honest measurement uses the original commit timestamps the source stamps into the binlog, surfaced through performance_schema. The coordinator view tells you what it has handed to the workers and when the source committed it:
SELECT channel_name,
service_state,
last_processed_transaction,
timediff(now(3),
last_processed_transaction_original_commit_timestamp) AS lag
FROM performance_schema.replication_applier_status_by_coordinator;Three caveats keep this honest. In the coordinator view, processed means buffered into a worker's queue, not yet committed by a worker, so it flatters progress slightly; the truer apply watermark is the minimum of last_applied_transaction_original_commit_timestamp across the worker view. Clock skew between source and replica inflates or deflates the result, so keep NTP boring. And on an idle source there is nothing to apply, so heartbeat events, set per channel with SOURCE_HEARTBEAT_PERIOD on CHANGE REPLICATION SOURCE TO, are what keep the watermark moving and the measurement meaningful during quiet hours.
Watching the workers
The per-worker view is where diagnosis happens:
SELECT channel_name, worker_id, service_state,
last_applied_transaction,
applying_transaction
FROM performance_schema.replication_applier_status_by_worker;Two patterns matter. One worker permanently busy while the rest idle is skew: usually a hot row or a small hot table serializing everything that touches it, because WRITESET cannot parallelize transactions that genuinely conflict, and the fix lives in the application or schema, not in a worker count. All workers idle while lag grows means the bottleneck is upstream of apply entirely: the IO thread, the network, or relay-log disk. The connection side of the story lives in performance_schema.replication_connection_status, and comparing the retrieved GTID set against the executed set in SHOW REPLICA STATUS separates a replica that cannot download fast enough from one that cannot apply fast enough. Those have completely different fixes, and conflating them wastes days.
Tuning notes that actually matter
Start with four to eight workers. Going past sixteen rarely helps unless write-set independence is high and cores are idle, because every worker is a real thread with real scheduling overhead. Keep replica_preserve_commit_order on if anything reads ordered state from the replica; the throughput cost is measurable but usually modest, and silent reordering underneath a consumer is a nasty surprise to debug. And if lag persists after parallel apply is healthy, look at the replica's own durability settings: a replica running fully synchronous flush settings on slow storage applies slower than the source commits, and since a replica can be rebuilt from the source plus backups, relaxing its flush behavior is often the real fix rather than more workers.
One last habit: when lag does spike, check what the slow query evidence says on the source first. A deploy that turned a fast statement into a scanner will show up as apply lag on every replica at once, and the slow query digest workflow finds that statement faster than any amount of staring at replica status.
Where MonPG fits, honestly
One disclosure, since it shapes how you should read this series: I work on MonPG, a PostgreSQL monitoring platform, and it does not monitor MySQL; MySQL support is in active development. When it ships, the 18:05 incident is the bar it has to clear: lag computed from original commit timestamps instead of the legacy counter, per-worker busy state so skew is visible without running a query, and retrieved-versus-executed GTID sets graphed together so download and apply stop being conflated. Progress lands on the MySQL monitoring (coming soon) page. The equivalent streaming-replication view already exists for PostgreSQL on the PostgreSQL side, the comparison pages lay out the differences, and more MySQL field notes live on the blog.