The page said replica lag, but the real story was in Last_SQL_Error: Could not execute Write_rows event on table shop.orders; Duplicate entry '74829113' for key 'PRIMARY', Error_code: 1062. The on-call engineer did what the old runbook said — SET GLOBAL sql_slave_skip_counter=1, START SLAVE — and the replica resumed. Twenty minutes later it stopped again on a different table. Skip, resume, stop, skip, resume. By midnight the counter had been bumped six times, replication was "healthy," and the replica was now serving a mix of rows that did not exist on the primary and missing rows that did. The skip counter had done exactly what it is designed to do — step over replication events without applying them — and each step widened the divergence between two servers that every dashboard claimed were in sync. The postmortem found the original cause in five minutes: a migration script had been pointed at the replica "just to warm it up" three days earlier. Errors 1062 and 1032 are not replication problems; they are evidence that a data problem already happened, and the only honest fixes work on the data, not the counter.
What do errors 1062 and 1032 on the SQL thread actually mean?
Error 1062 means the replica tried to insert a row whose unique key already exists in its copy of the table; error 1032 means it tried to update or delete a row that is not there — and both mean the replica's data diverged from the primary's before the event arrived. This is the point the skip counter obscures: by the time the SQL thread stops, the divergence has already happened somewhere else. A duplicate-key on a Write_rows event implies the row was created on the replica by a path that never went through replication — a direct write, a restored partial backup, a previous skip. A 1032 — Could not execute Update_rows event on table shop.customers; Can't find record in 'customers', Error_code: 1032 — implies the row was deleted or never existed on the replica, again through some side channel. Replication itself is almost never the bug: row-based events apply deterministically, and a replica that only ever applies the primary's event stream cannot produce these errors. So the error is a tripwire on a drift that already exists, which is why treating it as a speed bump is backwards. The replica drift and pt-table-checksum notes cover how that drift accumulates silently; 1062 and 1032 are the moments it becomes loud, usually when a later event collides with the earlier divergence.
Why is sql_slave_skip_counter so dangerous, especially with GTID?
Because skipping an event does not resolve the conflict — it deletes the event from history while leaving the divergent data in place, and with GTID enabled the counter does not even work, which pushes people toward worse improvisations. With traditional file/position replication, sql_slave_skip_counter=N tells the SQL thread to discard the next N events and carry on; whatever that event was supposed to change now never happens on the replica, on top of whatever divergence caused the error. One skip is a judgment call; a runbook that skips reflexively manufactures the drift I described at six skips and counting. Under GTID, the server refuses sql_slave_skip_counter outright — ERROR 1858: sql_slave_skip_counter can not be set when the server is running with @@GLOBAL.GTID_MODE = ON — and the sanctioned workaround is injecting an empty transaction with the skipped GTID, which has the same semantic effect dressed in better ceremony: you are still declaring an event unapplied. The deeper problem with both is asymmetry of information: the person skipping at midnight rarely knows what the event contained, whether the replica or the primary holds the truth, or how many downstream tables the skip will desynchronize next. Skipping is choosing divergence blindly, and it is only ever defensible when you have already inspected the event and decided the replica's version of reality wins — at which point you could have just fixed the row.
-- what the replica is actually complaining about (8.0 syntax)
SHOW REPLICA STATUS\G
-- read: Last_SQL_Errno, Last_SQL_Error,
-- Retrieved_Gtid_Set vs Executed_Gtid_Set (the gap is the stuck GTID),
-- Relay_Log_File / Exec_Master_Log_Pos for the event location
-- inspect the offending event before deciding anything
SHOW RELAYLOG EVENTS IN 'relay-bin.000041' FROM 88342107 LIMIT 5;
-- or mysqlbinlog the relay log at that position for the full row image
-- the GTID empty-transaction workaround — only after you understand the event:
STOP REPLICA SQL_THREAD;
SET GTID_NEXT = '3e11fa47-71ca-11e1-9e33-c80aa9429562:118234911';
BEGIN; COMMIT;
SET GTID_NEXT = AUTOMATIC;
START REPLICA SQL_THREAD;How do you investigate the conflict before touching anything?
Compare the exact row on both servers and read the event from the relay log — ten minutes of inspection tells you who is right, and that answer chooses the fix. Start with the error text, which names the table and the key. SELECT that key on the primary and the replica: for 1062, compare the full rows, because "duplicate" does not mean "identical" — our phantom order rows looked alike until a status column revealed which server had processed the shipment. For 1032, the row exists only on the primary, and the question is whether the replica deleted it (someone's cleanup script, a cascade from a drifted parent) or never received it (a skipped insert further back — check whether earlier errors were skipped in this replica's history). Then read the event itself with SHOW RELAYLOG EVENTS or mysqlbinlog against the relay log at Exec_Master_Log_Pos; with row-based replication and a full row image, the event shows the before and after values, which is usually enough to reconstruct intent. While you are in there, check the obvious crime scenes: read_only and super_read_only status on the replica, recent maintenance logged against the wrong host, and any restore or clone operations in the last days. The replica crash-safety notes matter here too — a replica that crashed with unsynced relay log info can reapply events and produce exactly these errors with no human involved at all.
What are the safe fixes, ranked from smallest blast radius to largest?
Fix the single conflicting row when the cause is understood and the divergence is provably that one row; resync the affected tables with pt-table-sync when divergence is bounded; rebuild the replica when drift is wide or the history is untrustworthy. The single-row fix is surgical: delete the phantom row or insert the missing one on the replica so the event applies cleanly, then START REPLICA SQL_THREAD — no counter, no empty transaction, and the replica converges by construction. The discipline is proof: one conflicting row is a fix, five conflicting tables is a pattern, and a pattern means the replica's whole history is suspect. The table-level fix is the checksum-and-sync workflow from the drift notes: pt-table-checksum to bound the divergence, pt-table-sync --print reviewed by a human, then --execute. The rebuild is the honest answer when skipping already happened and you cannot enumerate what was skipped — at that point the replica is an unknowable mixture, and rebuilding from a fresh backup or clone is cheaper than archaeology; the GTID errant transaction notes cover the nastiest version, where the replica holds transactions the primary never saw, which actively poisons a future failover. One more rule that has saved me twice: whatever fix you choose, run pt-table-checksum afterward against the affected tables, because the error that paged you is rarely the only divergence — it is just the one that collided with a unique key first.
How do you keep 1062 and 1032 from ever reaching the page?
Close the write paths, make crashes safe, and alert on the first occurrence with enough context to investigate instead of skip. The write-path closure is non-negotiable: read_only plus super_read_only on every replica, SUPER privileges audited, and deployment tooling that cannot be pointed at a replica by a typo — the majority of these incidents are human writes through an open door. Crash safety keeps the machine honest: sync_relay_log_info and relay_log_recovery settings from the crash-safety notes prevent the reapply-after-crash flavor. Then the alerting: a stopped SQL thread should page with Last_SQL_Error attached, and the runbook behind that page should start at "compare the row on both servers," not at the skip counter — delete sql_slave_skip_counter from runbooks entirely, and never, under any circumstances, set slave-skip-errors in my.cnf, which converts every future divergence into silent, permanent drift in exchange for never being paged. The goal is not a replica that never stops; it is a replica whose every stop gets investigated, because a replica you cannot trust is worse than no replica — it is a lie with a failover button attached.
Where MonPG stands on MySQL
I build MonPG, so the honest line: MonPG monitors PostgreSQL today, and MySQL support is in active development, not shipped. The signals in this piece — SQL thread stops paged with the full error text and offending GTID, read_only drift across the replica fleet, and post-repair checksum results on the same timeline as the incident — are exactly what the MySQL work is designed to surface, so a 1062 becomes a data question instead of a reflex. The MySQL monitoring (coming soon) page tracks that work as it lands. Until it ships, the same evidence-first approach runs on the PostgreSQL side today, and the rest of these MySQL field notes live on the blog.