The replica came back from the crash looking completely healthy. SHOW SLAVE STATUS said both threads were running, Seconds_Behind_Master fell to zero within minutes, and replication monitoring went green. It took a quarterly reconciliation job, weeks later, to notice that a slice of rows on the replica did not match the source, and by then nobody could say exactly when the divergence began. That incident rewired how I think about replica crash safety, because the failure was not a bug: it was a chain of individually reasonable defaults doing exactly what they were documented to do. This is what a replica actually has to remember across a crash, how it can silently replay or skip events, and the two features, relay_log_recovery and GTID auto-positioning, that turn the whole class of incident into a non-event on MySQL 8.0 and 8.4.
What a replica has to remember after a crash
Replication on a replica is two jobs sharing a notebook. The I/O receiver thread connects to the source, reads its binary log, and appends events to local relay log files. The SQL applier thread reads the relay logs and executes the events. Correctness depends on one piece of state: exactly how far into the relay logs the applier has gotten. Lose track of that position in either direction and the replica applies events twice or never applies them at all.
Where that position lives is the whole game. With relay_log_info_repository set to TABLE, the default since MySQL 8.0, the applier's position is stored in mysql.slave_relay_log_info, an InnoDB table, and it is committed in the same transaction as the event being applied. Crash anywhere, and the position and the data agree by construction; that is what crash-safe means. The old FILE repository wrote positions to a relay-log.info file flushed on its own schedule, so a crash could leave the recorded position behind the actual data. The connection metadata got the same treatment: the master info repository moved to a table by default in 8.0, the FILE options were deprecated as the terminology shifted from master and slave to source and replica in 8.0.23, and in 8.4 the table is simply what you get. The relay logs themselves have their own durability knob, sync_relay_log, which defaults to one flush every ten thousand events; a crash can take the unsynced tail with it, leaving the relay log short or corrupt at the end.
The two failure modes: replay and skip
Replay is the position falling behind the data. The replica restarts, the applier picks up from a position it already passed, and events execute a second time. With InnoDB tables and primary keys this is usually loud: duplicate key errors stop the SQL thread, which is painful but honest. With non-transactional tables, or idempotent-looking UPDATEs, replay can apply silently, and now you have rows that differ from the source with no error anywhere. Skip is the position jumping ahead of the data, and in my experience the common cause is human. After a crash people reach for the skip counter, sql_slave_skip_counter on 8.0 or sql_replica_skip_counter in current terminology, to jump past whatever event is failing. Every skip is a deliberate divergence unless you can prove the event was already applied, and the server will never tell you otherwise. Skipping under GTIDs means injecting empty transactions, which is at least auditable, but it is still divergence by hand. The case that scared me most was neither: a replica whose relay log tail was lost in the crash while its recorded position said otherwise, which is precisely the gap relay_log_recovery exists to close.
relay_log_recovery: throw the relay logs away
The insight behind relay_log_recovery is that the relay logs are disposable. They are just a local cache of the source's binary log, so after a crash there is no reason to trust them. With relay_log_recovery enabled, a replica starting up discards its relay logs entirely and re-fetches from the source everything past the applier's committed position. Gaps, corruption, and half-synced tails stop mattering, because nothing local is trusted. The setting is off by default on both 8.0 and 8.4; enabling it is one of the first things I do on any replica I inherit, because the project made crash-safe metadata the default and still left this last decision to you.
Two prerequisites are non-negotiable. The source must still have the binary logs the replica needs to re-fetch, so binlog retention, binlog_expire_logs_seconds on the source, has to comfortably exceed your worst-case replica downtime plus recovery time; if the logs are purged, recovery becomes re-cloning. And the applier position itself must be trustworthy, which circles back to the TABLE repository and to GTIDs, because re-fetching from a position you cannot trust just automates the divergence. Relay log recovery without crash-safe positions is a faster way to get the wrong answer.
GTID auto-positioning: the actual safety net
File positions are fragile because they are coordinates in a filesystem. GTIDs are durable because they are names for the transactions themselves. With gtid_mode on and the replica configured with auto-positioning, the replica connects and tells the source: here is the set of transactions I have executed. The source then sends exactly what is missing. A crash changes nothing about that conversation; the replica re-fetches and re-applies only what its own gtid_executed set says it lacks, and already-applied transactions are never requested again. Configure it on the replication channel and verify the state:
CHANGE REPLICATION SOURCE TO SOURCE_AUTO_POSITION = 1;
SHOW REPLICA STATUS;
SELECT channel_name, service_state, last_error_number, last_error_message
FROM performance_schema.replication_connection_status;On 8.0 releases before 8.0.23 the statement is CHANGE MASTER TO with MASTER_AUTO_POSITION, and SHOW SLAVE STATUS instead of SHOW REPLICA STATUS; same machinery, older spelling. In the status output, Retrieved_Gtid_Set and Executed_Gtid_Set are the two sets to compare when you want to know what is in flight, and the Auto_Position field tells you whether the safety net is actually engaged. GTIDs plus auto-positioning plus relay_log_recovery plus the TABLE repository is the combination that makes replica crashes boring, and boring is the highest compliment in replication.
My crash-safe checklist
On every 8.0 replica I touch: gtid_mode on everywhere in the topology, auto-positioning on every channel, the default TABLE repositories left alone, relay_log_recovery set to 1, sync_relay_log at 1 where the workload tolerates the extra fsyncs, and source binlog retention sized for realistic downtime. On 8.4 the TABLE repositories are simply what you get, the FILE options are gone, but relay_log_recovery, sync_relay_log, and auto-positioning remain choices rather than defaults, so the checklist does not actually get shorter. Then, the step everyone skips: drill it. Pull the power on a staging replica quarterly, mid-load, and watch what it does on the way back up. Crash safety you have never tested is a hypothesis, not a configuration. Monitor the applier and connection status tables, not just the lag number, because a stopped SQL thread with a happy lag metric is exactly how my opening story stayed hidden.
Drill it, then watch the right things
Drill the crash before production drills it for you; that is this whole article in one sentence. Where my own tooling fits: MonPG is a PostgreSQL monitoring product today and does not monitor MySQL yet; MySQL support is being built, and replication safety is one of the areas where the details above, applier state, GTID set comparison, error surfacing, matter more than any single lag gauge. The MySQL monitoring (coming soon) page tracks the progress honestly. If you also run PostgreSQL, where the equivalent worry is replication slots and WAL retention instead of relay logs and binlog expiry, that side is live on the PostgreSQL monitoring page, and the blog has the rest of the series.