The alert said "MySQL down", and the error log said something much worse: page 7157 in tablespace 3423 had a checksum mismatch, InnoDB had refused to continue, and the server now crashed about 40 seconds after every startup — just long enough for the load balancer to mark it healthy and send traffic. That page, it turned out, held a chunk of the sessions table, and every connection that touched it took the whole server down. Corruption is the failure mode where the database is trying to protect you from itself, and the fight is over how much you can salvage before you rebuild.
Here is the triage I now run by rote: why InnoDB crashes on purpose, how to scope the damage before touching a knob, what each innodb_force_recovery level actually skips, the dump-and-rebuild runbook, and how to figure out whether hardware did it — because the right fix depends on the answer.
Why does InnoDB crash the server on purpose?
Because a corrupt page is proof that the bytes on disk are not the bytes InnoDB wrote, and continuing could spread the damage. Every page read verifies a checksum — innodb_checksum_algorithm=crc32 is the 8.0 default, the old innodb_checksums switch having been removed — a value computed when the page was flushed and checked when it is read back. A mismatch means the page is garbage: applying redo to it, merging insert buffer entries into it, or worse, letting a query read fabricated rows, can convert one bad 16KB page into logical corruption across the table. So InnoDB asserts and aborts the server. It feels like hostility; it is containment. Note the boundary of the protection: the doublewrite buffer guards against torn pages from a crash during the write itself — the mechanism in doublewrite and torn pages — but a page that was written cleanly and rots silently on disk afterward is invisible to doublewrite, and it is exactly what checksums exist to catch.
How do you scope the corruption before touching anything?
Read the error log for the exact space ID and page number, map the space ID to a table, and verify offline if you can. The log line names the coordinates — page no 7157, space id 3423, plus the checksum values it computed and the ones it found on disk. The space ID resolves to a table through the InnoDB views:
SELECT it.NAME AS table_name,
ts.NAME AS tablespace_name,
df.PATH AS file_path
FROM information_schema.INNODB_TABLES it
JOIN information_schema.INNODB_TABLESPACES ts
ON ts.SPACE = it.SPACE
JOIN information_schema.INNODB_DATAFILES df
ON df.SPACE = it.SPACE
WHERE it.SPACE = 3423;
-- per-table logical check, if the server stays up long enough:
CHECK TABLE app.sessions EXTENDED;If the instance can be stopped, innochecksum on the .ibd file gives a page-by-page verdict offline and confirms whether the damage is one page or a pattern across the file. One discipline matters more than any tool here: stop letting the server crash-loop. Pull it from rotation, and check your replica before anything else — if the replica is clean and current, the shortest path back to service is promotion, and the salvage work becomes a background exercise instead of a live incident. Whether that option exists at 3 a.m. is decided long in advance by the durability settings covered in replica crash safety.
What do innodb_force_recovery levels 1 through 6 skip?
Each level trades another piece of InnoDB's self-consistency for the ability to read data out, and you climb them one at a time, restarting between attempts, only until the server stays up long enough to dump. The variable goes in the config file — it is not dynamic — and at any nonzero value InnoDB already refuses INSERT, UPDATE, and DELETE; level 4 and above puts the server fully read-only, which costs nothing in practice because everything you do under force_recovery is read-and-export anyway. The ladder: level 1 ignores corrupt pages on reads, letting SELECT skip over them, which is often enough on its own. Level 2 stops the background threads — above all purge — when a background thread is what trips on the corruption. Level 3 skips the rollback of incomplete transactions after crash recovery, so their half-applied changes linger. Level 4 stops insert buffer merges and statistics recalculation. Levels 5 and 6 are the data-loss territory: 5 skips the undo log scan and treats incomplete transactions as committed, and 6 skips the redo roll-forward entirely, meaning recently committed transactions may simply not be there. I have never needed 6 in anger, and the one time I used 5 the dump came back missing a day of orders that we later recovered from binlogs.
What does the dump-and-rebuild runbook look like?
Get a read-only copy of everything salvageable, starting with what you need most, then rebuild from scratch — never "repair" in place and carry on. Concretely: set innodb_force_recovery=1, restart, and immediately dump. Small critical tables first — users, credentials, configuration — then the big ones; for the corrupt table itself, dump primary-key ranges around the bad page so one skipped page does not kill the whole export. The log told you page 7157, but no arithmetic converts a page number into a key interval — B-tree splits, merges, and variable-length rows see to that — so work from key ranges you know, and if a range dies on the bad page, walk it from the other end with ORDER BY id DESC and keep what both directions yield. mysqldump is fine for small schemas; for anything large, parallel dumping with mydumper cuts hours, and the tradeoffs are in mydumper vs mysqldump for large databases. When the dump is done, wipe the instance — the data directory itself, not just the tables — initialize fresh, load the dump, replay binlogs up to the corruption point if you have them, the same discipline as the xtrabackup PITR workflow, and verify with row counts, checksums, and application smoke tests. Then remove innodb_force_recovery from the config. Nobody should ever run production traffic under it.
Hardware, bug, or operator error — how do you find the cause?
Assume hardware until the evidence says otherwise, because it usually is hardware. The useful questions: does corruption reappear on the rebuilt instance on the same machine? That is disk, RAID controller, or RAM until proven otherwise — check dmesg, SMART data, the controller's event log and its write-cache policy (a volatile cache without a battery backup lying about flushes is a classic), and memory errors via mcelog or EDAC. Did it appear on the replica too? Replication applies logical changes — the redo stream never leaves the box — so identical corruption on both sides points away from one machine's disk and toward a server bug, or corruption that entered as bad data written through SQL and replicated like any other write. Did it follow a power loss on a box running with doublewrite disabled, or a filesystem that ignores flush barriers? That is an operator-caused torn page. Genuine InnoDB corruption bugs exist, but in twenty years I have met two, against a dozen dead disks, three lying RAID caches, and one memorable RAM stick. The root cause picks the prevention — new hardware, new cache policy, or a config change — not just a rebuilt table.
Why are replicas and backups the real answer?
Because innodb_force_recovery is salvage, not recovery, and salvage is what you do while the business runs on something else. A clean replica turns a corruption incident into a ten-minute promotion plus a quiet afternoon of dumping; tested backups turn the worst case — corruption on every copy, which happens when logical corruption replicates — into a restore and binlog replay measured in hours instead of a forensic extraction measured in days. The uncomfortable corollary is that backups you have never restored are a rumor. Schedule restores, time them, and practice the promotion path before the error log teaches you the coordinates of your sessions table.
Where MonPG stands on MySQL
I build MonPG, so the honest disclaimer: MonPG monitors PostgreSQL today, and MySQL support is in active development, not shipped. Corruption events announce themselves in observable signals — crash-looping instances, error-log patterns, replication channels that stop with checksum-related errors — and alerting on those instead of on "the site is down" is exactly the kind of early warning the MySQL work aims to deliver. The MySQL monitoring (coming soon) page tracks it as it lands. Meanwhile the same monitoring philosophy runs today on the PostgreSQL side, and the rest of these field notes live on the blog.