Disk-full is the MySQL emergency with the widest gap between how simple it sounds and how badly it can go. The fix seems obvious - delete something - and that is exactly the moment someone removes a file InnoDB cannot live without, converting a stalled server into a corrupted one.
I have been on both good and bad versions of this incident. The good version freed 40 GB of binary logs in ninety seconds with one PURGE statement. The bad version involved a well-meaning engineer who removed binlog files with rm, leaving the binlog index pointing at ghosts. This playbook is the difference between the two, written for MySQL 8.x on InnoDB.
What a full disk does to a running MySQL
MySQL does not fail cleanly at 100 percent. For ordinary data writes, the documented behavior is to wait for free space and retry roughly once a minute, so sessions hang mid-write rather than erroring. From the application side this looks like a latency wall, not a disk problem, which costs you diagnosis time.
The binary log is harsher. If mysqld cannot write the binlog, binlog_error_action decides what happens, and the default ABORT_SERVER shuts the server down rather than silently continuing without replication history. So a full disk on the binlog volume can take the whole server down, not just slow it. Replicas add their own wrinkle: a replica that cannot write relay logs stalls replication, and the lag you see downstream started as a disk problem upstream.
The takeaway: treat 90-plus percent disk as an active incident even while queries still succeed, because the failure you are heading toward is a hang or an abort, not a graceful error.
What actually grows
Almost every MySQL disk-full I have seen came from one of five sources.
Binary logs are the leading cause. In 8.0 the default retention is binlog_expire_logs_seconds = 2592000 - thirty days - and a write-heavy workload can generate tens of gigabytes a day. Teams that never aligned retention with their actual backup and replica needs are carrying weeks of logs they will never use.
Undo grows when long-running transactions force InnoDB to retain old row versions. 8.0 keeps undo in dedicated tablespaces with innodb_undo_log_truncate enabled by default, but truncation cannot reclaim what an open transaction still needs - one forgotten REPEATABLE READ session from a week-old screen session can pin enormous undo growth.
Temporary space grows in two places: the global temp tablespace ibtmp1, which autoextends under spill-heavy workloads and only shrinks on restart, and session temporary tablespaces. Queries that spill drive this - MySQL internal temp tables on disk covers why a GROUP BY over a text column can quietly write gigabytes.
Logging left on is the embarrassing one: a general_log enabled during a debugging session and forgotten will out-grow everything else on a busy server. Slow query logs with long_query_time set to 0 do the same.
Finally, replicas accumulate relay logs when the SQL thread falls behind the I/O thread.
Find the consumer before you delete anything
Sixty seconds of diagnosis prevents the catastrophic rm. On the filesystem, du against the data directory sorted by size answers most of it. From inside the server, two queries cover the rest.
SHOW BINARY LOGS;
SELECT NAME,
ROUND(FILE_SIZE / 1024 / 1024 / 1024, 2) AS size_gb
FROM information_schema.INNODB_TABLESPACES
ORDER BY FILE_SIZE DESC
LIMIT 15;SHOW BINARY LOGS totals the binlog footprint. The tablespace query separates real table growth from undo tablespaces and the temp tablespace, which show up by name. Also check whether anyone left verbose logging on:
SELECT @@general_log,
@@general_log_file,
@@slow_query_log,
@@long_query_time,
@@binlog_expire_logs_seconds;What is safe to purge
Binary logs are the big safe win, with one precondition: nothing still needs them. Check every replica's SHOW REPLICA STATUS and confirm which source log file each is reading (Relay_Source_Log_File), and confirm your point-in-time backup tooling has shipped the logs it needs. Then purge through the server, never through the filesystem:
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;PURGE updates the binlog index atomically and refuses to remove logs in use. Deleting binlog files with rm leaves the index inconsistent, which can prevent a clean restart - this is the single most common self-inflicted wound in this incident.
Also safe: turning off general_log and removing or truncating its file, rotating the slow log (SET GLOBAL slow_query_log = OFF, move the file, FLUSH SLOW LOGS, re-enable), deleting old backup staging files and core dumps, and killing the long transaction that is pinning undo growth - killing the session lets truncation eventually reclaim the space. ibtmp1 is reclaimed only by a restart, which you schedule rather than perform mid-incident. Relay logs on replicas mostly manage themselves - relay_log_purge is on by default and removes them as the applier finishes - so a large relay log footprint is really an applier-lag problem; fix the lag rather than touching the files.
One reclamation trap deserves its own warning. Deleting rows does not shrink .ibd files; the space is only handed back to the filesystem by a table rebuild (OPTIMIZE TABLE or ALTER TABLE ... FORCE), and a rebuild needs free space for a full copy of the table. On a nearly full disk, the operation that reclaims table space is exactly the operation you no longer have room to run. That paradox is the single best argument for acting at 80 percent instead of 95.
What you must never delete
The never list, explicitly: ibdata1 (the system tablespace), anything in #innodb_redo (redo logs, 8.0.30+ layout) or the older ib_logfile files, undo tablespaces, any .ibd file, and the binlog index file. Deleting any of these on a live server risks a database that cannot start, and "restore from backup" becomes the recovery plan. Redo logs look temptingly large and regenerable; they are the crash-recovery journal, and removing them while mysqld is running is how you manufacture corruption.
The safe generalization: on an emergency, delete nothing inside the data directory by hand. Free space via SQL (PURGE), via configuration (logs off), or outside the data directory entirely.
Emergency order of operations
When the pager fires at 95 percent: first, buy headroom outside MySQL - old logs, package caches, forgotten dumps on the same volume; even a few GB converts a hang-imminent server into a stable patient. Second, run the sixty-second diagnosis above. Third, apply the biggest safe purge, usually binlogs. Fourth, kill undo-pinning transactions if that is the growth source. Fifth, if the volume can be grown online (cloud block storage usually can), grow it in parallel with the purging, not instead of it - the same workload will refill the same disk. Last, write down what grew and why; a disk-full without a postmortem is a scheduled recurrence.
Prevention is an alerting problem
Every one of these incidents is visible days in advance. The alert set that works: disk usage thresholds at 80 and 90 percent with the 80 treated as a ticket, not noise; growth-rate alerting - "at the current rate this volume is full in 72 hours" catches the binlog pile-up a static threshold misses; per-component tracking of binlog totals, undo tablespace size, ibtmp1 size, and relay log footprint so the alert names the consumer; and a standing check that general_log is off in production. Align binlog_expire_logs_seconds with the real recovery requirement and most fleets can cut it well below thirty days.
Where MonPG fits
MonPG is a PostgreSQL monitoring platform, and MySQL support is being built now - per-component disk growth, binlog retention visibility, and forecast-style alerts of exactly the kind this playbook depends on are part of that design. To be clear: MonPG does not monitor MySQL today. The MySQL monitoring (coming soon) page tracks what is planned. If PostgreSQL is also in your fleet, you can start there - the disk-full story on the Postgres side (WAL retention, replication slots) is a sibling of everything above.