The alert said disk at 97% on the primary, and by the time I shelled in it was at 99% and falling — upward. The datadir held 340 GB of mysql-bin files, eleven months of them, written continuously since a migration that had enabled log_bin for point-in-time recovery and never configured anything to remove them. On that MariaDB version the retention variable expire_logs_days defaulted to 0, and 0 means keep forever. The server had been dutifully archiving every write to disk for almost a year with no consumer: the point-in-time tooling that was supposed to ship the binlogs off-box had been descoped, and nobody noticed because nothing pages when a disk merely fills slowly. The cleanup then produced the second incident of the weekend: two hours after I purged, a staging replica that had been down for three weeks — nobody remembered it existed — came back, failed to find its continuation point, and started screaming Could not find first log file name in binary log index file, ERROR 1236, into an alerting channel that turned out to be wired to production.
Binary log retention is a two-sided problem: keep too much and the disk dies, purge too aggressively and replicas, backups, and flashback tooling die. These are the notes from that weekend and the policy that came out of it.
Why does the binlog grow forever by default?
Because MariaDB's historical contract was that enabling log_bin opts you into managing retention yourself: expire_logs_days defaulted to 0 — never purge — until MariaDB 10.6 changed the game by introducing binlog_expire_logs_seconds with a default of 2,592,000 seconds, exactly 30 days. Three consequences follow that bite mixed fleets. First, any server installed before 10.6 keeps its never-purge behavior across upgrades, while a fresh 10.6 install purges at 30 days, so two "identical" primaries can have opposite retention postures — the same split-fleet trap as the sql_mode default change I covered in the sql_mode strictness notes. Second, when both variables are set, binlog_expire_logs_seconds wins and expire_logs_days is ignored, so a legacy config line can be silently dead. Third, purging only happens on rotation events — when a new binlog file is opened, a FLUSH LOGS runs, or the server restarts — so a server with one enormous quiet binlog may not purge even when configured to. Audit what is actually in force:
-- what retention is configured (10.6+: seconds wins)
SELECT @@binlog_expire_logs_seconds, @@expire_logs_days,
@@log_bin, @@max_binlog_size;
-- the reality on disk: file names carry sequence numbers, and
-- the oldest file's mtime is your true retention window
SHOW BINARY LOGS;
-- how much of the disk is binlog? run on the host, not in SQL
-- du -sh /var/lib/mysql/*bin* | sort -h | tail -20The honest inventory query is just SHOW BINARY LOGS piped to your runbook: file names carry sequence numbers, and the first file's mtime on disk tells you your real retention window. Ours said eleven months against a policy document that claimed fourteen days. The config file and the running server had disagreed since a restart in March, which is the other lesson: verify with SELECT against the variables, not by reading my.cnf.
How do you purge safely without orphaning replicas?
PURGE BINARY LOGS is the only correct way to delete binlogs, and the pre-flight check is confirming every replica has fetched past the point you are about to delete — because MariaDB will happily purge files a lagging or stopped replica still needs, and that replica's recovery becomes a full rebuild. The statement takes either a target file (PURGE BINARY LOGS TO 'mysql-bin.000214') or a timestamp (PURGE BINARY LOGS BEFORE NOW() - INTERVAL 14 DAY), and it deletes strictly older files, updating the index atomically. The workflow that survived: on each replica, read Relay_Master_Log_File from SHOW REPLICA STATUS — the oldest such position across all replicas is your floor; purge to a file safely behind that floor; never, under any circumstance, delete binlog files with rm. Removing files by hand leaves the mysql-bin.index referencing ghosts, and the server repays you with ERROR 1236 on replicas and, in the worst case, a crash-recovery headache that makes the disk-full incident look gentle:
-- on every replica: the oldest fetched position is the purge floor
SHOW REPLICA STATUSG -- note Relay_Master_Log_File and Exec positions
-- on the primary: purge behind that floor, by file or by time
PURGE BINARY LOGS TO 'mysql-bin.000214';
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 14 DAY;
-- force a rotation so the retention setting takes effect now
FLUSH BINARY LOGS;The forgotten-replica problem is organizational, not SQL: the purge floor is only as safe as your inventory of consumers. Our fix was a daily job that reconciles SHOW PROCESSLIST binlog-dump connections against the infrastructure registry and pages when a registered replica has not connected in 24 hours — a replica that stops reading is indistinguishable from a replica that stopped existing, until you purge its continuation point. And keep the purge floor aligned with your backup cadence: binlogs are your point-in-time bridge between full backups, so retention shorter than the backup interval leaves a hole in recoverability that the mariabackup operational guide assumes does not exist. We run 14 days of binlogs against nightly incrementals, and the flashback tooling from the flashback PITR notes needs the same window.
What belongs in a binlog retention policy?
Three numbers, one alarm, and a drill. The numbers: a retention window long enough to cover your slowest legitimate consumer — for us 14 days, sized by backup interval plus a weekend; a disk headroom budget, computed as peak-day binlog volume times retention days plus 40%, because write bursts and maintenance weeks are when retention policies meet physics — our 340 GB incident was a 20 GB/day average hiding a 90 GB batch Saturday; and max_binlog_size left at its 1 GB default unless you have a reason, because gigantic individual files make every downstream tool slower. The alarm: pages at 80% datadir usage with the oldest binlog timestamp in the message, because "disk filling" and "retention broken" are the same incident with different first responders. The drill: quarterly, purge on a staging primary with a deliberately lagged replica attached, and watch what breaks — the first time we ran it, the drill caught that our monitoring replica read binlogs through a proxy whose position reporting was a day stale, which would have become a production rebuild within a quarter.
Multi-source topologies add one wrinkle worth its own sentence: each replication connection consumes independently, so the purge floor is the minimum across all of them, and a single abandoned connection_name from a decommissioned source can pin retention for the whole server. The GTID bookkeeping for that topology is in the multi-source replication notes; the retention consequence is that decommissioning a source must include verifying nothing still fetches from this primary in its name.
Where MonPG fits
The signals worth trending are the ones that would have paged before Saturday: datadir growth rate versus retention policy, oldest binlog age as the true retention window, binlog-dump connection inventory against the registry, and seconds of binlog on disk as a single derived number that collapses the whole policy into a gauge. Full disclosure, as in every article of this series: I work on MonPG, which monitors PostgreSQL in production today and does not monitor MariaDB yet. MariaDB support is coming soon and in active development — the /mariadb-monitoring page tracks where it stands — and WAL/binlog growth with consumer-lag context is exactly the class of signal it is being built around, because that is the PostgreSQL discipline it already applies. Until that ships, the queries above and a disk alarm at 80% are your kit. If PostgreSQL is also in your fleet, that monitoring is live today — see the PostgreSQL overview, or browse more field notes on the blog.