The slowest database incidents are the ones that build for six hours before anyone notices. InnoDB purge lag is the classic example. Nothing errors. No query fails. The server just gets gradually, mysteriously slower, undo tablespaces swell, and by the time someone looks, a SELECT that took 5 ms this morning takes 400 ms because it is wading through a million delete-marked row versions. The whole time, one number was climbing in plain sight: history list length.
This is a field guide to that number on MySQL 8.0 and 8.4: what it measures, why a single idle session can pin it, how purge and undo truncation actually work, and what I alert on.
What the history list actually holds
InnoDB is an MVCC engine. Every UPDATE and DELETE writes the previous row version into undo logs, so that older transactions can still read the world as it looked when they started. When a transaction commits, its undo records are not immediately deletable, because some open read view might still need them. Committed-but-not-yet-purgeable undo gets linked into the history list, and background purge threads walk that list, freeing undo records and physically removing delete-marked rows once no read view can possibly need them.
History list length (HLL) is the count of undo log units waiting for purge. It is InnoDB's equivalent of PostgreSQL's dead-tuple and oldest-xmin problem, with one big architectural difference: InnoDB stores old versions out-of-line in undo logs rather than in the table heap, so lag shows up as undo growth and version-chain traversal cost rather than table bloat. I unpacked that comparison in MySQL history list vs PostgreSQL vacuum; if you operate both engines, the failure shapes rhyme.
A healthy busy server usually keeps HLL in the thousands to low hundreds of thousands, sawtoothing as purge catches up. The absolute number matters less than the trend: a value that climbs monotonically for an hour means purge is not keeping up, or something is not letting it.
Reading the canary
The oldest way to see HLL is SHOW ENGINE INNODB STATUS, in the TRANSACTIONS section. For monitoring, the metrics table is cleaner:
SELECT name, count
FROM information_schema.innodb_metrics
WHERE name = 'trx_rseg_history_len';That counter is enabled by default and is cheap to poll. Trend it every 15 to 60 seconds. While you are there, watch your undo tablespaces too, because sustained HLL growth turns into disk growth with a lag:
SELECT t.tablespace_name,
f.file_name,
round(f.total_extents * f.extent_size / 1024 / 1024, 1) AS size_mb
FROM information_schema.files f
JOIN information_schema.innodb_tablespaces t
ON f.file_id = t.space
WHERE t.space_type = 'Undo';Undo growth is the lagging indicator, HLL is the leading one. By the time disk alerts fire, you have often been lagging for hours.
The classic incident: one idle REPEATABLE READ session
Here is the shape I have seen more than any other. MySQL's default isolation level is REPEATABLE READ. A session opens a transaction, runs one innocent SELECT, and then goes idle: a developer's forgotten shell, an application that crashed between BEGIN and COMMIT, an ORM holding a connection with autocommit off, a stuck report job. That first SELECT created a read view pinned to that moment. From then on, purge cannot remove any row version newer than that snapshot, because the idle session might still, in theory, read it.
The transaction does nothing. It holds no row locks anyone is waiting on, so lock monitoring stays green. It is not slow, so slow query analysis stays green, the same blind spot I described in MySQL slow query log vs pg_stat_statements: the queries doing damage are often the ones that finished hours ago. Meanwhile every UPDATE on the server adds to a history list that can no longer drain. Version chains on hot rows get long, and every read of those rows walks the chain backwards through undo to reconstruct the visible version. Secondary index reads get worse still, because delete-marked entries cannot be removed and range scans wade through garbage. CPU climbs, latency climbs, undo files grow, and the root cause is a connection that has executed nothing since 9 a.m.
Finding it is a two-table join away:
SELECT trx.trx_id,
trx.trx_started,
timestampdiff(SECOND, trx.trx_started, now()) AS open_seconds,
trx.trx_rows_modified,
p.id AS processlist_id,
p.user, p.host, p.command, p.time AS idle_seconds
FROM information_schema.innodb_trx trx
JOIN information_schema.processlist p ON p.id = trx.trx_mysql_thread_id
ORDER BY trx.trx_started
LIMIT 10;An old trx_started with command Sleep is your smoking gun. KILL the connection and watch HLL start to fall. It will not fall instantly; purge has a backlog to chew through, and that is normal.
Purge threads and the pressure valves
Purge runs on dedicated background threads, controlled by innodb_purge_threads, working in batches of innodb_purge_batch_size undo pages. On most 8.0 servers the defaults are fine, and I rarely see purge fall behind on raw throughput alone; when HLL climbs, a pinned read view is the cause far more often than purge being underpowered. Before adding purge threads, prove the threads are actually busy and still losing ground rather than blocked behind a snapshot.
There is also a blunt instrument: innodb_max_purge_lag. When HLL exceeds it, InnoDB starts delaying DML operations (up to innodb_max_purge_lag_delay microseconds each) to let purge catch up, deliberately trading write latency for version hygiene. It defaults to 0, meaning off, and I mostly leave it there. Throttling every writer because one session is holding a read view punishes the innocent, and the delay does nothing to fix a pinned snapshot anyway. It earns its keep only on write-hammered systems where purge genuinely cannot keep pace even with no long transactions in sight.
Undo tablespaces: growth and truncation
MySQL 8.0 finally gave undo its own dedicated tablespaces (undo_001 and undo_002 by default), and they can be truncated. With innodb_undo_log_truncate enabled, which it is by default, any undo tablespace that grows past innodb_max_undo_log_size (1 GiB by default) becomes a truncation candidate: InnoDB marks it inactive, waits for purge to release its contents, then shrinks it back to its initial size. Truncation requires at least two active undo tablespaces so one can take traffic while the other truncates, which is why the default is two and why you should think twice before dropping down to fewer.
Two operational notes from experience. First, truncation only reclaims space after purge has fully drained that tablespace, so if HLL is pinned, truncation silently cannot proceed; fix the pin first. Second, on write-heavy systems the truncate cycle itself has a cost, and an aggressive innodb_max_undo_log_size causes constant churn. I would rather give undo a generous ceiling and alert on deviation than force the engine to shrink files all day. You can also add capacity or retire tablespaces online with CREATE UNDO TABLESPACE and ALTER UNDO TABLESPACE ... SET INACTIVE, which is handy when undo lives on its own volume.
What I actually alert on
Three alerts cover almost every purge incident I have met. First, HLL trend: warn when trx_rseg_history_len exceeds a workload-appropriate ceiling, mine is usually in the low millions, and page when it has climbed continuously for 30 minutes regardless of level. Second, oldest open transaction age: warn at minutes, page at an hour, from information_schema.innodb_trx. This catches the idle-session incident before HLL even looks bad, and it is the single highest-value MySQL alert I know that almost nobody sets. Third, undo tablespace size deviation from baseline, as the backstop for everything the first two miss.
Do not alert on instantaneous HLL spikes. A big batch DELETE legitimately queues millions of undo records and purge grinds them down in minutes. The signal is sustained growth, not amplitude.
Where MonPG stands on this
Full disclosure: MonPG is a PostgreSQL monitoring platform today, and it does not monitor MySQL yet. We are building MySQL monitoring (coming soon) right now, and this exact incident shape is one of its design anchors: history list length trended as a first-class series, oldest-transaction age wired to the session that owns it, and undo growth correlated against both, so the answer to "why is everything slowly getting slower" is one screen instead of four SHOW commands under pressure. Purge lag and PostgreSQL's vacuum debt are the same disease in two dialects, and we have already built the workflow once. If PostgreSQL is also part of your fleet, you can start there today; the MySQL side is coming.