The page fired at 3:12 on a Tuesday: pg_wal at 100 percent, primary refusing connections, every write in the queue timing out. The database had been healthy for months. Checkpoints were spread, max_wal_size was generous, archive_command was succeeding. What killed it was a replication slot named debezium_poc — created eight weeks earlier for a change-data-capture proof of concept, abandoned when the project was shelved, and faithfully retaining every byte of WAL produced since. That was 380 GB of write-ahead log held for a consumer that no longer existed. Dropping the slot brought the volume from 100 percent to 11 percent within one checkpoint cycle. Nothing else was wrong. Nothing else had to change.
Replication slots exist because of a promise: a standby or a logical decoding client can disconnect, come back later, and re-read everything it missed. PostgreSQL keeps that promise absolutely. The slot records restart_lsn, the oldest position the consumer might still need, and the primary refuses to recycle any segment containing bytes at or after it. The promise does not care whether the consumer is ever coming back. An inactive slot is not a broken thing that will time out; it is a standing order to retain WAL forever, and the primary will fill its last free block before it breaks that order.
Why does one inactive slot fill the whole disk?
Because WAL retention for a slot has no ceiling. The recycling rule is simple: a segment can be removed only when everything it contains is older than every slot's restart_lsn and older than what any standby's wal_keep_size asks for. One slot with a two-month-old restart_lsn vetoes removal for all of it. And the knob people reach for, max_wal_size, does not help — it is a checkpoint trigger, not a quota. It tells the checkpointer to flush more often so recycling can happen sooner, but recycling is exactly what the slot forbids. I have watched max_wal_size sit at 20 GB while pg_wal grew past 300 GB; the setting was working perfectly and the disk filled anyway.
The growth rate is whatever your write rate is. A quiet OLTP primary might emit 20 GB of WAL a day and take weeks to fill a large volume, which is why these incidents smolder. The same incident during a backfill or an index build can do it in a weekend. The deceptive part is that everything looks fine meanwhile: replication green, checkpoints green, live standbys green. The only unhealthy number is one most default dashboards never chart.
How do you read pg_replication_slots when the disk is screaming?
Start with one query and look at three columns: active, restart_lsn, and the distance from restart_lsn to the current insert position. An inactive slot with a large distance is your culprit, full stop.
SELECT slot_name, slot_type, active, restart_lsn, wal_status,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
pg_size_pretty(safe_wal_size) AS headroom
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;The columns earn their keep under pressure. active = false means no client is connected; for a physical slot that can be transient during a reconnect, but inactive for hours is a smell, and on current releases inactive_since timestamps when it went quiet. restart_lsn is the pin: everything from there forward is retained. wal_status tells you the slot's relationship with max_slot_wal_keep_size: reserved and extended are healthy, unreserved means the slot has already lost its guaranteed headroom and could be invalidated, and lost means it already was — the consumer's next connection will get an error and the data it needed is gone from this primary's WAL. safe_wal_size is the count that matters when a cap is set: how many more bytes can be written before the slot is sacrificed.
For logical slots, also glance at confirmed_flush_lsn — the position the client has acknowledged — and at catalog_xmin, the horizon the slot holds back for decoding; more on that below. If the retained_wal number dwarfs everything else on the volume, the fix is one command, and you should not be sentimental about it.
SELECT pg_drop_replication_slot('debezium_poc');Space returns at the next checkpoint, when the now-unpinned segments are recycled or removed. If you are truly at 100 percent, also check pg_stat_archiver for a climbing failed count — a full pg_wal wedges the archiver too.
Is max_slot_wal_keep_size the circuit breaker you actually want?
For every slot that is not a hard HA requirement, yes — it converts a primary-down outage into a rebuild-one-replica event, and that is the right trade almost every time. The default, -1, means unlimited: given the choice between disappointing a subscriber and dying, the primary dies. I consider that the wrong default for production, but it is the default we have, so the setting is on you.
Set the cap to the amount of WAL you would rather regenerate a replica from than lose the primary over, which in practice means comfortably below your free-space budget. If the volume gives pg_wal 200 GB of room, a per-slot cap of 60 to 80 GB leaves slack for normal churn plus a checkpoint spike. Two mechanics to understand before you set it. First, it is evaluated per slot at checkpoint and segment-recycling time, so it is an invalidation threshold, not a reservation — the bytes are already on disk before the cap is consulted, a write burst between checkpoints overshoots it, and nothing in the cap accounts for what wal_keep_size or a stalled archiver is retaining on top. It complements monitoring rather than replacing it. Second, invalidation is final: wal_status flips to lost, the next connection attempt fails, and the standby or subscriber must be rebuilt from a fresh base backup or snapshot. Have the rebuild runbook ready; the whole point of the cap is choosing the cheaper failure, so make sure it is actually cheap.
Why do logical slots stall hardest around long transactions?
Because a logical slot pins two horizons, not one, and long transactions hold both of them open. restart_lsn cannot advance past WAL that decoding still needs, and decoding still needs everything from the start of the oldest in-flight transaction it might have to decode once that transaction commits. A nightly batch job that runs inside a single five-hour transaction therefore guarantees five hours of WAL accumulation no matter how fast the consumer drains, because none of that WAL can be acknowledged as consumed until the transaction commits and is decoded. During our backfill weekends I watched a healthy, fully caught-up Debezium consumer sit at zero lag while its slot's retained WAL grew at the write rate of the batch job — the consumer was fine, the horizon was not.
The second horizon is catalog_xmin, and it bites somewhere people do not watch. Logical decoding reconstructs tuples using the system catalogs, so the slot pins the oldest transaction ID whose catalog rows it might still need. VACUUM cannot remove dead rows from pg_catalog tables older than that horizon. Run a long transaction for days alongside an active logical slot and dead tuples pile up in the catalogs themselves — the bloat mechanics you know from user tables, applied to pg_class, on tables most autovacuum dashboards never chart. The vacuum half of that story is covered in the vacuum guide; here the point is that an old slot plus a long transaction degrades the catalogs, quietly. Prepared transactions make it worse in a subtler way: with two_phase enabled the slot must retain WAL until the prepared transaction commits, and forgotten prepared transactions are their own genre of incident.
What should you alert on instead of disk percentage?
Alert on retained WAL per slot, in bytes and in age, and on the active flag — disk percentage is the last symptom, not the first. Concretely: page when an inactive slot's restart_lsn falls more than a few gigabytes behind the insert position, page when active = false persists longer than your longest legitimate reconnect window, and warn when safe_wal_size shrinks past half its cap. Those three rules give you days of warning, because retained WAL grows at the write rate you already have dashboards for. Disk percentage, by contrast, sits at 60 percent for a month and then gives you an afternoon.
One habit that pays for itself: require a ticket or an expiry date for every slot created outside automation. Experimental slots created by humans are the ones that fill disks; platform-owned slots have owners and runbooks. A weekly review of pg_replication_slots takes thirty seconds and has caught more pending incidents for me than any other single query.
Watching replication slots with MonPG
MonPG monitors PostgreSQL in production today, and slot health is exactly the kind of leading indicator it is built to graph: restart_lsn lag per slot rendered as retained bytes and as time, the active flag's state changes, and safe_wal_size headroom when a cap is configured, so an abandoned consumer shows up as a steadily climbing area chart days before pg_wal is in danger. The PostgreSQL monitoring surface covers how those numbers are collected and alerted on, and the notes on WAL and checkpoint tuning pair well with this one for sizing the volume the slots are defending. Set the cap, graph the retention, and the 3 a.m. page becomes a Tuesday-afternoon ticket.