PostgreSQL Replication Slots: When Retained WAL Fills the Disk

The primary PANIC'd at 03:12 on a Saturday. The error was could not write to file "pg_wal/xlogtemp.24188": No space left on device, and by the time I opened the volume, pg_wal alone was holding 214 GB on a 250 GB disk. Write traffic that weekend was unremarkable — a few gigabytes a day, the usual background hum. The WAL had not grown because we were writing more. It had grown because PostgreSQL was not allowed to delete it. Three weeks earlier we had decommissioned a standby, and nobody dropped its physical replication slot. The slot sat there, inactive, its restart_lsn frozen at the moment the standby last acknowledged WAL, and PostgreSQL faithfully kept every segment from that point forward. A quiet weekend is exactly when this kills you: nobody is watching the dashboards, the consumer never comes back, and the one process that could clear the backlog is the one that no longer exists.

We recovered in about forty minutes, then spent the next week making sure it could never happen silently again. This article is the playbook: the mechanism, the monitoring queries, the circuit breaker, the recovery sequence, and why logical slots are worse than physical ones.

Why does a dead replication slot fill the disk?

A replication slot pins WAL: PostgreSQL refuses to remove any WAL segment newer than the slot's restart_lsn, whether or not any consumer is connected. That is the entire mechanism.

Without slots, WAL is recycled aggressively: once a segment has been checkpointed and no standby still needs it, it is renamed and reused, and pg_wal stays roughly bounded by checkpoint cadence. A slot changes the contract. When you create one, PostgreSQL records a restart LSN — the oldest WAL position the consumer still needs. From then on, every checkpoint checks every slot: if a segment contains WAL at or after any slot's restart_lsn, that segment is retained, in pg_wal, on the primary's local disk. The guarantee is deliberate — it is what lets a standby reconnect after hours of downtime and resume streaming instead of needing a full rebuild. The failure mode is equally deliberate: if the consumer never comes back, the guarantee has no expiry date. WAL accumulates at exactly your write rate, forever, until the disk is full. And when pg_wal cannot be written, PostgreSQL does not degrade gracefully — it PANICs and shuts down, because refusing to write WAL would mean abandoning durability. Our 214 GB was three weeks of modest writes that one dead slot made immortal.

The promise is two function calls: create deliberately, drop the moment the consumer goes away — and the drop belongs in the same decommission runbook, not in someone's memory.

-- Physical slot for a streaming standby
SELECT * FROM pg_create_physical_replication_slot('standby_eu_1');

-- Logical slot for a CDC consumer (e.g. a Debezium-style pipeline)
SELECT * FROM pg_create_logical_replication_slot('cdc_orders', 'pgoutput');

-- Dropping a slot (must not be in use by an active connection)
SELECT pg_drop_replication_slot('standby_eu_1');

pg_drop_replication_slot fails if a connection is actively using the slot, so the decommission order is: stop the consumer, confirm the slot is inactive, then drop. Drops run on the primary only. What got us was a process gap: the standby teardown was automated, the slot cleanup was a manual checklist item, and it got skipped. Slot lifecycle belongs in consumer lifecycle — in Terraform, in the decommission playbook, wherever the consumer is deleted.

How do you find runaway slots before they fill the disk?

You measure slot lag in bytes — the distance between the server's current WAL position and each slot's restart_lsn — and you alert on it, because that number is exactly how much WAL the slot forces you to keep. The view is pg_replication_slots, and the query we now run everywhere:

SELECT slot_name,
       slot_type,
       active,
       pg_size_pretty(
         pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
       ) AS retained_wal,
       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes,
       wal_status,
       safe_wal_size
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

The columns that matter:

• active — false means nothing is consuming this slot right now. An inactive slot with growing retained_bytes is the incident. An inactive slot on a weekend is the incident you find on Monday.
• retained_wal — the human-readable lag, and the number to alert on. Our thresholds: warn at 10 GB retained, page at 50 GB, and treat any inactive slot older than a few hours as guilty until proven otherwise.
• wal_status — reserved or extended is fine; unreserved means the slot has exceeded max_slot_wal_keep_size and some required WAL may already be gone; lost means the slot is dead and the consumer must resync from scratch.
• safe_wal_size — how many more bytes can be written before the circuit breaker invalidates this slot (NULL when no limit is set). Watching it trend toward zero tells you a slot is about to be killed, which is its own alert-worthy event.

One caveat: pg_wal_lsn_diff measures distance in WAL bytes, not files on disk — checkpoint retention and wal_keep_size add overhead, so treat it as a floor, and pair it with a filesystem alert on the pg_wal volume. Slot lag belongs on the same graph as replication lag — the same failure with a different consumer, covered in our post on PostgreSQL replication monitoring.

What does max_slot_wal_keep_size actually protect you from?

It caps how much WAL any slot may force PostgreSQL to retain: past that limit, the slot is invalidated instead of the disk filling and the primary PANICing. It is a circuit breaker, and like every circuit breaker, it sacrifices the downstream to save the house.

ALTER SYSTEM SET max_slot_wal_keep_size = '50GB';
SELECT pg_reload_conf();

The tradeoff deserves to be stated plainly. With the default of -1 (unlimited), a dead consumer can fill the disk and take the primary down — exactly what happened to us. With a limit set, the same dead consumer causes the slot to be marked lost; the primary survives untouched, but the replica or CDC pipeline behind that slot can no longer resume from its position. A standby must be rebuilt with a fresh base backup. A logical consumer must resnapshot its tables. You are choosing between "the primary dies" and "the consumer resyncs," and the consumer resyncing is almost always the right choice — replicas and pipelines are rebuildable, primaries are not.

Size it from your rebuild story, not your disk. If re-snapshotting a logical subscriber takes four hours and your write rate is 2 GB/hour, a 50 GB cap gives you a full day of slack before invalidation. Whatever number you pick, keep the alerting below it: the cap is the last line of defense, not the monitoring strategy. If slots routinely hit the cap, your consumers are broken, not your limit.

What do you do when the disk is already full and Postgres is down?

You free space without touching pg_wal itself, start PostgreSQL, then drop the dead slot so checkpoints can reclaim the retained WAL — in that order, and never by deleting files from pg_wal by hand. The sequence that worked for us:

1. Free space outside pg_wal. We compressed old logs on the same volume and moved archived WAL segments that a misconfigured archive script had left on the data volume. Even a few hundred megabytes is enough for startup. The one absolute rule: do not delete, rename, or "clean up" files inside pg_wal. Those segments may be exactly what recovery needs, and manual deletion is how a full disk becomes corrupted data.

2. Start PostgreSQL. With space available, the server starts, replays WAL, and comes up. Connections from consumers may immediately try to reattach — that is fine.

3. Identify and drop the dead slot. Run the monitoring query from above, confirm the slot's active is false and its consumer is genuinely decommissioned (not merely rebooting — we checked with the infra team before dropping), then:

SELECT pg_drop_replication_slot('standby_eu_1');

4. Let checkpoints do the cleanup. Dropping the slot does not instantly free the disk; the next checkpoint removes the now-unpinned segments. Force it if you are impatient: CHECKPOINT; and watch pg_wal shrink back to normal. Ours went from 214 GB to under 3 GB in about a minute.

5. Fix the process. The incident review produced exactly two action items: the slot-lag alert, and making slot cleanup a step in the decommission automation. Hardware upgrades were not on the list; a bigger disk would only have moved the PANIC to a later weekend.

Why are logical replication slots even more dangerous?

Logical slots pin WAL exactly like physical ones, but they additionally pin the system catalog via catalog_xmin, and they survive failovers and major upgrades poorly — two extra failure modes physical slots do not have.

The catalog pinning is the subtle one. A logical slot decodes changes using the catalog as it existed when the slot's oldest unconsumed transaction ran, so PostgreSQL cannot vacuum away catalog rows that transaction could still see. While it lags, pg_class, pg_attribute, and friends accumulate dead tuples that autovacuum may not remove. We have seen a forgotten Debezium slot grow pg_attribute to tens of gigabytes on a DDL-heavy database — bloat no regular VACUUM can fix, because the slot holds the horizon. The tell is catalog_xmin in the same view. If you have ever chased mysterious catalog bloat, the query patterns in our long transactions guide apply here too — a lagging logical slot behaves like an ancient transaction that never commits.

The lifecycle problem is the second tax. Logical decoding state is tied to the specific instance; historically, logical slots did not follow a promoted standby after failover, and major-version upgrades do not carry them across, so consumers must be re-created and re-snapshotted at exactly the moments your team is busiest. Newer PostgreSQL versions have improved slot failover support, but it requires deliberate configuration — verify it on your actual version before relying on it. The operational conclusion is unchanged: logical slots need even tighter lag alerting than physical ones, because they can hurt you through two independent mechanisms and are harder to reconstruct after an incident.

Watching replication slots with MonPG

The reason this class of incident keeps recurring is that nothing about a dead slot looks wrong on the standard graphs: CPU is idle, queries are fast, replication lag on the surviving replicas is zero. The only signal is WAL retention growing, one slot at a time. In MonPG we surface per-slot retained-WAL bytes next to the pg_wal volume usage and streaming lag graphs, so an inactive slot shows up as a divergence: disk climbing while throughput stays flat. Alerts can fire on retained bytes per slot and on wal_status leaving reserved, which catches both the slow leak and the moment a circuit-breaker cap starts chewing into safe_wal_size. That alert is what now pages us before the weekend does.

The rule to leave with: monitor slot lag with the same seriousness as replication lag, set max_slot_wal_keep_size as a guardrail, and treat every slot as a subscription that must be cancelled when the consumer dies. The database will keep its promise to retain WAL. Make sure you keep yours to clean it up.