PostgreSQL archive_command Failures: The Slow-Motion Disk Full

At 02:11 on a Tuesday, our security automation rotated the object-store credentials that wal-g used to push WAL. Every archive_command from that second on exited 1. Nothing paged, nothing slowed, no query failed — the database kept humming while pg_wal quietly accumulated sixteen megabytes at a time. By 07:40 there were 380 gigabytes of WAL on a 500-gigabyte volume, and the only thing standing between us and a full-disk outage was a filesystem alert that had nothing to do with Postgres. archive_command failure is the most patient incident PostgreSQL produces: it gives you hours of warning, written plainly, in a statistics view almost nobody graphs.

What exactly happens when archive_command starts failing?

The archiver marks the segment it was working on as failed and retries that same segment forever — oldest first, with only the odd out-of-order file after a promotion or crash as an exception — so every newer segment queues behind it while pg_wal grows without bound. Two properties of the design make this worse than it sounds. First, there is no skip and no circuit breaker: a command that fails because one specific segment's destination object already exists will fail at 03:00, at 04:00, and next March, and nothing behind it moves. Second, WAL recycling is gated on archiving — a segment cannot be removed until the archiver has shipped it — so the write path's byproduct becomes the disk's problem at exactly the database's write rate. At two gigabytes of WAL per hour with a hundred and fifty free, you have days. During a bulk load at forty an hour, you have until lunch. The failure stays silent because archiving is deliberately out of the query path; Postgres trades your safety margin for uptime without asking.

How do you spot it in pg_stat_archiver?

The signature is failed_count climbing while last_failed_time stays newer than last_archived_time, and last_failed_wal names the poison segment verbatim. The view is a single row and the check is a single query:

SELECT archived_count, failed_count,
       last_archived_wal, last_archived_time,
       last_failed_wal, last_failed_time
FROM pg_stat_archiver;

SELECT count(*) AS wal_files,
       pg_size_pretty(sum(size)) AS pg_wal_bytes
FROM pg_ls_waldir();

The alerting rule I run: page when now() - last_archived_time exceeds fifteen minutes and failed_count has increased over the window — the age catches a dead-stuck archiver, the delta catches a flapping one; on a busy server the delta moves within a minute of the first failure, and the age bounds worst-case detection at fifteen minutes even on a quiet one. Comparing last_archived_wal against the current segment tells you backlog depth in segments, and pg_ls_waldir tells you the same story in bytes. Since PostgreSQL 15 you can also replace the shell command with an archive_library loaded in-process, which removes a whole class of fork-and-exec failure modes; most of us still run wal-g or pgBackRest as commands, and the retry semantics above apply either way.

How do replication slots and wal_keep_size make the pile-up worse?

They are a second, independent retention pin on the same disk — a slot's restart_lsn and wal_keep_size both force WAL to be kept even when archiving is perfectly healthy, so when pg_wal grows you must check both governors before blaming either. The distinction matters for the runbook. An archiving failure retains everything since the stuck segment, and the fix is unclogging the archiver. An inactive slot retains everything since its restart_lsn, archiving or not, and the only fixes are advancing or dropping it — I once watched a deactivated CDC slot pin ninety gigabytes on a cluster whose archiver was fine. Check them together:

SELECT slot_name, active,
       pg_size_pretty(
         pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
       ) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

wal_keep_size is the gentler cousin: it guarantees a floor of retained WAL for standbys that lost their slot, sized by you and bounded by you. Slots are unbounded by default — max_slot_wal_keep_size can cap them, at the price of invalidating a slot that falls too far behind, which trades a disk emergency for a replica rebuild. The replication monitoring notes go deeper on slot hygiene; the point here is that pg_wal pressure has three discretionary authors — the archiver, the slots, the keep floor — on top of the checkpoint-to-checkpoint retention crash recovery always needs, and they share one disk.

What is archive_timeout actually trading?

It forces a segment switch once N seconds have passed with any activity since the last one — a lone checkpoint counts, so a truly idle server does not mint empty segments on the timer — which bounds how much unarchived WAL can sit inside the latest partial segment — in other words, it converts your recovery-point exposure from "however long a quiet database takes to fill sixteen megabytes" into a number you chose. The cost is that segments closed early by the timeout are archived at full length anyway, so a very low value inflates archive volume with mostly-empty files; the zeros compress to nearly nothing on disk but still count as objects and as restore bookkeeping. I run 60 to 300 seconds depending on write volume, and I treat 0 — off — as a deliberate RPO statement rather than a default. On a quiet dev database a two-minute archive_timeout writes more archive traffic than the workload does, which is the honest reason people disable it, and the honest reason production should not.

What is the emergency unclog runbook?

Unclog the archiver and let PostgreSQL chew through its own backlog — and never, under any pressure, delete files from pg_wal by hand, because the crash-recovery and replication machinery you will need at 03:00 reads exactly those files. The sequence that has served me:

• Read last_failed_wal and the archiver errors in the log; the usual causes are a full destination disk, expired credentials, a hung network mount, or a filename collision on the target.
• If the real destination needs time, repoint archive_command at a local fallback directory on a different volume with ALTER SYSTEM and pg_reload_conf() — the setting is reloadable, no restart, and the archiver starts draining immediately.
• Watch archived_count climb and pg_wal bytes fall; a healthy archiver clears a large backlog far faster than it accumulated.
• Ship the fallback files to the real destination, verify with your backup tool's archive check, then restore the original command and reload again.

ALTER SYSTEM SET archive_command = 'cp %p /wal_fallback/%f';
SELECT pg_reload_conf();

One temptation to refuse out loud: pointing archive_command at /bin/true. It unblocks the disk instantly by silently severing your PITR chain, and the day you do it is the day you will need the chain. If it ever happens, the only honest follow-up is an immediate fresh base backup and a written note that the continuous chain is severed at that point — older backups still recover up to the gap, but nothing replays across it. Simpler to never.

Catching it in minutes with MonPG

This incident is entirely visible in four series: the failed_count delta, the age of last_archived_time, pg_wal bytes on disk, and per-slot retained WAL. MonPG graphs all four as part of its PostgreSQL monitoring, and the alert described above — archiver age plus failure delta — is a two-condition rule, the kind you set once and forget until the night it saves you. The broader WAL picture, including the checkpoint behavior that decides how fast pg_wal turns over when archiving is healthy, pairs with the WAL monitoring notes. The 02:11 credential rotation would have paged by 02:16. Instead it got five free hours — which is the whole argument for graphing one-row statistics views.