The log line was 03:12:41: server process (PID 28417) was terminated by signal 9: Killed. Then the cascade every PostgreSQL operator learns to dread — this process was the active one, so the postmaster declared a crash, terminated every other backend, and started recovery. Total database downtime was under two minutes, but the connection pile-up behind it took twenty to drain, and the incident review started from a position of ignorance, because PostgreSQL had logged nothing beforehand. No ERROR, no WARNING, no slow statement, nothing to grep for. That silence is the signature: PostgreSQL did not kill anything. The Linux OOM killer did, and the database only found out when its child process vanished.
The query that did it was a nightly reconciliation report running under a role whose work_mem we had raised to 2 GB "just for the big jobs," executing two large hash joins under a parallel plan, on a host whose overcommit settings had never been touched. This post is the accounting I wish we had done before that night: why the kernel chose us, how to budget memory so it never has to choose, and what changes when the database runs in a container.
What does signal 9 in the PostgreSQL log actually mean?
It means an outside actor sent SIGKILL, and on a memory-pressure host that actor is almost always the kernel's out-of-memory killer — PostgreSQL never sends SIGKILL to its own backends, and it has no opportunity to log anything because the process is destroyed synchronously. The database-side evidence is always the same two lines: the signal 9 termination of one backend, followed by the postmaster terminating all remaining server processes and entering crash recovery, because PostgreSQL treats any dead backend as potentially having corrupted shared memory. The proof lives outside the database: dmesg or journalctl -k will contain an "Out of memory: Killed process" entry naming the PID, its RSS, and the score that doomed it. If you take one habit from this article, make it this: any signal 9 in the PostgreSQL log is a kernel-level investigation, not a database-level one, and the first command is journalctl -k, not a grep of the query log.
Two impostors are worth ruling out while you are in there. A monitoring agent or an over-eager runbook script can kill -9 a backend by PID — the journal will show no OOM entry in that case. And a plain SIGTERM from pg_terminate_backend looks similar in application behavior but is logged differently, as a graceful termination, not signal 9.
Why does the kernel pick the database process?
Because the OOM killer's heuristic is essentially "who holds the most memory," and on a database host that is always a PostgreSQL backend — the kernel computes an oom_score per process dominated by resident memory, and your busiest backend at the moment of exhaustion is the biggest legal target. PostgreSQL has known this for years and protects itself in one specific way: the postmaster sets its own oom_score_adj to -1000 on Linux, making itself nearly unkillable, while backends deliberately reset theirs to 0 and take their chances. The design intent is that sacrificing one backend should be survivable. In practice it is only partially survivable: the postmaster lives, but a killed backend still forces the full terminate-everything-and-recover cycle, every connection drops, and in-flight transactions roll back. The protection buys you a clean restart, not continuity, which is why the correct goal is never letting the killer wake up at all.
The moment of exhaustion itself is usually arithmetic, not mystery. Our night went wrong when three things multiplied: work_mem at 2 GB applies per plan node, not per query, and the report had two hash nodes; hash operations on current versions may exceed work_mem by hash_mem_multiplier, 2.0 by default since PostgreSQL 15; and the parallel plan gave each of four workers its own copy of that budget. One report, worst case north of 20 GB, on a host whose freeable memory at 03:12 was a fraction of that. None of these numbers were secrets — they were just never multiplied together until the kernel did it for us.
How do you budget memory so the OOM killer never wakes?
You add up the plausible worst case and make it fit under what the host can actually commit: shared_buffers, plus max_connections times a realistic per-backend envelope, plus maintenance_work_mem times autovacuum_max_workers, plus whatever the OS and everything else on the box needs. The naive formula — max_connections times work_mem — both overcounts and undercounts: most backends use almost nothing, while one reporting backend with parallel hash joins can exceed ten times work_mem, so I budget two numbers, an average case for capacity planning and a worst-plausible case for the overcommit ceiling. The worst-plausible number is the one that matters at 03:12.
The structural fix, and the one I now consider non-negotiable on dedicated database hosts, is vm.overcommit_memory = 2 with vm.overcommit_ratio set so that Committed_AS can never exceed RAM plus a deliberate slice of swap. With strict overcommit, a backend that asks for memory the host cannot honor gets a clean failure — PostgreSQL raises ERROR: out of memory, SQLSTATE 53200, the query dies, and the cluster lives. Without it, the kernel says yes to everyone and picks a victim later, which is strictly worse: you trade a failed query for a full cluster reset. Strict overcommit requires honest accounting of shared_buffers (a large, guaranteed commitment) and enough headroom that normal operation never brushes the limit — this is exactly where the baseline numbers from a memory tuning guide stop being aspirational and become inputs to a kernel parameter. The database-side half of the audit is two queries: the cluster-wide memory knobs, and every per-role override that punches through them — the elevated work_mem on our reporting role lived in exactly this second list, set once and forgotten:
SELECT name, setting, unit
FROM pg_settings
WHERE name IN ('shared_buffers', 'work_mem', 'maintenance_work_mem',
'hash_mem_multiplier', 'max_connections',
'autovacuum_max_workers', 'max_parallel_workers_per_gather');
SELECT rolname, rolconfig
FROM pg_roles
WHERE rolconfig::text LIKE ANY (ARRAY['%work_mem%', '%maintenance_work_mem%']);And if your connection count is the term blowing up the budget, that is a pooling problem, not a memory problem; the failure shape is covered in the connection storms field notes, and the fix is a pooler in front, not more swap.
What changes when PostgreSQL runs in a container?
The same killer, a smaller room, and worse evidence. Inside a container the relevant limit is the cgroup memory limit, and when the cgroup exhausts, the kernel OOM-kills within the container — the database sees the identical signal 9, but there is no host dmesg you can read from inside, and in Kubernetes the event shows up as OOMKilled on the pod's last state rather than anywhere in the PostgreSQL log. Two container-specific traps made our own rollout painful. First, the memory limit has to sit above your entire budget — shared_buffers, connection envelope, autovacuum, and the page cache PostgreSQL leans on — and people routinely size the limit from the shared_buffers number alone, then act surprised. Second, a limit raise without a budget review is just rescheduling the incident: the fix is the same accounting as on bare metal, expressed as a pod spec. If the platform supports it, keeping the database pod's oom_score_adj protective of the postmaster is worth doing, but it changes nothing about the backend math.
What do you do the morning after an OOM kill?
You identify the query, because "the kernel did it" is a mechanism, not a root cause. The trail: the killed PID from the PostgreSQL log, matched against the OOM entry in the kernel journal for its peak RSS, then matched backward through connection logs or your monitoring to the application, role, and statement. pg_stat_statements does not survive the crash, so if the report ran and vanished without a trace, the durable fix is per-role logging — log_statement for the reporting role, or better, auto_explain loaded with a threshold that captures execution plans for exactly the queries large enough to matter. Then the three-line prevention list: the memory budget written down with the worst-plausible number, strict overcommit so the failure mode becomes a query error, and a per-role work_mem review, because "raise work_mem for the big jobs" is how every OOM story I have ever heard begins.
Watching memory pressure with MonPG
OOM kills feel instantaneous, but the pressure that causes them builds over minutes and is visible in ordinary series: host memory headroom trending down through the batch window, backend counts against max_connections, per-statement latency for the report workload, and swap usage creeping before the cliff. MonPG tracks exactly these PostgreSQL and host series as part of its PostgreSQL monitoring, so the 03:12 scenario shows up as a headroom line bending toward zero with an alert attached, instead of a signal 9 with no warning. Do the accounting once, set strict overcommit, and then let the dashboard watch the budget you wrote down.