At 21:14 on a Tuesday, during our evening write peak, p99 commit latency on the primary went from 4 ms to 2,800 ms in about ninety seconds. The dashboard said CPU was pinned at 100% on all 32 vCPUs, so the first instinct was an I/O story — except iostat showed the WAL volume at 3% utilization with zero await. Memory was fine. No checkpoints in flight, no autovacuum workers doing anything unusual, no replication lag. When I pulled pg_stat_activity there were 1,437 sessions in state 'active' or 'idle in transaction', and nearly all of them shared the same two columns: wait_event_type='LWLock', wait_event='WALInsertLock'. Nothing was blocked on a row lock. Nothing was waiting on disk. A thousand backends were queued at a single spinlock-guarded door, burning every core on the box to stand in line.
That night taught me more about LWLocks than any documentation page had, because the failure mode is so counterintuitive: the database looks compute-bound, and the fix has nothing to do with compute.
What are LWLocks, and how are they different from regular locks?
LWLocks — lightweight locks — are short-lived locks that protect shared-memory structures inside PostgreSQL: the WAL insertion pointers, buffer mapping tables, the lock manager's own hash tables, SLRU caches, and similar internals. They exist so that hundreds of backends can touch the same shared structure without corrupting it, and they are held for microseconds at a time, not for the length of a transaction.
They are a completely different animal from the locks most people mean when they say "locking" in PostgreSQL. Regular locks — row locks, table locks, the whole ACCESS SHARE through ACCESS EXCLUSIVE family — are heavyweight: they live in a shared lock table, they are acquired and released by the lock manager, they participate in deadlock detection, they show up in pg_locks, and they can be held for the entire duration of a transaction. If you have read our piece on locks and deadlocks, everything in it describes that heavyweight world.
LWLocks skip all of that machinery. There is no deadlock detector watching them, no pg_locks row for them, no lock_timeout that will save you. A backend that wants an LWLock that is busy spins briefly, and if it still can't get in, it sleeps and gets woken when the lock is released. Which brings up the important part of our incident: "CPU at 100%" did not mean the CPUs were doing useful work. A large fraction of it was backends spinning on one lock. High CPU with idle disks is the classic silhouette of LWLock contention; every other high-CPU diagnosis starts with a slow query, and there was none here.
How do you see LWLock waits in pg_stat_activity?
You see them in the wait_event_type and wait_event columns of pg_stat_activity: any backend currently stuck on a lightweight lock reports wait_event_type='LWLock', and wait_event tells you which one. A single snapshot is noisy, so what actually worked for us was sampling the same aggregate a few seconds apart and watching which names dominated:
-- Snapshot of who is waiting on what, right now.
-- Run it a few times, ~3 seconds apart, during the incident.
SELECT wait_event_type,
wait_event,
state,
count(*) AS sessions
FROM pg_stat_activity
WHERE wait_event_type = 'LWLock'
AND backend_type = 'client backend'
GROUP BY wait_event_type, wait_event, state
ORDER BY sessions DESC;During our incident the output was almost comically one-sided: WALInsertLock with roughly 1,300 sessions, then a long tail of single-digit counts on names like WALWriteLock and XactSLRU. The names you will most often meet in production:
- WALInsertLock — backends waiting to copy their WAL records into the shared WAL buffers. The signature of a commit-rate problem.
- LockManager — a partition of the heavyweight lock table is contended. Usually a lock-count problem, not a commit problem.
- BufferMapping — contention on the mapping between pages and shared buffer slots; typically a hot-block or buffer-pool pressure symptom.
- XactSLRU and CommitTsSLRU — contention on the transaction-status and commit-timestamp caches. CommitTsSLRU in particular only shows up when track_commit_timestamp is enabled, which is worth knowing before you turn that setting on.
One caveat that burned a colleague later: a point-in-time snapshot can lie. A backend spends most of its life not waiting on anything, so a single sample understates short waits. Sample repeatedly — or better, continuously, which is what a monitoring layer should do for you.
Why does a high commit rate bottleneck on WALInsertLock?
Because every transaction that modifies data must write WAL, and copying WAL records into the shared WAL buffers is serialized behind WALInsertLock — so commits per second is effectively capped by how fast backends can take turns at that critical section. No matter how fast your disk is, you cannot commit faster than the lock lets you in.
The mechanics matter for the fix. A committing backend has to get its WAL records into the shared WAL buffers, and that insertion — copying records at the current insertion position — is guarded by an LWLock so two backends don't scribble over the same buffer space. At low commit rates nobody notices. At thousands of commits per second the critical section becomes the bottleneck for the whole cluster: each commit holds the lock for microseconds, but the queue in front of it is enormous.
Our workload was a worst case: a job scheduler that had drifted toward one transaction per task, roughly 11,000 commits per second at peak, each updating one or two rows. The disks were idle because the WAL volume was trivial — a few MB/s. The CPUs were pegged because a thousand backends were contending on one lock. Every remedy we reached for first — more IOPS, faster storage, more memory — targeted a resource that wasn't the constraint.
PostgreSQL does have a group-commit mechanism where one backend flushes WAL on behalf of others, and it helps with flush latency under load — but it does not remove the insertion-side serialization, and with enough tiny commits the queue forms long before flush throughput is the issue. This is not a bug or a mistuned parameter; it's the fundamental cost of extremely high commit rates, and it has looked broadly similar across the recent major versions we've run.
What actually reduces WALInsertLock contention?
Reduce the number of commits. That is the answer, and everything else is a way of paying for it in a different currency. Here's our cost sheet, in the order we tried things:
Batch commits in the application. We changed the job scheduler to group up to 200 tasks into one transaction with a short linger window, so latency per task stayed under ~50 ms. Commits per second dropped from ~11,000 to under 300, and p99 commit latency went back to single-digit milliseconds the same evening. Cost: a scheduler bug now fails a batch instead of a task, so we had to make batch retry and partial-failure handling explicit. This was 90% of the win.
Fewer, bigger transactions generally. If you control the write path, amortize: one transaction doing 500 inserts pays the commit machinery once; 500 transactions pay it 500 times. The tradeoff is the usual one — longer transactions hold row locks and snapshots longer, and past a point you start creating the problems described in our notes on long transactions. Batching is a dial, not a binary.
synchronous_commit=off. This lets COMMIT return without waiting for the WAL flush, decoupling commit latency from flush latency entirely. The honest cost: on an operating system or hardware crash you can lose the last small window of committed transactions — commits the client was told succeeded. The window is bounded by the WAL writer's flush cadence, typically a few hundred milliseconds by default, and you get lost tail commits, not corruption. For our metrics-ingest pipeline that was an acceptable price; for the billing tables it was not, so we set it per-database, not cluster-wide.
wal_compression=on. Compresses full-page images in WAL, cutting WAL volume and therefore flush pressure. It doesn't reduce the number of trips through WALInsertLock, so it didn't move our commit latency much — but it did reduce total WAL written by roughly half during checkpoints, which bought headroom on the flush side. Cost: some extra CPU on backends writing WAL, which is a strange thing to add when your CPUs are already pegged. Worth it for the volume reduction, not as a contention fix.
wal_level=minimal, only if you genuinely don't need replication. Minimal skips WAL for certain bulk operations and lowers overall WAL overhead. The cost is severe: no streaming replication, no PITR, no logical decoding. We run replicas everywhere, so this stayed theoretical for us; on anything with a replica attached, it's a non-starter.
If you're staring at the same graph, our guide to monitoring WAL in PostgreSQL covers the counters — WAL generation rate, flush lag, checkpoint behavior — that tell you which side of the commit path is the real constraint.
Why would sessions pile up on LockManager instead?
LockManager waits mean too many backends are fighting over the same partition of the heavyweight lock table, and the cause is almost always a runaway lock count, not a runaway commit rate. The lock table is internally partitioned to reduce contention, but that only helps when locks spread across partitions; when one hot object or one enormous lock set dominates, the partition guarding it becomes a queue.
The two ways we've produced this in production: first, a schema with several thousand partitions where a single ORM-generated query touched all of them, taking a lock on each partition and its indexes — tens of thousands of lock entries per statement, with every backend doing it at once. Second, a long-running transaction that accumulated locks for hours while short transactions piled up behind its partition. The tell is the same query as before with a different answer: wait_event='LockManager', plus a pg_locks count that keeps climbing. The fixes are boring: prune partitions in the query so it touches tens instead of thousands, consolidate partitions you no longer need, and kill or shrink the long transaction. None of this involves WAL — which is why the wait_event name matters before you start "fixing" anything.
How do you watch LWLock contention with MonPG?
By graphing the wait events themselves, not just active sessions. The incident above took us forty minutes to name, and it would have taken two with the right graphs up. What we now watch continuously in MonPG's PostgreSQL monitoring is exactly the query from earlier, sampled over time: sessions waiting by wait_event_type and wait_event, so WALInsertLock, LockManager, and BufferMapping each get their own line instead of hiding inside a generic "active sessions" count. Alongside that, commits per second, WAL bytes per second, and CPU on one screen — because the diagnosis lives in the combination: CPU pegged, WAL bytes low, commit rate high, LWLock waits climbing is WALInsertLock contention; CPU pegged with WAL bytes high is a flush problem and points somewhere else. MonPG alerts when the share of sessions waiting on LWLocks crosses the threshold we set after that Tuesday, so the next time the scheduler drifts back toward one-commit-per-task we find out from the alert, not from the p99 graph.