PostgreSQL Logical Decoding Performance on the Primary: What CDC Costs

The Tuesday we pointed the first Debezium-style connector at the orders primary, nothing broke, and that is exactly why the bill went unnoticed for a month. The primary was doing about 11,000 commits per minute at the afternoon peak and generating roughly 1.4 GB of WAL per hour. After we flipped wal_level to logical and attached one pgoutput slot, a single walsender settled in at around a third of a core, WAL generation drifted to 1.7 GB per hour, and p99 on order inserts moved less than a millisecond. Affordable. Then, over a long weekend, the consumer died on an expired schema-registry certificate, nobody paged on the slot, and by Monday morning the slot had pinned 92 GB of WAL and the data volume sat at 81 percent full. The outage mechanics of that scenario are their own story, covered in the replication slots and retained WAL field notes. What follows is the performance ledger: what decoding costs while everything is working, and how to measure that cost before you enable it.

What does logical decoding actually cost the primary?

Four things, and only one of them shows up in a CPU graph by default. First, wal_level = logical writes slightly more WAL than replica level — extra catalog and identity information the decoder needs — which in my measurements lands in the low single digits percent for a normal row workload, and a restart to enable, so it is a change you schedule, not flip. Second, each slot decodes on its own walsender backend: every committed transaction that touches the publication's tables is re-read from WAL, reconstructed, and serialized on the primary's CPU. One busy slot at a third of a core matched what we saw; three slots each decoding the same WAL stream pay that three times, because decoding is per-slot, not shared. Third, the reorder buffer, which deserves its own section below. Fourth, the retention obligation: the slot's restart_lsn pins WAL on the primary's disk until the consumer confirms it, so a slow or dead consumer converts directly into disk growth at your WAL generation rate. None of these is a reason to avoid CDC. All of them are reasons the phrase "just add a connector" should make you reach for a spreadsheet.

How does the reorder buffer use memory and disk?

The reorder buffer is where the decoder accumulates an in-flight transaction's changes until commit, because a transaction can only be sent downstream once its commit record arrives and its changes are complete. Small transactions are invisible here. Large ones are the trap: a single batch UPDATE that touches 30 million rows builds its entire change set in the decoding backend's memory before it can be streamed. When the buffer exceeds its ceiling, changes spill to disk files under pg_replslot, and spilling a huge transaction while it is still running can mean writing and re-reading gigabytes of change data on the primary's data volume. Since PostgreSQL 16 the ceiling is logical_decoding_work_mem, default 64 MB, per decoding session; earlier releases keyed the spill threshold off work_mem, which made it easy to hit by accident on a tuned-down OLTP config. PostgreSQL 14 and later expose the damage directly in pg_stat_replication_slots:

SELECT slot_name, spill_txns, spill_count,
       pg_size_pretty(spill_bytes) AS spilled,
       stream_txns, total_txns
FROM pg_stat_replication_slots;

Our monthly reconciliation batch was the offender: one transaction, forty minutes, and spill_bytes for the slot climbed by about 6 GB every time it ran, with decode latency for everything queued behind it rising to match. The fixes are workload-shaped, not config-shaped: chunk the batch into commits of a few hundred thousand rows, or raise logical_decoding_work_mem on a host with headroom if the big transaction is untouchable. Streaming of large in-progress transactions (logical decoding streaming, on for pgoutput subscribers that negotiate it) helps the consumer see changes sooner, but the buffer management on the primary remains yours either way.

pgoutput or wal2json: which output plugin is cheaper?

pgoutput is cheaper on every axis the primary cares about, and it needs no extension installed. It is the built-in protocol that logical replication itself uses — compact, typed, and close to the wire format — which is why modern Debezium defaults to it. wal2json earns its place when the consumer is simple: it emits one JSON document per change that a script or a queue shim can parse without a protocol library, and being able to eyeball the stream during debugging is genuinely useful. The cost is that every row is serialized to text with its column names repeated, every value stringified. On a thirty-minute replay of our afternoon window, the same change stream came out to roughly 1.4 GB through pgoutput and a little over 3 GB through wal2json, with the JSON encoder visibly hotter in the walsender's profile. That delta is paid three times: primary CPU to build it, network to carry it, consumer CPU to parse it. There is also test_decoding, the contrib plugin, which is the most verbose text format of the three and should be treated strictly as a lab instrument — it is what makes benchmarking easy, as the last section shows. If you are still choosing between CDC architectures, the MySQL CDC versus PostgreSQL logical decoding comparison covers the ecosystem side; the performance answer here is: default to pgoutput unless a human needs to read the stream.

When is REPLICA IDENTITY FULL worth its WAL amplification?

When the consumer genuinely needs the before-image of changed rows — and only on those tables. With the default replica identity, an UPDATE or DELETE logs just the key columns of the old row; REPLICA IDENTITY FULL logs the entire old row image, so every update writes the row's full width to WAL on top of the new version the update already writes. For a 400-byte row updated in place, that is close to doubling the WAL for that statement, and on an update-heavy table it shows: when we briefly set FULL on the order_lines table to satisfy an audit consumer, WAL generation during the evening batch roughly tripled for that window. The measurement takes two minutes, which is why there is no excuse for guessing:

SELECT pg_current_wal_lsn() AS before_lsn;
-- run a representative chunk of the update workload here
SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '<before_lsn>') AS wal_bytes;

Run it once with DEFAULT identity, once with FULL, same workload, and compare. Before reaching for FULL, check the cheaper rungs: DEFAULT already gives consumers the old primary-key values, which covers most join-back use cases, and REPLICA IDENTITY USING INDEX logs only the columns of a chosen unique index. The legitimate FULL cases are real — audit trails that must capture every old column, consumers that diff before and after without querying the source, and tables without a primary key that subscribers must apply changes against. Scope it per table, never per database reflex, and re-measure when the table's row width or update rate changes materially.

How do you monitor and benchmark decoding before production?

Monitor three numbers per slot and benchmark on a restored copy before granting the connector its production credentials. The numbers: retained WAL per slot, spill activity, and stream lag. Retained WAL is the one that fills disks, and pg_stat_replication_slots (PostgreSQL 14+) carries the spill and streaming counters; if your consumers are native logical replication subscribers, pg_stat_subscription_stats on the subscriber side adds apply-error and sync-phase visibility.

SELECT slot_name, active, restart_lsn,
       pg_size_pretty(
         pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
       ) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

Two safety rails belong in the same breath. max_slot_wal_keep_size (PostgreSQL 13+) caps how much WAL a slot may pin — set it to a fraction of free disk you can actually spare, and accept the trade-off with open eyes: a consumer that falls past the cap loses its slot and must re-snapshot, which is a loud failure instead of a silent disk-full. And alert on retained_wal growing while active is false; that pair is the exact signature of our long-weekend incident.

For the benchmark, restore a recent backup to a staging host, flip wal_level to logical, create a test_decoding slot — it needs no client and nothing to install — drive a realistic pgbench mix at your production commit rate, then drain the slot while watching the walsender's CPU and the pg_stat_replication_slots counters:

SELECT pg_create_logical_replication_slot('bench_slot', 'test_decoding');
-- pgbench -c 32 -T 900 against the restored database, then:
SELECT count(*)
FROM pg_logical_slot_get_changes('bench_slot', NULL, NULL);
SELECT slot_name, total_txns, pg_size_pretty(total_bytes) AS decoded,
       spill_txns, pg_size_pretty(spill_bytes) AS spilled
FROM pg_stat_replication_slots
WHERE slot_name = 'bench_slot';

Time the drain and divide changes by seconds: that is your decode throughput ceiling for this workload shape, and it tells you whether one slot keeps up with your peak commit rate or falls behind from the first minute. Repeat with your heaviest batch job included, because that is where the reorder buffer story is written. An hour of this is cheaper than the incident review.

Watching decoding overhead with MonPG

Every line in this ledger is counter-shaped. WAL generation rate per database, walsender CPU, per-slot retained WAL, spill transactions and bytes, and the active flag on each slot — those are the series that turn "the connector seems slow" into a named cause. MonPG graphs exactly these as part of its PostgreSQL monitoring, alongside the disk headroom the slots are quietly spending, so a stalled consumer pages on retained WAL at thirty percent full instead of on a dead primary at ninety-eight. Enable CDC with the spreadsheet first; then let the counters keep you honest.