CPU, Memory, and I/O12 min read

pg_stat_io: The First Honest I/O Picture PostgreSQL Has Had

Hit-ratio dashboards kept telling me the buffer cache was fine while storage latency tripled. pg_stat_io finally shows who is reading, writing, extending, and fsyncing — split by backend type and I/O context.

For most of my career, PostgreSQL I/O monitoring has been a bluff. You had pg_stat_bgwriter, a handful of cluster-wide counters that described checkpoint traffic in aggregate. You had pg_statio_user_tables with blks_read and blks_hit, per table but with no latency and no clue about who caused the read. And you had the famous cache hit ratio, a number that stayed green right up until users started complaining. I ran a primary through a noisy-neighbor storage event where read latency on the volume roughly tripled for forty minutes. The hit ratio dashboard never dipped below 99.5 percent. The graph that actually moved was pg_stat_statements mean time, and by the time that moves you are reading symptoms, not causes.

PostgreSQL 16 finally shipped the view I had been waiting for: pg_stat_io. It is the first system view that attributes I/O to the process class that did it and the access pattern it came from, and it quietly makes half the I/O folklore obsolete. If you run PG 16 or newer and your dashboards still lead with hit ratio, this is the view to rebuild around.

What pg_stat_io actually counts

The view breaks I/O down along three axes: backend_type (client backend, checkpointer, background writer, autovacuum worker, walsender, and friends), object (relation or temp relation), and context (normal, vacuum, bulkread, bulkwrite). For each combination you get reads and read_time, writes and write_time, extends and extend_time, hits, evictions, reuses, fsyncs and fsync_time, plus op_bytes, the size of a single I/O operation. On PG 16 every operation is one block, so op_bytes just restates your block size and reads doubles as a byte count once you multiply the two; PG 17's I/O combining lets one operation span several blocks, which is when the column starts earning its keep.

Two definitions matter before you trust any of it. A hit means the block was already in shared buffers. A read means the block had to come from storage — and storage includes the operating system's page cache, so a read can still be fast. And the _time columns only fill in when track_io_timing is on, which costs a clock call per I/O. I leave it on everywhere; the overhead has never once shown up in a profile, and latency attribution is the entire point of the exercise.

-- Who is doing the I/O, and in which context?
SELECT backend_type, context, reads, read_time, writes, write_time,
       extends, evictions, fsyncs
FROM pg_stat_io
WHERE object = 'relation'
ORDER BY writes DESC, reads DESC;

-- Counters are cumulative; open a fresh measurement window with:
SELECT pg_stat_reset_shared('io');

Every counter is cumulative since stats_reset, so the raw numbers are trivia. The signal lives in deltas: scrape the view on a cadence, subtract, and graph per second. A single snapshot of pg_stat_io is like a single frame of a film — you can stare at it as long as you want and still not know what is happening.

Why hit-ratio-only dashboards lie

The cache hit ratio — hits over hits plus reads — fails in three specific ways, and I have been burned by each. First, reads include operating system cache hits, so the ratio treats "read from disk" and "read from page cache" as the same event. When your storage layer degrades, the ratio does not move until the OS cache stops absorbing the load, which is exactly backwards from what an early-warning metric should do. Second, large sequential scans and bulk loads barely churn the shared pool: they run through small ring buffers carved out of it — bounded slices they reuse among themselves — and show up under the bulkread and bulkwrite contexts, so a reporting query hammering the disk barely dents the hit ratio. Third, the classic ratio lumps every consumer together — autovacuum, checkpoints, analytics, OLTP — into one green number. pg_stat_io splits them apart by backend type and context, so "the disk is slow" becomes "autovacuum worker reads jumped from 400 to 9000 per second at 14:00, and read_time per read went with it." That is a sentence you can act on.

Backends doing the checkpointer's job

On a healthy OLTP primary, the checkpointer and the background writer own almost all relation writes in the normal context. That division of labor is the whole design: those two spread dirty-buffer flushing over time so commit paths never wait on data-file I/O. When client backends show large write counts in context normal, it means a backend asked for a free buffer, found only dirty ones, and flushed a victim itself. That is synchronous file I/O sitting directly in a user's query path, and it feels to the application exactly like random commit stalls.

I hit this after a traffic spike roughly doubled our write rate. Within an hour, client backends were responsible for about 45 percent of relation writes, and commit latency had a sawtooth that matched checkpoint windows. The old view shows the same event in coarse form — pg_stat_bgwriter's buffers_backend counter is that flush, without attribution — but pg_stat_io tells you which backend class and which context, per second. The fix for us was boring and effective: bgwriter_lru_maxpages up from the default, bgwriter_delay down, and a higher max_wal_size so checkpoints stopped bunching up. The checkpoint half of that story is covered in the checkpoint tuning notes; the point here is that pg_stat_io is how you prove the problem exists before you touch those knobs. One nuance: extends from client backends are normal — appending blocks is application work. It is writes in the normal context that tell the story.

-- What share of relation writes are client backends absorbing?
SELECT backend_type, context, writes,
       round(100.0 * writes / nullif(sum(writes) OVER (), 0), 2) AS pct
FROM pg_stat_io
WHERE object = 'relation' AND writes > 0
ORDER BY writes DESC;

Evictions, extends, and the signals nobody graphs

Evictions count buffers thrown out of shared_buffers to make room for new ones. Some churn is fine; sustained high evictions alongside high reads is the shape of a working set that no longer fits in memory. Reuses count ring-buffer reuse under bulk strategies — high reuses with low evictions is exactly what a well-behaved big scan should look like, and it is why bulkread traffic deserves its own graph separate from the normal context. Extends are relation growth: a rate per context tells you whether growth comes from application writes or from vacuum's housekeeping. And fsyncs should be overwhelmingly the checkpointer's. A client backend fsyncing relation files is rare enough that any sustained nonzero number is worth an hour of digging.

The number I now watch closest is read_time divided by reads, per backend type. In that noisy-neighbor incident, autovacuum worker read latency degraded from about 0.4 ms to 9 ms a full ten minutes before client backend latency visibly moved. Autovacuum reads colder, less cached data, so it feels storage pain first. That is the early-warning channel the hit ratio was never going to give us, and it costs one division.

pg_stat_bgwriter still earns its keep

Do not uninstall the old dashboards. pg_stat_bgwriter is coarse, but it answers a different question well: are checkpoints timed or forced, and how much time goes to the write phase versus the sync phase. On PG 17 the checkpoint counters moved into pg_stat_checkpointer, which pairs even more cleanly with pg_stat_io. My rule of thumb: the bgwriter and checkpointer views for checkpoint cadence, pg_stat_io for attribution and latency. Together they cover the questions the old hit-ratio wall could not, and neither is expensive to collect.

Watching pg_stat_io with MonPG

MonPG monitors PostgreSQL in production today, and pg_stat_io deltas are exactly the kind of evidence it is built around: reads, writes, and read latency graphed by backend type and context, so "backends doing the checkpointer's job" shows up as a visible share shift instead of a commit-latency mystery. The PostgreSQL monitoring surface covers how those counters are collected and alerted on, and the field notes on debugging high I/O walk through the triage path for when the graphs turn red. Wire the view into your scraper on PG 16 and up, and retire the hit ratio to the second row of the dashboard where it belongs.