The customer report said a comment thread showed text from a different tenant's order notes. That sentence still makes my stomach drop, years later. The cause was not a permissions bug: it was a RAID controller whose firmware update had subtly broken its cache flush behavior, and for about a month it had been occasionally writing the wrong 8 KB page to the wrong place on disk. PostgreSQL never complained, because the cluster had been initdb'd years earlier with data checksums off, and a page that is internally plausible — valid header, valid layout, somebody else's rows — fails no check the database performs. We found the corruption the worst way, and the only reason it was not worse is that the same controller eventually produced a page that was not plausible, and a sequential scan died with ERROR: invalid page in block 38471 of relation. That error message is the sound of the database telling you it has been reading garbage silently for an unknown length of time.
We rebuilt the cluster from a verified backup, replaced the controller, and enabled checksums on every cluster we run. This post is everything I wish someone had handed me before that month: what checksums buy, what they cost — including the cost the documentation understates — and how to enable them on a cluster that already exists.
What do data checksums actually protect against?
Every data page PostgreSQL writes to storage gets a 16-bit checksum computed over its contents, and every page read back from storage is verified against it. That single mechanism catches the entire class of failures that live between the database's memory and the disk platter: torn pages from a crash mid-write, bit rot on aging media, misbehaving RAID firmware like ours, bad sectors, and storage appliances that acknowledge writes they have not actually persisted. What it does not catch is just as important to say out loud. Corruption that happens in memory before the page is written is invisible to a page checksum. Corruption inside the operating system's page cache is invisible. And logically consistent but wrong data — a buggy migration, an over-eager cascade, the index-vs-heap disagreement that collation version drift produces — is not a checksum problem at all. Checksums are the floor, not the ceiling: they make the storage layer honest, and nothing more.
The verification timing matters for how you detect problems. A page is checked when it is read from storage into shared_buffers, not on every access — a corrupt page that stays cached can serve wrong data for a long time before anything looks at the disk copy. This is why the periodic full read of your data is not optional hygiene. A nightly logical dump, a weekly pg_basebackup, even a scheduled VACUUM that walks the heap — anything that forces every page through storage reads — is a corruption sweep, and with checksums on, that sweep reports what it finds instead of silently archiving it into your backups.
Why is this still an opt-in at initdb?
History, mostly. Checksums arrived in PostgreSQL 9.3, defaulted to off, and stayed off by default through version 17 because the overhead, while small, was not zero and changing defaults on a setting this fundamental is done conservatively. PostgreSQL 18 finally flips the default to on, which tells you where the project's own cost-benefit analysis landed. Check where you stand before assuming anything:
SHOW data_checksums;
SELECT datname,
checksum_failures,
checksum_last_failure
FROM pg_stat_database
WHERE checksum_failures > 0;The second query is your early-warning system and it has existed since PostgreSQL 12: any nonzero checksum_failures count, anywhere in the cluster, means storage has already lied to you at least once and the question is only how often. Check it on the standbys too — a standby reads pages the primary may not touch for months, and ours has caught a bad disk on a replica before the primary ever knew. One caveat on the counter: it records detection events, and with checksums off it will sit at a comforting zero forever, which is exactly the false comfort that got us.
How do you enable checksums on a cluster that already exists?
Offline, with pg_checksums, and there is no way around the offline part as of PostgreSQL 17. The cluster must be shut down cleanly — an unclean shutdown leaves pages in states the tool refuses to touch — and then pg_checksums --enable rewrites every data file in place, computing and stamping checksums page by page. Budget the time honestly: it is a full sequential read and write of your entire dataset, so for a terabyte on decent storage you are looking at tens of minutes to a couple of hours, and the only way to know is to rehearse it on a restored copy, the same discipline as any other maintenance that scales with data size.
The part that surprises people is the standbys. Checksums are a physical property of the data files, and pg_checksums on the primary does not propagate: each standby either gets the same offline treatment or, more sanely, gets rebuilt from the upgraded primary with pg_basebackup, which is what we did — one rebuild, done, no per-node surgery. If your cluster was created with checksums off and rebuilding everything is not on the table this quarter, write "checksums: off" into the runbook as a known gap rather than letting it fade into folklore, and be extra religious about the verified-backup discipline below. For clusters being created today there is no decision left to make: pass --data-checksums to initdb, or take version 18 and accept the default.
What is the real cost of turning checksums on?
The CPU cost of computing and verifying a 16-bit checksum per 8 KB page is small — low single-digit percentages on the workloads I have measured, and effectively invisible on anything not already CPU-bound. That is the number most discussions stop at, and it is the wrong number to stop at, because the real cost is in the WAL. PostgreSQL must protect a checksummed page against torn-page corruption during crash recovery, and it does that the same way it does for wal_log_hints: the first modification of a page after each checkpoint logs the entire page image, and with checksums enabled, that rule extends to hint-bit-only updates — the tiny "this tuple is committed, stop asking clog" markers that otherwise dirty a page without logging anything. On read-heavy workloads that touch many pages without otherwise modifying them, enabling checksums can measurably increase WAL volume, because every freshly hint-bit-marked page now contributes a full-page write. On our largest cluster the increase was around ten percent — worth it, obviously, but a number I would rather have forecast than discovered, and one more reason the WAL monitoring notes and the checkpoint tuning notes belong in the same reading session as this one.
What do you do when a checksum failure actually fires?
You resist the urge to paper over it, first of all. zero_damaged_pages exists and will let PostgreSQL read past a bad page by zeroing it, and it is exactly as dangerous as it sounds: you are trading an error for silent data loss, which is the failure mode this entire article exists to prevent. Its only legitimate use is one-shot forensic salvage on a copy, never on the live cluster. The honest playbook runs through verification and replacement. Confirm the scope with the pageinspect extension — page_header(get_raw_page('events', 38471)) shows you the page's own view of its checksum and layout, and comparing what you see against the error tells you whether you are looking at one unlucky page or a pattern. Then repair by replacement: a corrupted index page is the easy case, because REINDEX CONCURRENTLY rebuilds it from the heap; a corrupted heap page means restoring that data from a backup you have verified, or accepting the loss of whatever rows lived on it if none exists. This is the moment where backup verification stops being a compliance checkbox, and it is why we treat a checksum failure as a storage incident with a database symptom — the controller, not the table, is the suspect.
Watching for silent corruption with MonPG
The operational shape of this whole topic is a counter that must never move: checksum_failures per database, tracked over time, alerting on any increase, graphed on the primaries and every standby. Around it sit the supporting series — WAL generation rate so the hint-bit cost of enabling checksums shows up as a measured step instead of a mystery, and backup verification status so the repair path is proven before you need it. MonPG tracks exactly these PostgreSQL series as part of its PostgreSQL monitoring, which means the first torn page your storage layer produces shows up as an alert with a timestamp, not as a customer reporting somebody else's order notes in their comment thread. Turn the checksums on, graph the counter, and rehearse the repair — in that order.