The monitoring dashboard said SYNCHRONIZED in reassuring green, and the analytics team insisted their reports were showing twenty-minute-old data. Both were telling the truth. The secondary was hardened to the millisecond and redone to twenty minutes ago, and the word synchronized never promised anything about redo. Availability Group lag is two different numbers wearing one name, and until you separate them you will misread every lag alert you ever get.
Here is the mental model I use for Always On Availability Groups on SQL Server 2016 through 2022, the DMV columns that matter, and the failure shapes that repeat often enough to recognize from across the room.
Why is AG lag two different numbers?
Because the log has two journeys: across the network from the primary into the secondary's log file, and then from the secondary's log file into its data pages. sys.dm_hadr_database_replica_states reports the two journeys separately. log_send_queue_size is the KB of log generated on the primary that has not yet been sent to the secondary, and log_send_rate is how fast it is draining. redo_queue_size is the KB of log already hardened on the secondary but not yet replayed into the data files, and redo_rate is how fast that is draining.
A growing send queue is a transport problem: network throughput, a slow log write on the secondary, or flow control throttling the primary. A growing redo queue is a replay problem on the secondary itself: redo threads starved for CPU or I/O, or redo blocked outright by reader queries. The LSN columns, last_sent_lsn, last_received_lsn, last_hardened_lsn, last_redone_lsn, and last_commit_lsn, let you place any transaction exactly on that pipeline. The query I keep saved:
SELECT ar.replica_server_name,
DB_NAME(drs.database_id) AS db_name,
drs.synchronization_state_desc,
drs.log_send_queue_size / 1024.0 AS send_queue_mb,
drs.log_send_rate / 1024.0 AS send_rate_mb_s,
drs.redo_queue_size / 1024.0 AS redo_queue_mb,
drs.redo_rate / 1024.0 AS redo_rate_mb_s,
drs.last_commit_lsn, drs.last_hardened_lsn, drs.last_redone_lsn,
drs.last_redone_time
FROM sys.dm_hadr_database_replica_states AS drs
JOIN sys.availability_replicas AS ar
ON ar.replica_id = drs.replica_id;Run it on the primary for the send side and on each secondary for the harden and redo side; each replica knows its own truth best.
What does synchronous commit actually guarantee?
It guarantees that the log records for your commit are hardened, meaning written to the secondary's log file on durable storage, before the primary acknowledges the commit to your client. It says nothing about redo, so readers on a perfectly healthy synchronous secondary can be minutes behind the primary's committed state. The guarantee is about not losing data on failover, not about freshness of reads.
The price of that guarantee is paid per commit, and the meter is the HADR_SYNC_COMMIT wait on the primary: the session sits there until the secondary's harden acknowledgment comes back. Flow control exists to keep a slow secondary from drowning; once unacknowledged log crosses internal gates, the primary throttles its log send, which shows up as even more HADR_SYNC_COMMIT. If that wait climbs, the question is never whether the guarantee is working, it is whether the secondary's log write path or the network got slower. The wait-stats side of this pairs with my notes on wait statistics triage.
How stale is my readable secondary, exactly?
redo_queue_size divided by redo_rate estimates the seconds of redo backlog draining at the current pace, and last_redone_time compared against the primary's clock approximates the wall-clock staleness a reader is experiencing right now. Treat both as estimates with failure modes: redo_rate is a recent-period average that reads zero when redo is idle or blocked, which breaks the division exactly when you need it most, and on an idle but fully caught-up primary last_redone_time simply stops advancing, so an old timestamp can mean a quiet system rather than a stale reader. What does not lie is the mechanism: every query on a readable secondary sees the world as of last_redone_lsn, no matter how current the hardened log is. That is the math behind the opening story: send queue zero, redo queue enormous, dashboard green, users angry.
The trap worth knowing: queries on the secondary take schema stability locks, and redo occasionally needs schema modification locks to replay DDL. One monster report can freeze redo for every other reader while the queue climbs and climbs. If your readable secondary goes stale at the same time every day, find the report that starts at that time before you blame the network. Short transactions, no DDL during reporting hours, and query timeouts on the secondary's workloads keep this shape away.
How do I estimate RPO on an asynchronous replica?
Compare the primary's last_commit_time against the secondary's last_hardened_time: the gap, in seconds, is your worst-case exposure on a forced failover. Resist doing the same subtraction on the LSN columns. last_hardened_lsn is a padded log-block position rather than a true LSN, LSN encodings are not byte counters, and subtracting two of them produces a number, not a byte count of data loss. If you need the exposure in bytes, log_send_queue_size measured against your log generation rate is the closer proxy. Either way, that is the honest RPO, measured, rather than the near-zero figure from the architecture slide deck.
I learned to track it during a storage wobble at a DR site. The async replica's hardened lag climbed to about ninety seconds, roughly six hundred megabytes of log at that workload's write rate, while every status check still said the AG was fine. It was fine; it was just ninety seconds of data away from us. After that, hardened-lag seconds became a first-class metric with a pageable threshold, because RPO you do not measure is RPO you do not have.
Why does the secondary fall behind during index rebuilds?
Because a rebuild is a log firehose. Rebuilding a two-hundred-gigabyte clustered index generates log at a rate redo simply cannot match, so the redo queue climbs in one long ramp for as long as the rebuild runs, and readable secondaries go stale until hours after it finishes. Redo was single-threaded per database before SQL Server 2016; since then it is parallel, with a shared worker pool capped per database, which is better but nowhere near infinite.
The mitigations are boring and effective. Schedule big rebuilds outside reporting hours. Question whether the rebuild is needed at all, since fragmentation is wildly over-feared, a point I argue in the index maintenance reality check. Give the secondary real storage and CPU instead of the leftover hardware it usually gets. And watch automatic seeding when you add or re-seed a replica, because initial synchronization rides the same pipe: sys.dm_hadr_physical_seeding_stats shows progress, throughput, and failure codes, and sys.dm_hadr_automatic_seeding on 2016 and later shows the state machine, which beats wondering why a new replica appears to hang at eighty percent.
Watching AG lag with MonPG when SQL Server support ships
The lag story deserves per-database trends, not a green light: send queue and redo queue over time, send and redo rates, harden and redo lag expressed in seconds, and HADR_SYNC_COMMIT tracked as a top wait on synchronous primaries. MonPG monitors PostgreSQL in production today; SQL Server support is on the roadmap and in active development, and the AG counters in this article are exactly what we intend to graph per replica. The honest status lives on the SQL Server monitoring (coming soon) page. Until then, the DMV query above on a schedule is most of the value for the price of one agent job.