The planned failover at 02:00 went exactly as the runbook described — right up to the part where the applications came back. The availability group moved roles in nine seconds, the new primary was green in the dashboard, databases synchronized, and the DBA on call declared success. Then the support channel lit up: every application server was throwing connection timeouts, and they kept throwing them for six full minutes while the database sat healthy and idle. The listener name was resolving to the old node's IP address. The connection strings, written years earlier by a team that had copied them from a standalone server, had no MultiSubnetFailover keyword, and each connection attempt was burning a twenty-one-second TCP timeout against a dead IP before even considering the live one. Nine seconds of database downtime, six minutes of application downtime, and the gap between those two numbers was entirely client-side configuration.
This is the article I wish the application teams had read before that night: what the listener actually is, why DNS caching and subnet behavior decide your real failover time, the cluster settings that matter, and how to measure failover from the client's point of view instead of the cluster's. Applies to SQL Server 2016 through 2022 availability groups, Windows Server Failover Clustering underneath.
What is the AG listener actually doing?
The listener is a cluster resource that owns a DNS name and one or more IP addresses, and its only job is to make the name resolve to whichever node currently hosts the primary role — the database engine itself knows nothing about it. When you connect to the listener name, the client asks DNS for the name's addresses, gets back one IP per subnet the AG spans, and picks one to dial. During a failover the cluster takes the listener's IP resource offline on the old node and online on the new one, then issues a DNS update so the name follows. That update is where the trouble starts: the DNS record has a TTL, every resolver and client DNS cache between you and the server honors it, and "the record now points elsewhere" does not reach a client whose cached answer has twenty minutes of life left. The cluster did its job in seconds; the name resolution layer operates on a completely different clock.
Why did the clients hang for minutes after a clean failover?
Because of two independent behaviors stacking: the client only tried one IP, and it waited out a full TCP timeout before failing. When a DNS lookup for the listener returns multiple addresses — the normal case for a multi-subnet AG, and effectively the case whenever the record set holds both the online and a recently-offline IP — a client without MultiSubnetFailover=true in its connection string tries the addresses one at a time, in order, serially. Each attempt against the dead IP waits for the operating system's TCP connect timeout, roughly twenty-one seconds on Windows with default retransmission. If the dead IP sorts first, the client sits through that full wait before falling through to the live address — per connection attempt, so a connection pool refilling forty connections after failover multiplies the pain into minutes of perceived outage even when DNS is perfectly correct. Add a stale DNS cache on top and the client is dialing the old node with no fallback address at all until the TTL expires. Our six minutes was both at once: stale caches on the app servers' resolver, then serial twenty-one-second timeouts once the caches refreshed mid-incident.
What does MultiSubnetFailover actually change?
Setting MultiSubnetFailover=true in the connection string tells the client driver to attempt all returned IP addresses in parallel — or to iterate them aggressively fast, depending on the driver — so the live address answers in milliseconds regardless of which one the dead one is, cutting failover reconnection from tens of seconds to roughly the TCP round-trip time. Every modern Microsoft driver supports it: SqlClient, ODBC, OLE DB, JDBC with multiSubnetFailover, and the keyword has been safe to leave on permanently for years, including against single-subnet listeners and standalone servers, so there is no reason for any connection string in the estate to lack it. The settings on the cluster side complete the picture, and they are inspectable rather than mysterious:
-- cluster-side listener properties (run in PowerShell on a cluster node)
-- Get-ClusterResource "AG_Listener" | Get-ClusterParameter
-- Key values:
-- RegisterAllProvidersIP = 1 (register every subnet IP in DNS)
-- HostRecordTTL = 1200 seconds by default (!)
-- client-side, the connection string that survives failover:
-- Server=AGLISTENER.corp.local,1433;Database=Orders;
-- MultiSubnetFailover=true;Connect Timeout=15;The interaction that bites: with RegisterAllProvidersIP=1 — the default — DNS holds an A record per subnet, and clients without MultiSubnetFailover get the full list and dial serially, exactly the failure above. With MultiSubnetFailover=true everywhere, registering all IPs is what you want, because the parallel dial finds the live one instantly. The old advice to set RegisterAllProvidersIP=0 so only the online IP is registered dates from drivers that could not do parallel attempts; it reduces DNS to one record but makes the TTL problem absolute, since there is no second address to try while the single record is stale.
How do I fix the DNS caching half of the problem?
You lower the listener's HostRecordTTL cluster property so resolvers and client caches expire the record quickly, accepting more DNS query traffic in exchange for failover times bounded by seconds instead of the default twenty minutes. The default HostRecordTTL of 1200 seconds means a client that resolved the listener at 01:59 trusts that answer until 02:19 — longer than most maintenance windows. Setting it to 30 or 60 seconds is the standard production posture for an AG whose clients cannot be guaranteed to run MultiSubnetFailover-aware drivers:
-- PowerShell on a cluster node:
-- Get-ClusterResource "AG_Listener" |
-- Set-ClusterParameter -Name HostRecordTTL -Value 60
-- Then take the listener network name offline/online once to apply.Two honest caveats. First, HostRecordTTL is a hint that well-behaved resolvers honor; some corporate forwarding resolvers and some JVM DNS caches clamp or ignore TTLs, which is why the connection-string fix matters more than the DNS fix — the Java security property networkaddress.cache.ttl defaulting to cache-forever on some old JVM builds caused a whole separate incident for us. Second, none of this helps in-flight connections: failover kills existing sessions outright, and the application's retry logic decides whether users see a blip or an error page. Test the whole path — kill the role, watch a real client reconnect, time it — because the cluster log and the client experience are different measurements of the same event.
How does this relate to what the AG itself is doing during failover?
The server-side timeline is its own discipline — role move, database recovery on the new primary, synchronization state of the secondaries — and it is covered in my AG lag monitoring notes; the listener layer sits entirely in front of it and is invisible from inside the database. That invisibility is the operational trap: every dashboard in the incident showed green because every dashboard measured the database, while the clients measured the name. The complete failover-time budget is role move plus DNS propagation plus client timeout behavior plus application retry, and only the first term appears in sys.dm_hadr views. When I audit an AG now, the checklist ends with a client-side test from an application server, because that is the only instrument that sees all four terms at once.
Watching failover behavior with MonPG when SQL Server support lands
The signals that matter here are failover events with real client-visible reconnection time, listener DNS TTL as a configuration fact, and the fraction of connecting clients whose drivers support parallel IP attempts. A failover that the cluster reports as nine seconds but clients experience as six minutes is a configuration bug, not a database event, and it should page whoever owns the connection strings. MonPG monitors PostgreSQL in production today; SQL Server support is on the roadmap and in active development, and the SQL Server monitoring (coming soon) page carries the honest status. Until it ships, put MultiSubnetFailover=true in every connection string you can find, drop HostRecordTTL to 60, and time your next planned failover from the app tier, not the cluster log.