MySQL Server Has Gone Away: wait_timeout, Idle Pools, and the 8-Hour Cliff
Every morning at 06:05 the first report query failed with 'MySQL server has gone away' and succeeded on retry. The pool held connections idle past the default 8-hour wait_timeout, and the server had closed them silently overnight. The fix was pool validation — but first we had to enumerate every other cause of the same message.
The error had a schedule, which should have told us everything sooner. Every morning at 06:05, give or take a minute, the first report job of the day failed with MySQL server has gone away, retried, succeeded, and the on-call note said transient network blip. It ran that way for months because a self-healing failure is the easiest kind to ignore. The actual mechanism was arithmetic: our application pool acquired connections in the evening, the report tier sat idle overnight, and MySQL’s wait_timeout — default 28,800 seconds, exactly eight hours — closed every one of those connections on the server side while the pool slept, believing them alive. The first query of the morning wrote to a socket the server had already reaped, and gone away is what the client library calls a write to a dead peer. The retry worked because the pool discarded the corpse and opened a fresh connection. Nothing was wrong with the network at 06:05; the network had been perfectly healthy all night, which was precisely the problem. What made this incident worth writing up was not the fix — pool validation, one config line — but the diagnostic path, because gone away is a symptom with at least five causes and wait_timeout is only the most common one.
What does wait_timeout actually do, and why is the failure silent?
wait_timeout is the server’s idle budget for a non-interactive client connection: if no statement arrives for that many seconds, the server closes the connection and frees the thread. Interactive clients — anything the server believes is a mysql shell — get interactive_timeout instead, same default of eight hours. The silence is a TCP property, not a MySQL choice: when the server closes its end, the FIN it sends is routinely absorbed without the client noticing, because the client’s socket is idle in a pool with no read pending. The client learns the connection is dead only when it next writes — the first query of the morning — and the write either fails outright or gets an RST in reply. That is the half-open trap: one side has reaped, the other side has not heard, and no amount of server-side logging was going to warn us, because from the server’s view the close was routine housekeeping. The error log does note it at high verbosity as an aborted-connection-adjacent line, and the Aborted_clients status counter ticks, but nothing about a clean idle reaper looks like an error worth paging on — because it is not one. The bug lives in the gap between the server’s connection lifecycle and the pool’s assumptions, and you close that gap on the pool side.
-- the two idle budgets and their current values
SELECT @@wait_timeout, @@interactive_timeout;
-- is the server reaping or killing? trend these:
SHOW GLOBAL STATUS LIKE 'Aborted_clients';
SHOW GLOBAL STATUS LIKE 'Aborted_connects';
-- who is idle right now, and for how long (seconds)?
SELECT user, host, time, state
FROM information_schema.processlist
WHERE command = 'Sleep'
ORDER BY time DESC
LIMIT 20;
-- the transfer-side timeouts, for the other gone-away causes
SELECT @@net_read_timeout, @@net_write_timeout, @@max_allowed_packet;How do you fix the pool so idle death never reaches a query?
Three mechanisms, and we run all three because each covers a hole in the others. First, validation on checkout: the pool runs a trivial probe — SELECT 1, or better the driver’s isValid() which uses the protocol ping — before handing a connection to application code, at a cost of one round trip per checkout that nobody has ever measured in production. This alone would have fixed the 06:05 failure. Second, idle eviction on the client side: configure the pool to retire connections after less idle time than wait_timeout — we use 300 seconds against the default 28,800, an enormous safety margin — so the client closes gracefully before the server ever reaps. Third, keepalive where the driver supports it: TCP keepalive or periodic pings on long-idle pooled connections, which keeps NAT state and server-side liveness aligned on paths with middleboxes that have their own idle timeouts, typically far shorter than MySQL’s — cloud load balancers and NAT gateways reap idle flows at 350 seconds or less, and that middlebox timeout, not wait_timeout, is the real cliff for connections crossing one. The connection pool sizing notes cover how many connections you should hold; this is the companion question of how long you may hold them. One practice to reject: lowering wait_timeout on the server to force the issue. It does force it — onto every client, including the ones you forgot, including long-running maintenance jobs whose connections idle between statements. Fix the pools; leave the server default alone unless you enjoy finding forgotten clients at 06:05.
What else produces the same error message?
Four causes we now rule out in order before touching any timeout, because gone away is the client library’s generic name for the connection died and the interesting question is always which death. One: the server restarted — crash, OOM kill, or an unplanned failover — and every pooled connection is a corpse at once; the tell is a fleet-wide burst of the error at a single timestamp and a fresh Uptime value. Two: the query exceeded max_allowed_packet on send or receive, which kills the connection mid-statement; the tell is that the same large statement fails repeatably rather than only after idle time, and the max_allowed_packet notes walk that diagnosis. Three: net_read_timeout or net_write_timeout fired mid-transfer — a stored procedure that thinks for ninety seconds between result chunks will trip net_write_timeout’s default of 60 on the server side while the client stares at a stalled read. Four: something killed the query — an administrator, a pt-kill policy, a resource governor — and the client’s next interaction meets the closed connection; the long transaction notes cover the policies that kill on purpose. The diagnostic habit that sorts these fast: when the error fires, record whether it followed idle time, a big statement, a long-running procedure, or a server event. The 06:05 pattern — same error, same time, first query after idle — is wait_timeout with a pool-shaped smoking gun, and everything else has its own signature.
Where MonPG stands on MySQL
For MonPG product capabilities and setup information, see the MySQL monitoring page. Use the diagnostics in this article to identify the measurements and operational checks your deployment needs.