MariaDB Too Many Connections: The Storm, the 151 Default, and the extra_port Escape Hatch

The deploy was unremarkable: a rolling restart of twelve application nodes, two at a time, the kind of change the team had shipped a hundred times. What none of us had priced in was that every restarted node would open its full connection pool on boot, all twelve pools at roughly once, while the connections from the dying processes were still sitting on the server idle, unreaped, waiting out a wait_timeout of eight hours. max_connections was 800 — already raised once from the 151 default after a previous scare — and the arithmetic of twelve nodes times 75 pool connections, doubled briefly by old and new processes coexisting, sailed straight through it. ERROR 1040: Too many connections. Health checks failed, the load balancer ejected nodes, restart logic retried harder, and the storm fed itself. The final indignity was operational: when I tried to connect as root to see what was happening, I got the same error as everyone else, because MariaDB’s reserved extra connection for SUPER users was already consumed by a monitoring agent someone had granted SUPER to. Twelve minutes of outage, caused by nothing except arithmetic.

Connection exhaustion is the most purely preventable class of database incident I know, and it keeps happening because three defaults and one courtesy feature interact in a way nobody owns. These are the notes from making it structurally impossible on our fleet.

What actually fills the connection table?

Three populations, and the storm is always a collision between them. First, the working set: connections actively running queries, almost never the problem. Second, the idle pool: connections held open by application pools, sleeping in SHOW PROCESSLIST with Command = Sleep, waiting for work that the pool will route to them eventually — this population is sized by your pool configuration multiplied by your node count, and it is the one that doubles during a rolling restart. Third, the zombies: connections whose clients died without closing, held by the server until wait_timeout or interactive_timeout expires, eight hours by default, which means every crashed deploy, every force-killed app process, and every laptop that closed its lid mid-session leaves a connection squatting on the server for a third of a day. The storm math is simply all three populations spiking at once against a ceiling nobody re-derived after the fleet grew:

-- the ceiling and how close you live to it
SELECT @@max_connections, @@wait_timeout, @@interactive_timeout;
SHOW GLOBAL STATUS LIKE 'Threads_connected';
SHOW GLOBAL STATUS LIKE 'Max_used_connections';
SHOW GLOBAL STATUS LIKE 'Aborted_connects';

-- the population breakdown, per host and per state
SELECT USER, HOST, COMMAND, COUNT(*) AS n,
       MAX(TIME) AS oldest_idle_seconds
FROM information_schema.PROCESSLIST
GROUP BY USER, HOST, COMMAND
ORDER BY n DESC;

Max_used_connections is the number that should be in every capacity review: it is the high-water mark since startup, and if it sits above 70% of max_connections you are one rolling restart away from this article’s incident. The processlist breakdown is the diagnostic that ends arguments — in our storm it showed 900 connections from app hosts in Sleep state and exactly 14 doing work, which redirected the postmortem from the database to the pool configuration in about ninety seconds.

How do you get in when the server is refusing everyone?

MariaDB keeps one extra connection slot beyond max_connections for a user with the SUPER privilege, and our storm burned it because a monitoring account had been granted SUPER years earlier for reasons nobody could reconstruct. First fix of the aftermath: audit SUPER grants and strip them from anything that is not a human-break-glass account. The second, better mechanism is extra_port: MariaDB can listen on a dedicated administrative port with its own extra_max_connections allowance, and because application traffic has no business knowing that port exists, it stays reachable when the main port is drowning. Configuring it is two lines and a firewall rule:

-- my.cnf: an admin door that the storm cannot crowd out
-- [mariadb]
-- extra_port = 3307
-- extra_max_connections = 10

-- verify after restart
SELECT @@extra_port, @@extra_max_connections;

-- then, through that door, the surgical response:
-- kill idle connections by host, oldest first, working
-- connections untouched
SELECT CONCAT('KILL ', ID, ';')
FROM information_schema.PROCESSLIST
WHERE COMMAND = 'Sleep' AND TIME > 300
  AND USER = 'app';
-- review the list, then execute it; never blind-pipe KILL

The kill discipline matters: KILL against Sleep connections is safe — the client discovers the loss on its next checkout and reconnects — but KILL against a running query rolls back its transaction, and against a big one that rollback can itself take minutes. During the storm we killed nothing for the first five minutes precisely because the processlist showed only idle connections draining naturally as pools hit their own limits; the recovery came from the deploy pipeline pausing itself, not from surgery. And if your failed-connection pattern instead involves hosts being blocked after repeated handshake errors — the max_connect_errors path — that is a different incident with the same ERROR-1040-adjacent symptoms, covered in the blocked hosts notes.

How do you size it so the storm never forms?

The formula we now enforce is connection budget equals max_connections times 0.8, divided across application nodes times pool size, with the result treated as a hard platform limit rather than a suggestion. Concretely: with max_connections at 800 and twelve nodes, each node’s pool may open at most 53 connections, and the pool configuration is validated against the budget in CI, because a pool size is an application-level decision with database-level blast radius and it should fail a build, not a page. wait_timeout dropped from eight hours to 600 seconds for application accounts, so zombie connections reap in minutes instead of persisting through an entire incident and its aftermath. The monitoring agent moved to a dedicated account with the privileges it actually needs and a reserved slot it can never exhaust. And for the workloads where thousands of mostly-idle connections are genuinely the shape of the traffic, the answer is the thread pool rather than a bigger number — the scheduling and queueing behavior is covered in the thread pool tuning notes, and the short version is that it decouples connections from worker threads so idle connections stop consuming scheduling capacity. Per-user connection caps via MAX_USER_CONNECTIONS add a final backstop, so a single misconfigured service can degrade itself but cannot take the server down with it — the same least-privilege instinct as the roles and privileges notes, applied to resource limits.

Results, for the record: max_connections back down to 800 with a real budget behind it, pools capped at 50, wait_timeout at 600, extra_port on every primary, and a standing alert at 75% of ceiling. The next rolling restart of the same twelve nodes peaked at 61% utilization and nobody noticed it happened, which is the correct amount of notice for a deploy.

Where MonPG fits

For MonPG product capabilities and setup information, see the MariaDB monitoring page. Use the diagnostics in this article to identify the measurements and operational checks your deployment needs.

Related documentation

• MariaDB: Aria storage engine