The 3am version of this incident always looks the same. An app deploy goes out, the new pods come up with a connection pool misconfigured to ten times the old size, and within four minutes Threads_connected climbs from a healthy 300 to the max_connections ceiling. New logins get the too-many-connections error, the health check fails, the load balancer pulls the node, and the retry storm makes everything worse. I have been paged for this exact shape at least four times, at four different companies.
Why does MySQL fall over so hard here? The answer lives in how it executes connections, and the thread pool is one of the two real fixes. The other is never letting the connections in.
One thread per connection: the default model
Community MySQL maps every client connection to its own OS thread inside mysqld. The thread parses, plans, executes, and returns results, then either services the next statement on that connection or sits in Sleep. Creating threads is not free, so thread_cache_size keeps a pool of recently used threads around for reuse; with a sane cache you mostly avoid the create-thread cost on reconnect.
This model is simple and, up to a point, excellent. With tens or a couple hundred mostly-idle connections, each active query gets a thread, the OS scheduler does a decent job, and there is no intermediary adding latency. The failure is not the threads themselves; it is what happens when too many of them are runnable at once.
Why it falls off a cliff
Three things compound. First, context switching: hundreds of runnable threads on 32 cores means the kernel spends a growing share of CPU just switching, and every switch evicts cache lines the next thread wanted. Second, internal contention: more concurrent execution means more pressure on InnoDB latches, the lock manager, and the transaction system, and parts of those serialize. Third, memory: every connection can allocate sort, join, and read buffers on demand, and a thousand connections that each decide to do a big sort will push the host toward swap or the OOM killer, which I covered in the MySQL memory diagnosis post.
The signature I watch for is Threads_running, not Threads_connected, climbing into the dozens while throughput flattens or falls. That is the server telling you concurrency has passed the useful point. The counters:
SELECT variable_name, variable_value
FROM performance_schema.global_status
WHERE variable_name IN (
'Threads_connected',
'Threads_running',
'Threads_created',
'Threads_cached',
'Max_used_connections',
'Connection_errors_max_connections',
'Aborted_connects'
);Max_used_connections is your high-water mark since the last restart; Connection_errors_max_connections counts how many times you actually hit the ceiling and refused someone. A nonzero rate on the latter is a paging-grade signal. For a tour of what the threads themselves are doing, the thread states post is the companion read.
What the thread pool actually does
The thread pool plugin replaces one-thread-per-connection with a fixed set of worker threads grouped into pools, plus a queue. Connections hand their statements to the pool; workers execute them; when a statement blocks on I/O or a lock, the pool can start another worker or let an existing one pick up other work, depending on the implementation's stall detection. The goal is to keep the number of concurrently executing statements near the number of cores, which is where MySQL's throughput peaks, and let everything else wait in line outside the engine instead of thrashing inside it.
Availability is the awkward part. Oracle ships the commercial thread pool plugin only with MySQL Enterprise Edition. Percona Server for MySQL ships its own implementation for free, built on the same idea, with knobs like thread_pool_size (the number of groups, defaulting to the core count), thread_pool_stall_limit (how many milliseconds a worker may be unresponsive before the pool spawns another thread), thread_pool_max_threads, and thread_pool_oversubscribe. MariaDB has a thread pool built in. On the community builds of MySQL 8.0 and 8.4, you do not get one at all, which is worth knowing before you design an architecture around it.
Two operational lessons from running the Percona build. Keep thread_pool_size at the core count or slightly under; bigger pools reintroduce exactly the contention you were escaping. And know about the administrative back door: since MySQL 8.0.14 there is a dedicated admin interface, admin_address and admin_port, and on Percona builds those connections bypass the pool, so you can still log in when the pool is saturated. Older Percona releases called the same idea extra_port and extra_max_connections; on current builds the stock admin interface is the portable answer. Configure it before the incident, not during.
When the pool helps, and when it just rearranges the queue
The thread pool shines for workloads of many short statements from many connections: OLTP APIs, chatty ORMs, dashboard backends. In a connection storm it changes the failure mode from "server melts at 400 runnable threads" into "requests queue and latency rises predictably," which is a vastly better night.
It does not fix a queue full of blocked work. If every worker is waiting on the same hot row lock or a metadata lock — see the metadata lock post for that flavor of misery — the pool is just a tidier way to be stuck. Long reports mixed into the same instance will occupy workers; priority queues exist for exactly this in both the Enterprise and Percona implementations, but the cleaner fix is routing reports to a replica. And the pool does nothing for the client side: if the application opens 5,000 connections, you still pay per-connection memory and handshake cost. The first fix for a storm is always the application's pool, and connection pool sizing is where I would start; a multiplexer like ProxySQL in front of MySQL is the other legitimate lever.
Sizing max_connections honestly
max_connections is a safety valve, not a performance target. I set it from the top down: what the app tiers can genuinely open under worst-case fan-out, plus headroom for administrative connections, capped by what the host can afford in worst-case per-connection memory. Then I watch Max_used_connections over weeks. If the peak never exceeds a third of the limit, the limit is decoration. If the peak approaches it, the fix is upstream pooling or the thread pool, not a bigger number — a bigger number just moves the cliff further out and makes the eventual fall harder.
One screen for the next storm
If "Threads_running against deploy markers" sounds like the screen you wanted at 3am, that is the screen being built for the MySQL side of MonPG. Today the product monitors PostgreSQL only; MySQL support is on the way. The design notes from this incident class are specific: connection churn next to release events, Max_used_connections against the ceiling, and queueing behavior treated as a first-class signal rather than a mystery. If your fleet also runs Postgres, the shipping version of that evidence-first workflow is at MonPG for PostgreSQL, and the rest of the series lives on the blog.