The first alert said connection timeouts. The second said the connection pool was exhausted. I RDP'd into the box expecting a runaway query and found the strangest dashboard I had seen in months: CPU at eleven percent, disk latency flat at two milliseconds, memory with forty gigabytes free, and the application unable to complete a single login. sys.dm_os_waiting_tasks showed over nine hundred sessions in THREADPOOL, a wait type that means something more fundamental than any lock or latch: the instance has run out of worker threads. Sessions were connected, authenticated, and waiting for a thread to execute on, and none existed. New requests were stacking up faster than old ones could finish, which on a thread-starved box is a death spiral measured in seconds.
That morning's root cause turned out to be a blocking chain holding three hundred worker threads hostage while a slow report held a table lock. But the THREADPOOL wait is the symptom of several different diseases, and the right fix depends entirely on which one you have. This is the write-up I wanted that morning: what worker threads are, how many you actually have, how to see who is consuming them, and why the instinct to raise the ceiling is usually wrong. Everything here applies to SQL Server 2016 through 2022.
What is a worker thread, and how many do I actually have?
A worker thread is the execution unit SQL Server schedules to run a request, and the instance keeps a bounded pool of them — not one per connection. Every batch your application sends gets assigned a worker from the pool, the worker executes the request, and the thread returns to the pool for the next request. Connections are cheap and plentiful; workers are expensive and finite. The default ceiling is computed from CPU count and architecture: on 64-bit systems it is 512 plus sixteen per logical core above four, so a sixteen-core box gets 512 + (16-4) times 16, which is 704 workers. You can check your own:
SELECT max_workers_count
FROM sys.dm_os_sys_info;
SELECT scheduler_id,
status,
current_workers_count,
active_workers_count,
work_queue_count,
pending_disk_io_count
FROM sys.dm_os_schedulers
WHERE status = N'VISIBLE ONLINE';The second query is the one to run during the incident. When active_workers_count approaches max_workers_count and work_queue_count is climbing, runnable work is waiting for a thread — that is THREADPOOL in DMV form, before you even look at wait stats. The schedulers view also quietly explains why the box looks healthy: the workers that exist are busy or blocked, so the CPUs are mostly parked. Eleven percent CPU with a four-figure queue is not a contradiction. It is the signature.
One structural point that matters for the fix: workers are not reserved per database or per application pool. A single connection string with runaway concurrency can consume the worker supply for every application on the instance. There is no tenant isolation at the worker layer, which is why one bad report can take down the login endpoint of an unrelated service.
What actually consumes the pool?
Three patterns account for nearly every THREADPOOL incident I have worked. First, blocking: a waiting request still owns its worker. Every session queued behind the lead blocker in my blocking chain notes is holding a thread while it waits on LCK_M_, so a deep chain is a worker leak — three hundred blocked sessions means three hundred workers out of circulation, doing nothing, unable to serve anyone else. The queue feeds itself: blocked sessions hold workers, fewer workers means slower service for everyone else, slower service means more concurrent sessions, more sessions means more workers demanded. That is the death spiral from the opening story.
Second, parallelism doing multiplication. A query running at DOP 16 does not consume one worker, it consumes sixteen plus a coordinator, and a workload of such queries multiplies the worker demand by the degree of parallelism. I have seen a single mis-tuned report at DOP 32 on a busy instance pull the pool down by a third by itself. If your instance shows THREADPOOL pressure alongside high CXPACKET waits and you have not tuned the settings from my MAXDOP and cost threshold notes, that is the first thing to fix, because it is the cheapest worker recovery available.
Third, long-running requests that are not blocked and not parallel — they are just slow or stuck on external waits. The classic offenders are linked server queries waiting on OLEDB, sp_OA automation calls, SQLCLR waiting on something outside the engine, and BULK inserts against a slow file share. These sessions look innocent in blocking queries because nothing is blocking them; they are legitimately busy, on a thread, for minutes at a time. A workload with a steady stream of five-minute external-wait queries needs five minutes of worker-time each, and at high enough arrival rates the arithmetic guarantees exhaustion regardless of how fast the local queries are.
How do I see who is holding the workers?
Group the waiting tasks by wait type first — that one aggregation tells you which of the three diseases you have — then pull the running requests by elapsed time to find the thread-hogs that are not waiting at all:
SELECT wait_type,
COUNT(*) AS waiting_tasks
FROM sys.dm_os_waiting_tasks
GROUP BY wait_type
ORDER BY waiting_tasks DESC;
SELECT r.session_id,
r.status,
r.wait_type,
r.command,
DATEDIFF(second, r.start_time, SYSDATETIME()) AS elapsed_s,
s.open_transaction_count,
SUBSTRING(t.text, (r.statement_start_offset / 2) + 1, 200) AS stmt
FROM sys.dm_exec_requests AS r
JOIN sys.dm_exec_sessions AS s
ON s.session_id = r.session_id
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) AS t
ORDER BY r.start_time;If the top wait type is LCK_M_ anything, go walk the blocking chain — the workers will come back the moment the lead blocker is dealt with, and the whole THREADPOOL section of the postmortem collapses into the blocking section. If the waits are OLEDB or dominated by one long command type, you are in the external-wait disease. If THREADPOOL itself dominates with low elapsed times and no long requests, look at the arrival rate side: connection storms from the application, a retry loop hammering the instance, or a scheduled job farm all firing at the top of the hour. sys.dm_os_schedulers' work_queue_count trended over a minute tells you whether you are draining or filling.
Why is raising max worker threads the wrong first move?
Because the ceiling is almost never the constraint — the consumption is. sp_configure can raise max worker threads, and there are legitimate cases for it, but during an incident it is a trap for two reasons. First, the change requires a restart to take effect, so it cannot help you today. Second, more workers means more threads competing for the same CPUs and the same locks; if the disease is blocking, you have just given the chain more victims to hold. The death spiral gets wider, not shorter. I watched a team raise the ceiling from 704 to 2,048 on a box whose problem was a forty-minute table lock. The incident lasted exactly as long as the lock did, and afterwards they had a larger number in a config file and no explanation for why it had not helped.
The correct first moves are always subtractions: resolve the blocking, cap the parallelism, kill or reschedule the external-wait jobs, throttle the connection storm at the application. Only after the consumption side is genuinely lean — short transactions, sensible DOP, no five-minute linked-server queries — does the ceiling conversation begin, and it usually begins with a connection count question. If you legitimately run several thousand concurrent active requests, then yes, size the pool for it, on a quiet day, with a restart planned. As an emergency lever it does not exist.
One setting adjacent to this that people reach for instead: fiber mode, lightweight pooling, which switches workers from threads to fibers. Do not. It is off by default for good reasons — components like SQLCLR do not support it, error handling gets stranger, and it has not been the answer to a modern THREADPOOL incident in my career. The documented guidance and the field experience agree: leave it off.
What do I change so it does not happen again?
The durable fixes are the boring ones. Keep transactions short so LCK_M_ chains stay shallow — the sleeping-session-with-open-transaction pattern from the blocking article is a worker leak wearing pajamas. Set MAXDOP and cost threshold for parallelism deliberately instead of inheriting zero and five. Replace chatty linked-server patterns with staged data movement. Put connection limits and retry backoff in the application so a wobble does not become a storm. And watch the two numbers that predict the incident before it pages: active_workers_count against max_workers_count, and work_queue_count on the visible schedulers. A box drifting toward eighty percent of its worker pool on a normal Tuesday is telling you next month's peak will not fit, and it is telling you while you can still fix the consumption side calmly.
Watching worker saturation with MonPG when SQL Server support lands
The counters that would have turned my morning from an incident into a ticket are active versus max workers as a ratio, work queue depth per scheduler, THREADPOOL wait time trended next to LCK_M_ wait time so the death spiral shows up as two rising curves, and the count of long-running requests by command type. MonPG monitors PostgreSQL in production today; SQL Server support is on the roadmap and in active development, and the SQL Server monitoring (coming soon) page carries the honest status. Until it ships, the schedulers query above on a thirty-second schedule into a logging table is the whole monitoring stack for this failure mode — and unlike most instrumentation projects, it fits in a lunch break.