The alert said the database was refusing connections, which was not quite true. The database was refusing connections from exactly one IP address — 10.0.4.17 — and that one address was the NAT gateway for the entire Kubernetes namespace, so "one host" was in practice the whole application tier. The error the apps logged is burned into my memory: Host '10.0.4.17' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'. The cause took an hour to find because nothing had failed. A liveness probe, added to the database sidecar the previous Friday, opened a TCP connection to port 3306 every five seconds and closed it immediately — a textbook connectivity check. Every one of those connections aborted during the handshake, every abort incremented the host's error counter, and when the counter crossed max_connect_errors — default 100, reached by Sunday morning — MariaDB did exactly what it is designed to do and blocked the host. The Monday outage was the security feature working as documented.
This failure class is rite-of-passage stuff for MariaDB operators, and it keeps happening because the pieces are individually sensible: a DoS guard that blocks hostile hosts, a health check that verifies a port is open, a network that puts many clients behind one address. The interaction is the incident. Here is the mechanics, the prevention, and the monitoring that has kept us out of it since.
What exactly counts as a connection error, and when does blocking happen?
MariaDB counts failed handshakes per client host — connections that die during connect, authentication, or the initial protocol exchange — and blocks the host when that count reaches max_connect_errors, which defaults to 100. Established sessions that drop later are a different category entirely (those land in Aborted_clients and never block anyone); only errors before the session is fully up count toward blocking. The per-host bookkeeping lives in the host cache, and since MariaDB ships the Performance Schema version of it, the state is directly queryable:
-- per-host error state: who is close to being blocked?
SELECT IP, HOST,
SUM_CONNECT_ERRORS,
COUNT_HOST_BLOCKED_ERRORS,
COUNT_HANDSHAKE_ERRORS,
COUNT_AUTHENTICATION_ERRORS
FROM performance_schema.host_cache
WHERE SUM_CONNECT_ERRORS > 0
ORDER BY SUM_CONNECT_ERRORS DESC;
-- global counters: errors DURING connect vs sessions dying LATER
SHOW GLOBAL STATUS WHERE Variable_name IN
('Aborted_connects', 'Aborted_clients', 'Connection_errors_max_connections');
SHOW GLOBAL VARIABLES WHERE Variable_name IN
('max_connect_errors', 'host_cache_size', 'skip_name_resolve');Two operational facts fall out of the mechanics. The counter is cumulative and only resets on success — one successful handshake from a host zeroes its error count — so a host that occasionally succeeds never blocks; blocking requires an unbroken run of failures, which almost always means an automated client that never completes auth. And the block key is the host as MariaDB sees it: an IP when DNS is off or fails, a hostname otherwise. That is why NAT and load-balancer health checks are so dangerous — the blast radius of one blocked entry is everything sharing that address.
Why do health checks and scanners cause most blocks?
Because a bare TCP connect-and-close never completes the MySQL handshake, and from the server's perspective it is indistinguishable from a hostile probe — one connection error, every time, forever. Our incident's probe ran every five seconds from a single source, reached 100 failures in eight minutes... and then reset itself every time a real connection from the NAT gateway succeeded, which is why it took all weekend instead of eight minutes: the counter climbed only when the apps were idle. Saturday night, with the tier scaled down and quiet, the failures finally strung 100 together uninterrupted.
The same pattern comes from three other recurring sources. Security scanners doing port sweeps — if your vulnerability management does not exempt database ports or use protocol-aware checks, every scan is a blocking attempt. Load balancer TCP health checks pointed directly at 3306, which fail the handshake unless the balancer speaks the protocol. And misconfigured clients hammering auth with wrong credentials — COUNT_AUTHENTICATION_ERRORS in the host cache separates those from handshake aborts, which is how you tell a scanner from a broken deployment. The durable fixes, in order of preference: make health checks protocol-aware (mariadb-admin ping completes a real handshake, as does a SELECT 1 over real credentials from a dedicated probe account), point infrastructure checks at the application health endpoint instead of the database port, and only then consider raising max_connect_errors. Raising the threshold treats the symptom; it also weakens an actual DoS guard, so if you raise it, do it with the threat model in mind, not as an incident-silencing reflex.
How do you unblock a host, and how do you keep it unblocked?
FLUSH HOSTS clears the host cache and unblocks everything immediately — mysqladmin flush-hosts from a shell, or TRUNCATE TABLE performance_schema.host_cache from SQL, which is the same operation and the one to script because it needs no extra tooling. All three forms are safe mid-traffic; the cache rebuilds itself. The reflex to avoid is running the flush from a cron job as permanent duct tape: we did that for a month, and it converted a clear signal — something is generating broken connections — into silence, right up until a genuinely misbehaving client got masked by it. Unblock once, find the source from the host-cache counters, fix the source, and let blocking stay loud.
While you are in the config, settle skip_name_resolve deliberately. With it OFF (the historical default), every new connection pays a reverse DNS lookup; we measured 80 ms added to connect latency in an environment with a broken resolver, versus 2 ms with it ON — and failed lookups can themselves count toward connection errors. With it ON, grants must use IP addresses or wildcards rather than hostnames, and the host cache keys strictly by IP, which makes the NAT blast-radius question above even more explicit. My default on modern fleets: skip_name_resolve=ON, grants by subnet wildcard, connection latency freed from DNS, and one less failure mode in the connect path. If connection pressure is a broader theme on the box, the interplay with connection handling is covered in the thread pool tuning notes.
What should you monitor so this never pages you again?
Trend the two aborted counters separately, because they mean different things: Aborted_connects climbing means broken handshakes — probes, scanners, bad credentials — and is your early warning for a future block, while Aborted_clients climbing means established sessions dying — timeouts, network cuts, clients that never close — and points at application hygiene instead. Alert on the rate, not the absolute number; both counters only ever grow. On top of that, alert directly on blocking itself: a nonzero COUNT_HOST_BLOCKED_ERRORS sum in the host cache, sampled every minute, would have paged us on Saturday night instead of Monday morning, and the query is one line from the block at the top of this article. For per-account attribution when the errors are auth failures, the per-user counters from the userstat observability notes name the culprit account in one query.
The last mile is correlation. Connection-error rates move with deploys and infrastructure changes — new probes, new scanners, new NAT rules — so put the Aborted_connects graph on the same dashboard as your deployment timeline and the postmortem writes itself. If you collect Performance Schema data centrally, the low-overhead configuration from the Performance Schema low-overhead setup keeps the instrumentation bill sane while you do it.
Where MonPG fits
The signals worth trending here are the ones this article is built from: Aborted_connects and Aborted_clients as separate rates, blocked-host counts from the host cache, and connect latency as the canary for DNS problems. Full disclosure, as in every article of this series: I work on MonPG, which monitors PostgreSQL in production today and does not monitor MariaDB yet. MariaDB support is coming soon and in active development — the /mariadb-monitoring page tracks where it stands — and connection-path health is on the list of signals it is being built around: handshake errors and host blocks surfaced as first-class alerts instead of a surprise in the application logs. Until that ships, the queries above are your early-warning kit. If PostgreSQL is also in your fleet, that monitoring is live today — see the PostgreSQL overview, or browse more field notes on the blog.