Taming the MySQL 8.0 Error Log with log_filter_dragnet and Components

The upgrade to 8.0 fixed a hundred things and broke one we did not expect: the error log went from a manageable 40MB a day to 1.8GB. The volume was almost entirely [Note]-class noise — an aborted-connection line for every load-balancer health check that hung up rudely, per-connection chatter at log_error_verbosity 3, and a chatty audit component someone had enabled and forgotten — but buried in that flood was real signal, and the week a page_cleaner stall warning scrolled past unread in forty megabytes of health-check spam, I decided the log needed engineering, not stamina. MySQL 8.0’s answer is its component-based logging pipeline, with log_filter_dragnet as the rule engine: you can throttle or drop individual messages by exact error code, symbol, SQL state, priority, or subsystem. We installed it, wrote a satisfyingly broad first rule — drop everything below WARNING — and congratulated ourselves on the quiet. Months later, during a crash investigation, I went to the error log to reconstruct the timeline and found the silence pristine and useless: the rule had deleted the startup banners, the shutdown records, and the connection churn that would have told me when the trouble actually started. The fix was not abandoning dragnet; it was learning to write rules like a surgeon instead of a censor.

How does the 8.0 logging pipeline actually work?

Every error-log event flows through a chain of components named in log_error_services — filters first, a sink last — and each component gets a chance to transform, suppress, or write the event. The built-in pieces: log_filter_internal applies log_error_verbosity (which priorities pass at all) and log_error_suppression_list (specific message codes to silence); log_filter_dragnet applies the user-written rules in dragnet.log_error_filter_rules; and the sinks — log_sink_internal writing the classic file, log_sink_json emitting JSON lines, log_sink_syseventlog going to syslog — do the actual output. Two operational facts matter more than the architecture diagram. First, dragnet is not loaded by default: you install it with INSTALL COMPONENT ‘file://component_log_filter_dragnet’, which registers it in the mysql.component table so it persists across restarts, and then you add it to log_error_services ahead of the internal filter and the sink — order matters, because the chain executes left to right. Second, both the service list and the rules are dynamic variables: SET GLOBAL applies them live to every subsequent log event, which means you can deploy a filter without a restart — and also deploy a bad one without a restart, which is how my banner-eating rule went fleet-wide in an afternoon.

-- what does the current pipeline look like?
SELECT @@log_error_services, @@log_error_verbosity;

-- one-time: load the rule engine (persists in mysql.component)
INSTALL COMPONENT 'file://component_log_filter_dragnet';

-- put dragnet first in the chain, before the internal filter and sink
SET PERSIST log_error_services =
  'log_filter_dragnet; log_filter_internal; log_sink_internal';

-- the rules themselves: throttle the spam, drop nothing by class
SET PERSIST dragnet.log_error_filter_rules =
  'IF err_symbol=="ER_ABORTING_CONNECTION" THEN throttle 1/60.
   IF prio==ERROR THEN throttle 1/60.';

Which dragnet rules have actually earned their keep?

Throttle rules aimed at specific, identified message symbols — never drop rules aimed at priority classes. The syntax is deliberately simple: a condition over the event’s fields (err_code like MY-010584, err_symbol like ER_ABORTING_CONNECTION, SQL_state, prio, subsystem, the originating component) and an action of drop or throttle N/M, where N/M means “let N through per M seconds.” The rule that repaid its cost within a day was the one in the block above: throttle aborted-connection notes to one per minute, which turned 1.4GB of daily health-check spam into a representative trickle that still told us the noise existed — and crucially, still told us when the pattern changed, because a throttle shows you the rate, while a drop shows you nothing. A second keeper throttles a known-harmless plugin chatter message by its exact MY-code, after we spent a week confirming it was, in fact, known-harmless. The discipline embedded in both: a rule names a specific message we have already observed and judged, and it throttles rather than deletes, so the log stays an honest sample instead of a curated fiction. If you need the ground truth of which lines are flooding before writing rules, the error log triage notes cover the reading side, and a week of counting messages by code in your log pipeline gives you the hit list with numbers attached.

What are the traps, including the one that bit me?

Three traps, and mine was the third. Trap one: persistence is split. INSTALL COMPONENT survives restarts via the mysql.component table, but dragnet.log_error_filter_rules is an ordinary dynamic variable — SET GLOBAL and it evaporates at the next restart, which produces the confusing incident where filtering “mysteriously turned off” after maintenance. Use SET PERSIST for both the service list and the rules, and then remember that persisted values live in mysqld-auto.cnf and override my.cnf — the SET PERSIST drift notes are the companion piece on keeping that file honest. Trap two: overlapping silence. log_error_suppression_list, log_error_verbosity, and dragnet rules all suppress messages through different knobs, and six months in, nobody remembers which knob is muting what; keep all filtering in one mechanism — we standardized on dragnet plus verbosity — and write the rules down where the next engineer will find them. Trap three, my scar: never write a rule on a class you have not enumerated. “Drop everything below WARNING” reads as hygiene and acts as amnesia, because priority classes contain the narrative lines — banners, recovery progress, connection churn — whose absence you will only notice when you need them, the way a clean log after a crash tells you nothing about the hour before it. Throttle by symbol, at a rate you chose, after watching the message for a week. That rule has survived every incident since.

How do you run error-log filtering as a discipline instead of a hack?

Treat the rules as code — versioned, reviewed, tested on one node, and audited monthly — because they are a control on your observability, not a cosmetic preference. Concretely: the dragnet rule string lives in the same repository as my.cnf, changes go through review with the justification attached (“throttle ER_ABORTING_CONNECTION to 1/60: 1.4GB/day of LB health checks, verified note-class, ticket 4821”), and a new rule deploys to one replica for a week before the fleet. The monthly audit answers two questions: is each rule still justified by current volume, and is the log itself behaving — a log that suddenly gets quieter is a signal too, since silence can mean a new suppression or a dying application rather than a healthy server; the log-volume anomaly check catches both. One more practice that paid off: run one instance in staging with the full unfiltered pipeline at verbosity 3, so when production’s throttled view leaves you wondering what a message’s real frequency is, you have a reference stream to compare against. The goal was never a small log file; it was a log where every line that survives is worth reading, and where the reader can trust that nothing they needed was silently erased. That is a design property, and dragnet is simply where you implement it.

Where MonPG stands on MySQL

For MonPG product capabilities and setup information, see the MySQL monitoring page. Use the diagnostics in this article to identify the measurements and operational checks your deployment needs.

Related documentation

• MySQL: server logs