The most informative crash I ever debugged produced an error log with nothing in it. A 5.7 primary fell over at 03:12 on a Tuesday, monitoring fired, and by the time I shelled in, mysqld_safe had already restarted it and the instance was serving traffic with a fresh uptime. The error log showed a clean startup sequence and absolutely nothing before it — no assertion, no signal 11, no InnoDB complaint. That silence was the diagnosis: InnoDB crash recovery from a genuine internal crash almost always leaves evidence, so a log that goes from normal chatter to a startup banner means the process died from outside. Sure enough, dmesg had it — Out of memory: Killed process 22107 (mysqld) total-vm:98.4G, anon-rss:91.2G. The kernel had executed the database, and the error log's job in that incident was to prove where not to look. Since then the error log is the first file I open in any incident and the file I tail during any risky operation, because almost everything MySQL knows about its own distress passes through it — but only if you know which lines are noise, which deserve a page, and what it means when the expected lines are missing.
What actually changed in the MySQL 8.0 error log format?
Every line became structured: a timestamp, a priority tag ([System], [Note], [Warning], [ERROR]), a message number like MY-011980, and the subsystem in brackets, replacing 5.7's free-text lines. This is not cosmetic. The priority tag is your first filter — [ERROR] lines are almost always actionable, [Note] lines are almost never, and [Warning] is where the interesting judgment calls live. The message number is stable across versions, which makes runbooks and alert regexes survivable across upgrades: match on the MY-code, not the English text, because the text gets reworded. Verbosity is controlled by log_error_verbosity, default 3 (errors, warnings, and notes); dropping it to 2 silences the notes, and on a busy server the notes are 95 percent of the volume — connection counts, shutdown and startup banners, audit chatter. The logging pipeline itself became component-based in 8.0: log_error_services picks the sink (the default log_filter_internal plus log_sink_internal writing the file, or log_sink_json / log_sink_syseventlog for JSON and syslog), and log_filter_dragnet can suppress or throttle specific message numbers entirely. If your 8.0 error log looks thinner than your 5.7 logs did, check whether someone installed dragnet rules, because a throttled error log is a blindfold with a config line.
-- where is the log going, and at what verbosity?
SELECT @@log_error, @@log_error_verbosity, @@log_error_services;
-- 8.0: which messages are being suppressed or rate-limited?
SELECT @@log_error_suppression_list;
-- a 5.7 line vs an 8.0 line for the same event:
-- 5.7: 2024-03-12T03:12:01.4123Z 0 [Note] InnoDB: Starting crash recovery.
-- 8.0: 2024-03-12T03:12:01.4123Z 0 [System] [MY-011980] [InnoDB] Starting crash recovery.Which warning lines deserve a page, and which are noise?
Page on [ERROR], and page on a short list of specific [Warning] lines that predict user-visible pain; the rest of the warnings are trend data, not alerts. The ones that have earned their place in my alert rules, with the judgment attached. Host 'web-04' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' — the server has crossed max_connect_errors (default 100) against one host, usually a misconfigured app box hammering with a wrong password, and every connection from that host now fails instantly; page, because it looks like a partial outage. InnoDB: page_cleaner: 1000ms intended loop took 4218ms. The settings might not be optimal. — despite the wording, this is rarely a settings problem; it is the flush threads reporting that the I/O subsystem stalled for four seconds, and a burst of these means your disk is saturated or your cloud volume hit a burst-balance wall; the redo capacity and checkpoint stall notes cover the follow-up. InnoDB: Difficult to find free blocks in the buffer pool with its failed-flush-attempts count — eviction is starving, almost always on a write burst against a too-small pool. Aborted connection 8451 to db: 'shop' user: 'app' host: '10.0.3.14' (Got an error reading communication packets) — one of these is a client that hung up rudely; a thousand per minute is a load balancer health-check misconfiguration or an app leaking connections, and it correlates with the Aborted_clients status counter. The noise to stop paging on: per-connection notes at verbosity 3, SSL version warnings on internal networks, and the startup deprecation list, which belongs in a ticket, not an alert channel.
How do you read a crash recovery sequence under pressure?
Read it as a four-phase narrative — redo scan, redo apply, rollback of uncommitted transactions, normal startup — and the lines tell you both how long the outage will last and whether the crash is about to repeat. A healthy recovery after a kill or power event looks like InnoDB: Starting crash recovery., then progress through the redo log (InnoDB: Applying a batch of redo log records ... with percentage lines on large logs), then InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 18740233 row operations to undo, and finally ready for connections. The redo apply phase is proportional to how far behind checkpointing was — another reason redo sizing matters — and the rollback phase is proportional to the largest uncommitted transaction at crash time, which can be the longest phase by far; the kill-and-rollback field notes cover why that phase cannot be skipped. The patterns that change your response: recovery that restarts from the beginning after an assertion or signal 11 means crash recovery itself is crashing — you are in corruption territory now, and the path goes through the force recovery ladder, not through patience. And a recovery that completes cleanly but keeps happening weekly at odd hours is not a MySQL bug until proven otherwise; it is an external kill, which brings us to the empty log.
What does silence in the error log actually mean?
A startup banner with no preceding shutdown or crash lines means mysqld died without executing its own death code — in practice, almost always the OOM killer or a kill -9 from a human or a systemd unit. This inversion is the single most useful error-log skill I know: the log's absence of evidence routes you to dmesg, journalctl -k, or /var/log/messages for the kernel's side of the story, and the kernel is blunt — Out of memory: Killed process (mysqld) with the RSS at the moment of execution. The follow-up investigation is memory arithmetic, not log reading: buffer pool plus per-connection buffers times real concurrency plus the miscellany of sort and temp allocations against the box's RAM, and the fix lives in the memory usage diagnosis workflow — or in innodb_dedicated_server if the box is truly single-purpose. One more silence worth knowing: a replica that stops applying with no error in the error log has usually stopped with an SQL thread error that only appears in SHOW REPLICA STATUS, not the log — the error log is not complete, and replication has its own reporting channel. Silence is a finding, but always verify which channel should have spoken.
How do you turn the error log into an instrument instead of a text file?
Ship it to your log pipeline with the MY-code preserved, alert on a denylist rather than grepping ad hoc during incidents, and rotate it on a schedule so the file stays greppable when you need it raw. Concretely: the denylist approach inverts the usual mistake — instead of alert-regexes for everything scary, you alert on [ERROR] plus the specific warning list above and review new line patterns monthly, which keeps alert volume at a level humans answer. Rotation matters more than people expect: the error log grows unbounded by default, and I have watched a two-year-old 14GB error log turn a simple grep into an I/O event on the very box being debugged; FLUSH ERROR LOGS closes and reopens the file (8.0 handles this cleanly, including renaming first), so wire it into logrotate with copytruncate avoided. During planned risky work — upgrades, big DDL, failovers — tail -f the error log in a spare terminal the whole time; the log speaks seconds before the metrics pipeline does, and in a failover those seconds are the difference between narrating the incident and investigating it. Finally, test your own alerts: inject a known line (a deliberate max_connect_errors block from a scratch host works) and watch it fire end to end. An error-log alert you have never seen fire is a hypothesis, not a control.
Where MonPG stands on MySQL
I build MonPG, so the honest line: MonPG monitors PostgreSQL today, and MySQL support is in active development, not shipped. The signals in this piece — error-log lines parsed into alertable events, crash-recovery phases on a timeline, aborted-connection bursts correlated with client hosts, and the OOM-kill silence annotated against kernel logs — are exactly what the MySQL work is designed to surface, so the log's story arrives before the customer's does. The MySQL monitoring (coming soon) page tracks that work as it lands. Until it ships, the same evidence-first approach runs on the PostgreSQL side today, and the rest of these MySQL field notes live on the blog.