The first time I ran a three-node MariaDB Galera cluster in production, I learned its defining lesson at an inconvenient hour: a Galera cluster has no such thing as a local problem. The application team reported that "the cluster is slow." Writes that normally took single-digit milliseconds were taking seconds, on all three nodes, against tables nobody had touched in the deploy that evening. The database was not overloaded. Two of the three nodes were nearly idle. The third node had a degraded RAID controller and was flushing at a fraction of its normal rate, and because Galera is synchronous, that one sick node was holding back every write on the healthy two.
That mechanism has a name — flow control — and understanding it is the difference between debugging Galera for an hour and debugging it for five minutes. This post is about how flow control works in MariaDB's bundled Galera replication, which counters actually tell the truth, and what to do when your cluster pauses.
How Galera replicates, in one paragraph
Galera is certification-based, virtually synchronous multi-primary replication, and it ships inside MariaDB Server 10.1 and later — no separate plugin to install, just the wsrep provider library and the usual cluster settings. When you commit a transaction on any node, the node broadcasts a writeset to every other node. Each node certifies the writeset (does it conflict with anything else in flight?) and then queues it for its applier threads. The commit returns to the client once the writeset is certified cluster-wide, not once every node has finished applying it. That gap between certification and apply is where queues live, and queues are what flow control watches.
Because every node certifies every write, the cluster's write throughput is bounded by the slowest applier. If one node cannot apply writesets as fast as the cluster generates them, its receive queue grows. Left alone, the queue would grow until the node ran out of memory or fell so far behind that it needed a full state snapshot transfer to catch up. Flow control exists to prevent exactly that.
What flow control actually is
When a node's receive queue exceeds a configured depth, that node broadcasts a pause message to the cluster. Every node, including the healthy ones, stops sending new writesets until the lagging node drains its queue below a resume threshold. From the application's perspective, writes simply stall — on every node, even the fast ones. This is not a bug. It is the design: Galera would rather stall everyone briefly than let one node drift into a state where it must be rebuilt.
Two provider options control the behavior, both visible inside the wsrep_provider_options variable. gcs.fc_limit is the receive queue depth that triggers a pause, and its default in current Galera 4 builds is 16 writesets. gcs.fc_factor is the fraction of that limit the queue must drain back below before replication resumes: the resume threshold is fc_limit multiplied by fc_factor, so with the current defaults of 16 and 1.0 replication resumes once the receive queue falls back below 16 writesets, not only when it has emptied completely. You can inspect the live values without restarting anything:
SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';
-- The counters that matter during a stall:
SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_paused';
SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_sent';
SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_recv';
SHOW GLOBAL STATUS LIKE 'wsrep_local_recv_queue%';
SHOW GLOBAL STATUS LIKE 'wsrep_local_send_queue%';wsrep_flow_control_paused is the headline number: the fraction of time, since the last FLUSH STATUS, that this node spent paused by flow control. A healthy cluster sits at 0.0. Anything persistently above zero means someone in the cluster cannot keep up. The subtlety is that it is a cumulative ratio, not a gauge — during an incident, poll it on an interval and look at the delta, or reset it with FLUSH STATUS so you are reading the current window rather than the average since Tuesday.
Finding the node that is actually slow
Here is the part that saves the hour. wsrep_flow_control_sent counts pause messages this node sent — in other words, how often this node told everyone else to stop. The node with the climbing wsrep_flow_control_sent is your laggard. The nodes with climbing wsrep_flow_control_recv are victims. During an incident I open one session per node and sample wsrep_flow_control_sent and wsrep_local_recv_queue a few seconds apart; the guilty node identifies itself almost immediately. The healthy nodes will also show a growing wsrep_local_send_queue during the pause, which looks alarming but is just writes backing up behind the stall.
Once you know which node is slow, the question becomes why, and the honest answer is that it is almost never "Galera overhead." It is something local to that machine:
- Storage trouble on one node. A degraded RAID array, a noisy-neighbor VM, an EBS volume that hit its burst balance. The appliers are write-heavy and fsync-sensitive; when flush latency climbs, apply throughput collapses. Check disk latency on the laggard first.
- A giant transaction. One transaction that updates two million rows becomes one enormous writeset that every node must certify, queue, and apply. Galera caps writeset size (wsrep_max_ws_size, around 2 GB), but anything even close to that will wedge flow control for a long, visible stretch. Batch large updates and deletes.
- DDL under TOI. Schema changes replicate with Total Order Isolation by default, which serializes them against the cluster. A long ALTER on one table can look exactly like a flow control incident from the outside.
- A donor or a joining node. While a node serves a state snapshot transfer (SST, usually via mariabackup) it is busy doing IO that has nothing to do with applying writesets. Schedule joins and rebuilds for quiet windows.
- Backup and reporting load aimed at one node. If your proxy sends all heavy reads to "the spare node," you have manufactured your own laggard.
- Certification conflicts on hot rows. Writers on different nodes hitting the same rows cause certification failures and rollbacks that waste apply capacity. That failure mode deserves its own article; the short version is to keep conflicting writes on one node.
Fixing flow control without superstition
The superstitious fix is to raise gcs.fc_limit so the cluster stops pausing. Do not. A deeper queue does not make the slow node faster; it just lets it fall further behind before anyone complains, and when it finally cannot catch up you are doing an SST at the worst possible time. Treat the pause as the alarm, not the problem.
The real fixes, in rough order of usefulness: repair or isolate the slow node's storage problem; break big transactions into batches; move bulk loads and backups off the write path or into quiet windows; and check that wsrep_slave_threads (the applier thread count) is sane for the laggard's hardware — the appliers parallelize by writeset dependency distance, and a node starved of apply threads on a many-core box is a self-own. Watch wsrep_cert_deps_distance while you tune it: it tells you how much parallel apply work the workload actually offers.
One legitimate lever is wsrep_desync. Setting wsrep_desync=ON on a node tells the cluster to stop considering that node's queue for flow control decisions, which is the right way to run a heavy backup or an SST donor without stalling production writes. It is also a loaded gun: a desynced node is allowed to fall arbitrarily far behind, so turn it off when the job finishes and verify the queue drains before you call the maintenance done.
Longer term, the fix is architectural honesty: flow control stalls are a capacity signal. If a single busy node can stall the cluster weekly, your write throughput is closer to the edge than your dashboards admit, and the answer is to fix the bottleneck or reduce the write rate — not to tune the alarm.
Watching it before it watches you
The counters above are only useful if someone is trending them. wsrep_flow_control_paused sampled every minute per node, wsrep_local_recv_queue_avg, and wsrep_flow_control_sent per node will catch every incident of this shape minutes to hours before the application team files the ticket. Galera exposes everything; almost nobody graphs it until after their first 3am.
Graph the pause ratio per node before the incident, with whatever tooling you already have — every counter in this post comes out of SHOW GLOBAL STATUS, and a cron job that samples them beats a 3am surprise. I work on MonPG, which monitors PostgreSQL today, not MariaDB; MariaDB monitoring is coming soon and in active development, and flow control is on the shortlist of signals it is being built around: per-node pause ratio trended as a first-class series, pause-message attribution so "which node is slow" is one glance instead of three sessions, and recv-queue depth correlated against disk latency so the root cause narrows itself. Until the MariaDB side ships, the queries in this post are the toolkit. And if PostgreSQL is also in your fleet, the same evidence-first workflow already exists there — see the PostgreSQL overview and how the engines compare, or browse the field notes on the blog.