Galera Flow Control on MariaDB: Why One Slow Node Freezes Every Write

The cluster froze every ninety seconds like it was on a timer. Three MariaDB 10.11 nodes in a Galera cluster, about 1.2 TB of data, and from two in the afternoon onward every write — INSERT, UPDATE, DELETE, on any of the three nodes — stalled for two to four seconds, recovered, then stalled again. The slow query log was empty. The processlist showed ordinary INSERTs sitting in the commit phase. No deadlocks, no lock waits, CPU and memory comfortable on all three boxes. The only number that moved in rhythm with the stalls was wsrep_flow_control_paused, and it took me two days to find the cause: node three was an older machine with a SATA SSD whose write performance had degraded. It was fsyncing in roughly 40 milliseconds where its twins managed 3. Every replicated writeset reached all three nodes at the same time, but node three applied them more slowly than the cluster delivered them — and Galera's answer to that situation is to stop the entire cluster and wait.

That behavior is not a bug, and it is not optional. It is flow control, the mechanism that keeps a synchronous cluster from tearing itself apart, and once you understand why it exists the tuning options shrink to a handful of honest choices. This is the model recap I needed that Tuesday, the metrics that actually diagnose it, the three knobs people reach for, and the architectural fixes that worked when the knobs did not.

How does Galera's synchronous replication actually work?

Galera is certification-based and virtually synchronous, and both halves of that phrase matter for flow control. When your transaction commits, the node does not just apply it locally and ship the result afterward. It collects the writeset — the affected row keys plus the changes — and broadcasts it through the group communication layer, which delivers it to every node in the same cluster-wide order. Each node then certifies the writeset against transactions that committed recently: do any of these rows overlap with something else in flight? If certification passes, the commit is acknowledged to the client, and the writeset lands in each node's receive queue, where applier threads pick it up and apply it.

The "virtually" is the important word. Certification and ordered delivery are synchronous — every node agrees on what committed and in what sequence before the client hears back. Application is asynchronous — each node replays the same writesets in the same order, but on its own hardware, at its own pace. Most of the time the gap between delivery and apply is milliseconds and nobody thinks about it. But it means apply speed is a per-node property of a cluster-wide pipeline. The cluster commits at the speed of certification, yet it only stays consistent if every member keeps up with the apply. This is also the fundamental trade-off against classic asynchronous replication, which I covered in the MariaDB replication vs Galera tradeoffs piece: async replicas may lag forever without bothering the primary, while Galera refuses to let any member drift — and "refuses" has a mechanism behind it.

Why does flow control exist in the first place?

Flow control exists to stop any one node from falling arbitrarily far behind on apply, because an unbounded apply gap is dangerous in three compounding ways. First, the lagging node serves stale reads — connections routed to it see progressively older data while looking perfectly healthy from the outside. Second, certification gets shakier: the longer a node trails, the more its certifying happens against state it has not yet applied, which raises the odds of certification conflicts and aborted transactions elsewhere. Third, drift that grows without bound ends in the worst outcome — a node so far behind that the only honest way to rejoin it is a full state snapshot transfer, which is an expensive, cluster-touching event.

So Galera watches each node's receive queue. When the number of received-but-unapplied writesets on a node exceeds gcs.fc_limit — 16 by default — that node broadcasts a flow control message, and the whole cluster pauses replication. Commits stall on every node, not just the slow one, until the queue drains below fc_limit multiplied by gcs.fc_factor. Then the cluster resumes, the queue starts growing again if the underlying imbalance is still there, and the cycle repeats — my ninety-second metronome. The frozen writes were the cluster protecting its slowest member from drowning, exactly as designed.

How do you read wsrep_flow_control_paused and paused_ns?

wsrep_flow_control_paused is the fraction of wall-clock time — since server start or the last FLUSH STATUS — that this node spent paused by flow control, and wsrep_flow_control_paused_ns is the same stall time as cumulative nanoseconds. A paused value of 0.18 means the node has spent eighteen percent of its life frozen; on the afternoon of my incident, node three sat between 0.15 and 0.22 during the bad windows and near zero overnight. The paused counter is a rate-of-time average, which makes it awkward to trend across status resets, so for graphing I prefer paused_ns: sample it twice, subtract, and you have exact stalled time per interval. Divide by the interval and you get the same fraction without worrying about when counters were last flushed.

The counters that tell you who is complaining are wsrep_flow_control_sent and wsrep_flow_control_recv. A node with a climbing sent counter is the one asking for pauses — it is the bottleneck, the one whose queue keeps overflowing. A node with recv climbing and sent flat is a victim: healthy, but throttled by somebody else's queue. On my cluster, node three's sent counter ticked up with every stall while nodes one and two only received. That one comparison replaces hours of guessing about which machine to suspect.

-- flow control: how much, and who is asking for it
SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_flow_control_paused',
   'wsrep_flow_control_paused_ns',
   'wsrep_flow_control_sent',
   'wsrep_flow_control_recv');

-- how deep is the apply backlog on this node?
SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_local_recv_queue',
   'wsrep_local_recv_queue_avg',
   'wsrep_cert_deps_distance',
   'wsrep_apply_oooe',
   'wsrep_local_state_comment');

-- applier parallelism, the knob most clusters leave at default
SHOW GLOBAL VARIABLES LIKE 'wsrep_slave_threads';

Two supporting numbers finish the picture. wsrep_local_recv_queue and its average tell you how deep the backlog runs between pauses — if the average keeps ratcheting upward even while flow control is firing, the node is losing ground and tuning will only postpone the reckoning. And wsrep_cert_deps_distance tells you how many writesets could, in principle, apply in parallel, which is the ceiling on what applier threads can buy you. More on that below.

Should you tune gcs.fc_limit, fc_factor, and fc_master_slave?

You can, and sometimes you should — but know going in that all three knobs change when the pain arrives, not why it arrives. They live in wsrep_provider_options, they apply cluster-wide, and they govern the flow control trigger itself:

-- check what the provider is actually running with
SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';

-- in the config, e.g. /etc/my.cnf.d/server.cnf, under [galera]:
--   wsrep_provider_options = "gcs.fc_limit=256; gcs.fc_factor=0.5"
-- defaults to know: gcs.fc_limit = 16, gcs.fc_factor = 1.0,
-- gcs.fc_master_slave = NO

gcs.fc_limit is the queue depth that triggers a pause. Sixteen writesets is tight for a write-heavy cluster: a modest burst fills it in a blink and the cluster pauses on noise. Raising it to a few hundred lets a briefly-lagging node buffer the burst and catch up without dragging everyone into a pause. The honest cost: queued writesets sit in memory, and a deeper queue means that when a pause finally does fire, the slow node has further to climb before the cluster resumes. You are trading frequent short stalls for rare longer ones.

gcs.fc_factor sets the resume point as a fraction of fc_limit. At the default of 1.0 the cluster resumes almost the moment the queue dips under the limit, which invites oscillation — pause, resume, pause, resume, with every writer paying latency on each swing. Dropping it to 0.5 means the node must genuinely halve its backlog before traffic restarts: fewer, cleaner pauses. Lower values stretch each pause further; there is no free setting here. gcs.fc_master_slave is the specialist: set it on a cluster with exactly one writer and Galera uses a more aggressive single-master flow control mode instead of the conservative multi-master default. On a genuinely multi-writer cluster, leave it alone. Every one of these knobs treats the symptom. A queue that grows between pauses is the disease, and no fc_limit in the world cures it.

What actually causes flow control stalls?

In every case I have debugged, the root cause was one of three things: storage on one node, oversized transactions, or applier parallelism left at the default. Storage first, because it was my Tuesday. Applying writesets is write-heavy work, so fsync latency is destiny, and a node does not have to be broken to be slow — it only has to be slower than its siblings. The classics are an aging SSD whose sustained write rate has collapsed, a RAID controller battery failure that silently flips the write cache from write-back to write-through, or a noisy-neighbor VM host. Flow control does not care that node three is fine most of the time; it cares about the tail. Compare apply rates across nodes during a stall, then go look at disk latency on the outlier.

Oversized transactions are the second cause, and they compound with the first. A single UPDATE touching half a million rows replicates as one writeset that a node must apply largely serially; while that monster replays, normal traffic queues behind it, the queue crosses fc_limit, and the cluster pauses. Our nightly batch job — 200,000-row updates in single transactions — was the amplifier on top of the weak disk, and the stall metronome tracked its schedule almost exactly. If your pauses correlate with batch windows, that is your lead; the detection workflow from hunting long transactions applies here unchanged.

The third cause is wsrep_slave_threads sitting at 1, which is the shipped default on the MariaDB versions I have run — meaning exactly one applier thread and zero parallel apply. Writesets touching different rows are independent, and wsrep_cert_deps_distance measures how wide that independent window runs on your workload; applier threads are how you exploit it. Sizing guidance I have landed on: start near the measured cert_deps_distance, cap it well below your core count, and watch wsrep_apply_oooe — the out-of-order apply fraction — climb as you add threads. The cost is real: more threads mean more out-of-order application and more certification pressure, so sixteen threads on a four-core box buys you contention, not throughput. But going from 1 to 8 threads on a workload with a wide dependency distance is often the single largest fix available.

What is the architectural answer when tuning is not enough?

The architectural answer is to accept that cluster write throughput is capped by the slowest applier, and to stop feeding that cap work it cannot digest. Once you internalize that, the design moves follow naturally. Route all writes to one healthy node through your proxy or load balancer instead of spraying writes across all three — multi-writer Galera multiplies certification conflicts on hot rows anyway, so single-writer routing usually buys you throughput on two fronts at once. Spread reads freely; they carry no apply cost.

Split the big transactions. Our 200,000-row batch updates became chunks of 10,000 rows per commit, run in the small hours, and the effect was larger than any knob: five hundred small writesets apply in parallel across applier threads and interleave with live traffic, while one giant writeset monopolizes a thread and dams the queue behind it. Fix or retire weak hardware rather than tuning around it — flow control will find the slowest node no matter how generous fc_limit gets, so keep the members homogeneous. And schedule bulk loads, imports, and maintenance writes into windows where a stall costs nothing. The cost sheet for the architectural route: single-writer routing gives up write-side failover transparency unless your proxy handles it, chunking batch jobs means touching application code, and replacing hardware means budget. All three were cheaper than the stalls.

Where MonPG fits

The signals worth trending here are exactly the ones that diagnosed my Tuesday: flow-control paused fraction and paused_ns per node, flow-control sent per node to identify the complainer, receive-queue depth, and cert_deps_distance against configured applier threads — sampled continuously, because a stall pattern that fires every ninety seconds at 2pm is invisible in a nightly report. Full disclosure, as in every article of this series: I work on MonPG, which monitors PostgreSQL in production today and does not monitor MariaDB yet. MariaDB support is coming soon and in active development — the /mariadb-monitoring page tracks where it stands — and Galera health is high on the list of signals it is being built around: these exact wsrep counters, graphed per node over time instead of discovered mid-incident with SHOW STATUS. Until that ships, the queries above are your early-warning kit. And if your fleet also runs PostgreSQL, that monitoring is live today — see the PostgreSQL overview, or browse more field notes on the blog.