I once rebooted a Galera node for a routine kernel patch expecting a fifteen-minute maintenance: drain, reboot, rejoin, done. The reboot took ten minutes. The rejoin took three hours. Instead of a quiet incremental catch-up, the node demanded a full state snapshot transfer — hundreds of gigabytes streamed from a donor that was also serving production reads, in the middle of the evening, while I watched a status variable refuse to say Synced. The node had done nothing wrong. The cluster's write cache had simply rolled over during the nightly batch window, and the writesets from before my reboot no longer existed anywhere to stream. That was the night I learned that in Galera, the difference between a boring rejoin and an incident is a ring buffer.
That ring buffer is the gcache, and the two rejoin paths it decides between are IST and SST. This post is how the decision actually gets made in MariaDB's Galera cluster, how to size the cache for your real downtime, what your SST method choices cost the donor, and what to watch while a node finds its way back.
How a rejoin actually works
When a node rejoins the cluster, it announces its last known state: the cluster state UUID plus the sequence number of the last writeset it applied. The cluster picks a donor, and the donor answers one question: do I still have every writeset that came after that sequence number? Those writesets live in the donor's gcache — an on-disk ring buffer (the galera.cache file, sized by gcache.size, default 128 MB) that exists for exactly this purpose. If the answer is yes, the joiner gets an Incremental State Transfer: the donor streams just the missing writesets, the joiner applies them, and the node is Synced in seconds or minutes. Nobody copies any data files. If even one writeset in that range has been overwritten by newer entries in the ring, incremental transfer is impossible, and the joiner falls back to a full State Snapshot Transfer — a complete copy of the dataset from the donor.
The brutal part is that the decision is binary and made by the oldest missing writeset, not by the size of the gap. A node that is behind by ten transactions is exactly as SST-bound as a node behind by ten million, if transaction number one of those ten is gone from every donor's cache.
Sizing the gcache for your real downtime
The sizing question is simple to state: how many bytes of writesets does the cluster generate during your longest expected absence? Measure the write rate from the cluster's own counters, multiply by the window, and double it for safety:
-- Writeset volume over time; sample these twice and take the delta:
SHOW GLOBAL STATUS LIKE 'wsrep_replicated_bytes';
SHOW GLOBAL STATUS LIKE 'wsrep_received_bytes';
-- The gcache lives inside the provider options string:
SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';
-- Look for: gcache.size = 128M (the default — almost always too small)
-- During any join, this one variable runs the state machine:
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';Worked example: if wsrep_replicated_bytes grows by 30 MB per minute, an hour of downtime needs 1.8 GB of ring. An eight-hour overnight window with a batch spike might need 30 GB, which sounds absurd until you price three hours of SST against thirty gigabytes of cheap disk. Two refinements that matter in practice. First, size for your busiest hour, not your average hour — nightly batches are precisely when rejoins happen, because batches are precisely what triggers maintenance. Second, on current Galera 4 versions, gcache.recover lets a restarted donor rebuild its ring from disk; historically, restarting a donor zeroed its cache and doomed the next joiner to SST, so a rolling restart of the whole cluster used to guarantee at least one full snapshot. If you run an older version, treat every donor restart as an SST generator and plan accordingly.
When IST is impossible: picking the SST method
Sooner or later every cluster does a full transfer — a rebuilt node, a long outage, a new member. wsrep_sst_method decides what that costs, and the honest tradeoff is between donor impact and transfer speed:
- mariabackup is the method you want in almost every case. It takes a physical hot backup on the donor and streams it to the joiner, using MariaDB's backup-stage locks to get a consistent snapshot with only brief blocking windows. The donor keeps serving through most of the transfer, at the cost of heavy read IO. Credentials and encryption are handled through wsrep_sst_auth and the method's TLS options.
- rsync is a raw file copy and often the fastest bytes-on-the-wire option, but the donor spends the bulk of the transfer under a global read lock — writes to the donor stall for the duration. For small datasets or a donor you have deliberately drained, that can be an acceptable trade. For a production donor with real write traffic, it is an outage you scheduled yourself.
- mysqldump is a logical dump: slow, blocking, and only defensible for tiny datasets. It exists; do not build your recovery plan on it.
Donor selection deserves one deliberate decision. Left alone, the cluster picks a donor for you, and it does not know which of your nodes has IO headroom or sits on degraded storage. Set wsrep_sst_donor on the joiner to a comma-separated preference list so rebuilds land on the node you would have chosen — and keep that list consistent in your automation, because the time you will need it most is the time everything else is already on fire.
The donor's bad day
While a donor serves an SST, its wsrep_local_state_comment reads Donor/Desynced. The desynced half matters: the node tells flow control to stop counting its receive queue, which protects the cluster from stalling behind the donor's extra workload — and also means the donor is allowed to fall behind on apply while it streams. A donor that serves a three-hour snapshot may return with its own backlog to work through: the writesets that piled up in its receive queue while it was desynced have to be applied before it is truly Synced. Watch it come back to Synced before you declare the maintenance over; a donor that re-enters traffic still behind is a flow-control incident waiting for its moment.
The practical rules that fall out of this: schedule rebuilds for quiet windows, never let automation pick your busiest writer as a donor during peak, and remember that two simultaneous joiners mean two simultaneous donors — if your cluster is three nodes, that is a one-node cluster with extra steps.
Watching the state machine instead of guessing
Everything a rejoin does is visible in one status variable. wsrep_local_state_comment walks the same path every time: Joining while the node requests and receives state, Donor/Desynced on the node serving the transfer, Joined once the state is in place, and Synced when the node is fully caught up and serving. The error log says the decision out loud — the lines around state transfer tell you plainly whether IST covered the gap or an SST was required, and which donor served it.
The alerts worth having are all derived from those two sources: a node sitting in Joining longer than your known SST duration is a stuck transfer, not a slow one; any production node flipping to Donor outside a maintenance window means something joined that you did not schedule; and repeated SSTs after short outages are the gcache telling you it is too small, in the most expensive language available. One automation trap to close: a supervisor that restart-loops a node through a failing SST will happily stream terabytes overnight while paging no one. Alert on the state, not on the process being alive.
Alert on the state machine, not the process
Take one thing from this post: alert on wsrep_local_state_comment and trend the replicated-bytes rate against your gcache.size, and a three-hour surprise SST stops being possible. Galera fails quietly — a status string and a log line, not a counter anyone graphs — which is exactly why it goes unmonitored. MonPG monitors PostgreSQL today; MariaDB support is coming soon — in active development now, and cluster state machines, gcache headroom, and donor impact during transfers are the MariaDB-specific signals it is being built to trend. On the PostgreSQL side, the same replication-lag-first workflow already exists — see the PostgreSQL overview, the engine comparison, and the blog.