The slowest part of a Galera rolling restart should be the restart. One night I watched a routine kernel patch on a 600 GB cluster turn into a three-hour incident: the rebooted node came back, asked for its missing writesets, got told they were gone, and started a full state transfer that pinned the donor's disks the entire time. Redundancy down to one healthy copy, IO latency doubled cluster-wide, all because of a 128 MB default nobody had revisited since the proof of concept. The fix was a four-line config change. The lesson was that gcache.size is not a tuning nicety — it is the knob that decides what "a node rejoins" costs.
What is the gcache, mechanically?
The gcache is a per-node, on-disk ring buffer of replicated writesets — the Galera cache file, galera.cache in the data directory by default, preallocated at startup to gcache.size. Every writeset the node replicates or receives is appended to the ring; when the ring wraps, the oldest entries are overwritten. The point of the cache is retransmission and state transfer: any node can be asked "give me every writeset since sequence number N", and it can answer from the ring if the ring still reaches back that far.
Two escape hatches sit alongside the ring. Writesets too large for a ring slot spill into the page store — files prefixed gcache.page, nominally sized by gcache.page_size (128 MB by default) and free to grow as far as the disk allows while they are needed, with gcache.keep_pages_size governing only how much of the store lingers after use — and there is an optional in-memory store (gcache.mem_size) for setups with RAM to spare. For sizing purposes the ring is what matters: it is the part that is guaranteed preallocated and bounded, and it is the part that fills first under write load. Remember also that every node keeps its own gcache, because any node can become a donor. Sizing one and not the others is sizing none of them.
IST versus SST: what does a too-small gcache actually cost?
When a node rejoins, it tells the cluster its last applied sequence number. If a donor's gcache still contains every writeset since that number, the joiner gets an Incremental State Transfer — a stream of just the missing writesets, applied in minutes while the donor keeps serving normally. If any single writeset in that range has been overwritten, incremental is impossible and the joiner falls back to a full State Snapshot Transfer: a complete copy of the donor's data.
The cost of that fallback depends on your wsrep_sst_method. With mariabackup — the method current releases and guides recommend, and what I run on every cluster — the donor stays online but spends the next hours reading the entire dataset and pushing it over the network while its buffer pool churns. With rsync — still the product default for wsrep_sst_method — the donor is locked read-only for the whole copy, which means your cluster's write capacity just lost a member for the duration. Either way, for the length of the transfer you are running with degraded redundancy and degraded latency, and on a big dataset "the duration" is measured in hours. My 600 GB incident over gigabit Ethernet was ninety minutes of transfer alone, plus catch-up. A four-gigabyte gcache would have made it a three-minute IST.
How do you size gcache.size from real traffic?
You measure your replication write volume and multiply by the longest absence you want to absorb. The raw material is a pair of status counters: wsrep_replicated_bytes, writeset bytes this node has originated and sent to the others, and wsrep_received_bytes, writeset bytes it has taken in from them. The ring caches both streams, so their sum is your cluster's write workload serialized. Sample each twice across a known interval, divide, and you have the bytes-per-second the cache must hold. Measuring only the replicated counter is the classic under-sizing bug: on any node that originates little of the write load — which, on a single-writer cluster, is every node but one — it sees a fraction of the real traffic.
-- replication write volume: sample twice, sixty seconds apart
SHOW GLOBAL STATUS LIKE 'wsrep_replicated_bytes';
SHOW GLOBAL STATUS LIKE 'wsrep_received_bytes';
-- bytes per second = ((second replicated + second received)
-- minus (first replicated + first received)) / 60
-- do this during your HEAVIEST window, not at noon on a quiet dayThen the arithmetic: gcache.size should hold peak replication bytes-per-second multiplied by your maintenance window, multiplied by a safety factor. My rule of thumb is 1.5 to 2 times headroom, because batch jobs cluster and because the night you need the window is never an average night. Worked example: if your evening batch pushes 40 MB per minute of replicated writes and you want a node to survive a two-hour maintenance reboot, that is 4.8 GB of traffic, so an 8 GB gcache. The 128 MB default covers about three minutes of that workload — which is why default-sized clusters SST after even short outages. gcache.size is not dynamic; changing it needs a restart, which is one more reason to set it deliberately, once, before the cluster is big enough that restarts hurt.
One asymmetry to keep in mind: a bigger-than-needed gcache costs disk and a preallocated file, and nothing else. A smaller-than-needed one costs full state transfers at the worst possible times. This is not a knob to economize on.
How do you verify a rejoin will be IST before you restart?
Compare two sequence numbers: the joiner's next needed position and the donor's oldest cached position. The donor exposes the latter as wsrep_local_cached_downto — the lowest sequence number still present in its gcache. The joiner's side is its wsrep_last_committed from before it went down, or the seqno line in its grastate.dat file on disk, plus one: that plus-one is the first writeset it will ask for. If that next needed sequence number is at or above the donor's cached_downto, every writeset in the gap is still in the ring and the rejoin will be incremental — assuming, as usual, that the joiner's state UUID matches the cluster's, which it does whenever the node is rejoining the cluster it left rather than coming up from a stale snapshot. If the position the joiner needs has already been overwritten in the ring, nothing you do at that point avoids the snapshot.
-- on the donor: oldest writeset still available in the ring buffer
SHOW GLOBAL STATUS LIKE 'wsrep_local_cached_downto';
-- on the joiner before shutdown (or read seqno from grastate.dat):
SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';
-- IST is possible when: joiner wsrep_last_committed + 1 >= donor wsrep_local_cached_downtoI run this check as a habit before any planned restart on a busy cluster, and I trend wsrep_local_cached_downto against wsrep_last_committed on every node to see the window in hours instead of bytes. Watching the window shrink during batch season has caught under-sized gcaches for me twice, both times before anyone rebooted into a surprise SST. During an actual transfer, wsrep_local_state_comment on the joiner walks through Joining, Joined, and Synced, and the donor's own comment is the discriminator: a donor serving a full snapshot shows Donor/Desynced, while a donor that never leaves Synced is serving incrementally. That pairing is the fastest way to confirm which kind of transfer you are getting, because an SST announces itself immediately on the donor side.
Does the gcache survive a restart?
Only if you ask it to. The gcache.recover provider option controls whether a node rebuilds its gcache from the on-disk ring file and page store at startup — and despite the file persisting on disk, recovery is off by default in the current provider documentation, which means a restarted node comes back with an empty gcache and cannot serve IST to anyone until it has cached fresh writesets. That produces the classic cascade: you restart node A for maintenance, and while A's cache is empty, node B blips — now B needs a transfer and nobody can donate incrementally, so B gets a full SST. Setting gcache.recover=yes (a static provider option, so it goes in the config file's wsrep_provider_options list, alongside gcache.size) closes that hole at the price of a slower startup while the cache is scanned. On every cluster I run, that trade is not close.
The full checklist, then: size the ring from measured peak bytes times your longest planned absence times two; set it on every node identically; enable gcache.recover everywhere; and verify with wsrep_local_cached_downto before any reboot that you care about. Galera gives you the incremental path for free — the only way to lose it is to let the buffer that feeds it fill up unnoticed.
Where MonPG fits
The monitoring shape is two trends and one alert: replication bytes per second from the replicated-plus-received counters so the sizing math stays honest, the cached-downto window per node in hours, and a page the moment any transfer comes out as SST instead of IST — because an unexpected SST is always a symptom, of either an undersized cache or a node that was gone longer than anyone thought. The honest disclosure: I work on MonPG, which monitors PostgreSQL today and does not monitor MariaDB yet — MariaDB support is in active development and coming soon, and exactly these signals are on its Galera design list: IST-window depth trended per donor, SST events flagged as incidents rather than footnotes, and gcache sizing drift caught in config audits. Until it ships, the queries above carry the load. If PostgreSQL is also part of your fleet, that side of the workflow exists now — start with the PostgreSQL overview and engine comparison, and find the rest of the series on the blog.