The worst Galera outage I ever owned started with the network team doing completely reasonable maintenance. We ran two MariaDB 10.11 nodes behind a proxy, and when the switch between them went down for ninety seconds, both nodes stayed up, both kept answering pings, and both refused every write for the next forty minutes. The application did not fail over to anything; it just stalled, throwing error 1047 until a human re-formed the cluster by hand. The database did exactly what it was designed to do. We had designed it wrong, and the design error was the number two.
Galera's split-brain protection is the feature everyone wants and the mechanism everyone misconfigures. This is the arithmetic I now run through with every team before they pick a node count, plus the recovery drill for the day the whole cluster loses power.
How does Galera decide which nodes may accept writes?
By quorum: a group of nodes may accept writes only if, together, they hold more than half of the total vote weight of the previous primary component. Every node votes with its pc.weight, a provider option that defaults to 1, so in a default three-node cluster each membership change asks whether the surviving group holds at least two of the three votes. The group that passes that check becomes (or stays) the Primary Component, and wsrep_cluster_status reads Primary on every member of it. Everyone else flips to Non-Primary and stops serving.
The reason for the strict majority is that a node that loses contact cannot tell the difference between "the others died" and "my network died". If both halves of a partition kept writing, you would get two divergent copies of the database — the split brain the protocol exists to prevent. Demanding more than half the votes guarantees that at most one side can ever pass the check, because two disjoint groups cannot both hold a majority of the same total. It is not a heuristic; it is arithmetic, and it is the whole reason node counts matter.
One refinement worth knowing: quorum is computed against the last known primary component, and it is weighted, not per-node. You can change a node's vote at runtime by setting pc.weight, which is one of the few provider options that is dynamic. I use weights rarely and reluctantly — they make the arithmetic harder to reason about during an outage — but they are the right tool when you deliberately want one failure domain to outrank another.
Why is an even number of nodes a design smell?
Because an even count buys availability on paper and delivers less of it than the odd number below it. Take the two-node cluster from my incident: when the link partitions, each side holds one vote out of two. One is exactly half, not more than half, so neither side can form a primary component and the entire cluster refuses writes. Your two-node cluster tolerates zero failures for write availability — a single standalone MariaDB server is strictly more available, and cheaper.
Four nodes have the same shape of problem. A 2-2 partition is a full write outage, and four nodes tolerate exactly one node failure, which is the same failure tolerance as three nodes with an extra machine's cost and an extra certification hop. The general rule: with weight 1 everywhere, survival requires floor(n/2) + 1 visible votes, so three nodes survive one failure, five survive two. Odd numbers only. If someone proposes a four-node cluster, the correct response is to propose three, or five, or three plus an arbitrator.
What is garbd and where should it run?
garbd — the Galera Arbitrator daemon — is a process that joins the cluster as a voting member but stores no data and applies no writesets. It ships in the Galera package alongside the provider library, connects through the same gcomm address as the real nodes, and exists for exactly one job: casting a tie-breaking vote so a cluster can run an odd number of votes without an odd number of databases. For the two-node cluster that cannot grow a third database, adding garbd turns a 1-1 deadlock into a 2-of-3 majority for whichever side keeps the arbitrator.
Placement is the entire decision. The arbitrator must live in its own failure domain, or you have not added a tie-breaker — you have just weighted one side. garbd on the same host or rack as node A means "A's side always wins", which is a legitimate choice if you make it on purpose, and a nasty surprise if you make it by accident. It is lightweight enough to run almost anywhere reachable — a small VM, a utility box in a third site — but do notice that it still receives the full replication stream, so give it a network path that can keep up. One more operational note: because it holds no data, garbd can never be an SST donor, and it should never appear in your application or proxy connection lists.
What actually happens to a node that loses quorum?
It stays alive and refuses to serve. wsrep_cluster_status flips from Primary to Non-Primary, and any statement that needs certification — writes, and by default reads as well — fails with ERROR 1047, the "WSREP has not yet prepared node for application use" error that every Galera operator learns to recognize. Reads are refused too because a node outside the primary component cannot certify how current its data is; if you knowingly want stale reads from a partitioned node for debugging, wsrep_dirty_reads allows exactly that, explicitly, per session. Do not leave it on for applications.
The two provider options you should never touch in production are pc.ignore_sb and pc.ignore_quorum. The first lets a node process updates while split-brained, the second skips the quorum calculation entirely. Both exist for narrow master-slave recovery scenarios, and both are how you manufacture genuine divergent data if you set them on more than one side of a partition. If a runbook tells you to set either one without a paragraph of justification, distrust the runbook.
-- the four signals that tell you whether this node may accept writes
SHOW GLOBAL STATUS
WHERE Variable_name IN
('wsrep_cluster_status', 'wsrep_cluster_size',
'wsrep_local_state_comment', 'wsrep_ready');
-- the full provider option list, weights included
SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';The checks that matter for alerting are blunter than most dashboards make them. wsrep_cluster_status should be Primary on every node you expect to serve; anything else is a page. wsrep_cluster_size should equal your expected member count; a drop means a node left, and a drop to one-third of the expected size means the survivors are one failure from a full outage. wsrep_ready OFF means the node is not serving at all. Sample these per node, not through a proxy that hides which member you are talking to — I have watched a load balancer happily report a healthy cluster while two of three nodes sat non-primary behind it.
How do you bring the cluster back after a full outage?
You pick the most advanced node, bootstrap it, and let the rest rejoin from it — and the picking is the part people get wrong. Since the provider releases that shipped with MariaDB 10.1.20, pc.recovery defaults to true, which means each node persists its last primary-component state (view and position) to disk. After a genuine crash that kills everything at once, that on-disk state is what lets you reconstruct who knew what. After a clean shutdown of all nodes, there is no primary component left at all, and the first node up must be told explicitly to create one.
The drill, which you should rehearse before the power event: on every node, run galera_recovery — or mariadbd --wsrep-recover — and read the recovered position line from the output or error log. It prints the state UUID and a sequence number. The node with the highest sequence number holds the newest writes, and that is the node you bootstrap: galera_new_cluster on systemd builds, or starting that node with --wsrep-new-cluster. Only after it is up and primary do you start the others normally, so they pull whatever they missed via IST or SST.
-- after galera_recovery identifies the most advanced node and it is bootstrapped:
SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';
-- emergency promotion of a stranded component, ONLY when the other side is confirmed dead:
SET GLOBAL wsrep_provider_options = 'pc.bootstrap=1';The disaster scenario is bootstrapping a stale node first. If a node holding newer transactions then tries to join, it is ahead of the cluster it is joining — there is no backwards IST — and the resolution is a state transfer that discards its newer work. Your most recent writes vanish quietly, at 3am, in the middle of a recovery everyone is rushing. The pc.bootstrap emergency switch above is for the different case where the cluster is still running but partitioned and you have verified — with your own eyes, not a ping — that the other side is down. Verified dead, not unreachable. Unreachable might mean alive and writing.
What is the two-datacenter trap, and what fixes it?
The trap is pretending that a cluster stretched across two datacenters can keep both sides writable through a partition. It cannot, ever: when the inter-DC link drops, at most one side can hold a majority, so at most one side keeps accepting writes. That is not a Galera limitation you can tune away; it is the price of split-brain protection, and any design document that promises active-active across two sites through a network partition is promising divergence with extra steps.
The real fix is a third failure domain. Put garbd — or a small third node — in a third site with independent power and network, and now either datacenter plus the tie-breaker forms a majority, so one full DC loss leaves the cluster writable. If a third site genuinely does not exist, then decide in writing which datacenter owns writes, give its nodes the larger pc.weight, and make the failover to the other DC an explicit, manual, runbooked event. What you must not build is the automation I see proposed every year: a script in DC2 that bootstraps a new primary the moment DC1 stops answering. The day the link partitions but both DCs are alive, that script creates two primaries and the divergent data quorum existed to prevent.
Where MonPG fits
The monitoring shape here is small and unforgiving: per-node wsrep_cluster_status and wsrep_cluster_size trended continuously, an alert on any non-primary minute, and a record of every bootstrap event, because an unexpected bootstrap is a story you want told by the graphs and not by the postmortem. Disclosure, as always in this series: I work on MonPG, which monitors PostgreSQL today — it does not monitor MariaDB yet. MariaDB support is coming soon and in active development, and cluster health is on the shortlist of signals it is being built around: component status per node, cluster-size drift against the expected membership, and quorum-edge conditions surfaced before they become write outages. Until the MariaDB side ships, the status queries above are the toolkit. And if PostgreSQL is also in your fleet, that workflow is live today — see the PostgreSQL overview and how the engines compare, or browse the field notes on the blog.