MariaDB Multi-Source Replication: Why GTID Domain IDs Are Not Optional

For eight months the setup was the quietest part of our infrastructure. Three primaries — a billing database on MariaDB 10.11, a catalog database on 10.6, and an analytics feed owned by another team — each replicated into one reporting replica, also 10.11, through three named replication connections. Dashboards read from the replica, ETL jobs read from the replica, nobody touched the primaries for reporting traffic, and replication lag sat under two seconds on all three channels. Then the catalog primary's hardware died on a Saturday, its standby was promoted by the usual failover automation, and the reporting replica's catalog channel came back on its own. What it did next took me until Monday to fully understand: it re-applied roughly 40,000 transactions it had already applied, failed on duplicate keys, and left the catalog schema on the replica in a state where I could no longer trust a single aggregate. The root cause was one line that had never been set on any of the six servers involved: gtid_domain_id. Every one of them was writing GTIDs into domain 0.

Multi-source replication is one of MariaDB's genuinely differentiating features, and it is also one of the easiest to get subtly wrong, because it works perfectly right up to the first failover. This is the model I wish someone had handed me before Saturday: how named connections actually behave, what GTID domains do and why "leave it at default" is a loaded decision, and the operational discipline that makes a multi-source replica survive a source failover.

What does multi-source replication actually give you?

Multi-source replication lets one replica pull binlogs from several independent primaries at once. Each source is a named connection with its own IO thread, its own SQL thread, its own relay logs, its own position, and its own lag — fully independent replication streams sharing one server. The syntax difference from classic replication is a connection name threaded through every command:

-- each source is a NAMED connection
CHANGE MASTER 'billing' TO
  MASTER_HOST='billing-primary.internal',
  MASTER_USER='repl',
  MASTER_PASSWORD='not-in-your-repo',
  MASTER_USE_GTID=slave_pos;

CHANGE MASTER 'catalog' TO
  MASTER_HOST='catalog-primary.internal',
  MASTER_USER='repl',
  MASTER_PASSWORD='not-in-your-repo',
  MASTER_USE_GTID=slave_pos;

START ALL SLAVES;

-- per-connection status, or everything at once
SHOW SLAVE 'catalog' STATUS\G
SHOW ALL SLAVES STATUS\G

-- the replica's applied positions, ACROSS all domains
SELECT @@GLOBAL.gtid_slave_pos;

Two properties of this design shape everything else. First, the streams share nothing except the server: if the billing channel breaks, the catalog channel keeps applying, and Seconds_Behind_Master for one connection tells you nothing about the others. SHOW ALL SLAVES STATUS exists precisely because checking one channel at a time does not scale. Second — and this is the part that bit us — the data from all sources lands in one server, but the schema names do not have to be disjoint by configuration. If two sources both have a database called app, their tables intermix on the replica silently. Convention is your only guardrail: we enforce one-database-per-source and verify it in the provisioning playbook, because nothing in the server will.

The use case is consolidation, and it is a legitimate one: one analytical replica for several OLTP primaries, one backup target, one migration staging area while several legacy systems converge. It is not a write-scaleout mechanism and not a HA mechanism — it is a fan-in. Every operational question about multi-source reduces to "what happens to this fan-in when one arm changes shape," and the answer lives in the GTID machinery.

What is a GTID domain, and why does the default hurt?

A MariaDB GTID is a triple: domain_id, server_id, sequence_number. The domain_id is the part everyone ignores, because the default is 0 and single-source replication works fine on the default. The domain exists to answer one question: which independent stream of writes did this transaction come from? Every server writes its local transactions into the domain named by its gtid_domain_id, and the replica's applied position — gtid_slave_pos — is a comma-separated list of positions, one per domain. When you set MASTER_USE_GTID=slave_pos, the replica connects to a source and says, in effect, "send me everything in your stream after what I have already applied," matched per domain.

Now replay my Saturday. The catalog primary and its standby both had gtid_domain_id=0, which is correct for a failover pair — a promoted standby must continue the same domain so downstream replicas see one continuous stream. That part worked. What did not work was that the old primary had taken a handful of local writes during an earlier incident — a schema fix applied directly with sql_log_bin on, never replicated anywhere. Those errant transactions lived in domain 0 with sequence numbers that the standby's own writes later reused. From the reporting replica's perspective, its saved position in domain 0 pointed at events that meant something different on the new source than they had on the old one, and the stream it received replayed transactions over data that already contained them. Same domain, two different histories. GTID could not tell them apart because we had told it they were the same stream.

The discipline that prevents this has two rules, and both are cheap. Rule one: every independent write topology gets its own gtid_domain_id, set at birth, in the config file, before the server ever takes a write. Billing's pair gets domain 10, catalog's pair gets 20, analytics gets 30. A failover pair shares a domain deliberately; unrelated topologies never share one by accident. Rule two: nothing writes to a replica or standby outside the replication stream without sql_log_bin=0 for the session, full stop — one errant local transaction in a shared domain is exactly the poison that corrupted my Saturday. If you must take a local write on a standby, binlog-disabled and documented, or in a scratch schema that replicates nowhere. The audit workflow for finding errant transactions after the fact is close kin to the one described for MySQL in hunting GTID errant transactions; the domain discipline is what keeps you from needing it.

How do you fail over one source without breaking the fan-in?

The failover procedure for one arm of a multi-source replica is a per-connection operation, and the connection name is what saves you. When the catalog primary died, the correct sequence — the one I should have run — was: stop the catalog connection only, verify the new source's GTID stream against the replica's saved position for catalog's domain, repoint the connection, and restart it:

STOP SLAVE 'catalog';

-- on the NEW source: does its binlog history contain our position?
SELECT @@GLOBAL.gtid_binlog_pos;

-- on the replica: what have we applied from domain 20?
SELECT @@GLOBAL.gtid_slave_pos;

-- repoint ONLY the catalog channel at the promoted standby
CHANGE MASTER 'catalog' TO
  MASTER_HOST='catalog-standby.internal',
  MASTER_USE_GTID=slave_pos;

START SLAVE 'catalog';
SHOW SLAVE 'catalog' STATUS\G

The comparison between gtid_binlog_pos on the new source and your domain's slice of gtid_slave_pos on the replica is the entire safety check. If the new source's binlog position for the domain is behind what the replica has applied, the promoted standby was itself behind when promoted, and you are about to lose transactions silently — that is a business decision to make with open eyes, not a replication detail to notice later. If it is ahead, the replica will simply catch up. And if you ever have to rebuild one channel from scratch, RESET SLAVE 'conn' ALL clears just that connection's state and leaves the other channels untouched, which is exactly the granularity you want during an incident at 2am.

Two more production notes that cost us time before they became checklist items. Keep server_id unique across every server that participates in any of these topologies, including the standby servers that might someday be promoted — a duplicated server_id in a chain causes the replica to silently skip events, and the symptom is missing rows, not an error. And decide deliberately about log_slave_updates on the multi-source replica: if anything downstream ever replicates from it, or if you take binlog-based point-in-time backups from it, you need the replicated events written to its own binlog, and they will be written tagged with their original domains — which is another reason domain hygiene matters, because that binlog is now a three-domain stream that a point-in-time recovery tool has to untangle. The PITR side of that story is covered in the flashback field notes.

What does day-to-day monitoring look like?

Per-connection, always. The metrics that matter are per named channel: connection state of the IO and SQL threads, Seconds_Behind_Master per channel, and relay-log disk consumption per channel, because one channel that stalls on a network partition fills disk while the other two look healthy. Our dashboard graphs lag as three independent lines with independent alerts, and the alert text names the connection, not the server. A single aggregated "replication lag" metric for a multi-source replica is a way to get paged about nothing while a real stall hides in the average.

The structural metrics are worth trending too: the number of domains present in gtid_slave_pos (it should equal your source count plus any deliberate local-write domain, and a new domain appearing means someone wrote locally with the binlog on), and the divergence rate between gtid_binlog_pos and gtid_slave_pos per domain as an early lag signal that works even when Seconds_Behind_Master's heartbeat-based estimate is lying to you. Under parallel replication the picture gets one layer deeper — the optimistic parallel applier interacts with multi-domain streams in ways worth understanding, and I covered that machinery in the optimistic parallel replication piece.

Where MonPG fits

The signals worth trending on a multi-source replica are the per-channel ones from the middle of this article: IO and SQL thread state per named connection, per-channel lag, relay-log growth per channel, and the domain count in gtid_slave_pos as a tripwire for local writes. Full disclosure, as in every article of this series: I work on MonPG, which monitors PostgreSQL in production today and does not monitor MariaDB yet. MariaDB support is coming soon and in active development — the /mariadb-monitoring page tracks where it stands — and replication health is on the list of signals it is being built around: per-channel state and lag surfaced continuously instead of reconstructed from SHOW ALL SLAVES STATUS during an incident. Until that ships, the queries above are your early-warning kit. And if PostgreSQL is also in your fleet, that monitoring is live today — see the PostgreSQL overview, or browse more field notes on the blog.