The longest forty minutes of my on-call year was an ADD COLUMN. The table held 900 million rows, the cluster was three healthy MariaDB 10.11 nodes, and the moment the ALTER started, every INSERT in the application began queueing — on all three nodes, not just the one running the migration. When the ALTER finally committed, the queue drained in seconds and nobody could explain why a schema change on one node had frozen writes everywhere. The explanation was three letters long: TOI.
Schema changes on Galera are a different discipline from standalone MariaDB, and the discipline has exactly two levers: wsrep_OSU_method=TOI and wsrep_OSU_method=RSU. This is how each one behaves under load, where each one bites, and the runbook I now follow before any DDL touches a cluster.
What does wsrep_OSU_method=TOI actually do to the cluster?
Total Order Isolation is the default, and it is exactly what it sounds like. The DDL statement is replicated as a statement and applied at the same point in the cluster-wide writeset sequence on every node. All nodes execute the same ALTER at the same logical moment, which keeps the schema identical everywhere — and which means every node takes the same metadata lock and the same table lock for the full duration of the operation. On a blocking ALTER, every write touching that table, on every node, waits or fails certification until the DDL commits. My frozen forty minutes was the cluster working precisely as designed.
TOI exists to prevent the nastier failure: DDL racing DML differently on different nodes and producing divergent schemas or certification storms that abort members. The price is that the slowest possible execution — a full table rebuild on your biggest table — becomes a cluster-wide event. The old wsrep_strict_ddl guardrail, which rejected DDL it judged unsafe for cluster-wide execution, was deprecated in MariaDB 10.6 in favor of wsrep_mode=STRICT_REPLICATION and removed outright in 10.7; on current releases the server trusts you to know what you are doing. Repay that trust with a timer: if the statement will not finish in seconds, it does not go out under TOI at peak.
When is TOI perfectly safe for DDL?
When the DDL is fast. Since MariaDB 10.3, ADD COLUMN runs instantly by default through ALGORITHM=INSTANT — a metadata change measured in milliseconds — so most column additions under TOI are a non-event even on huge tables. CREATE INDEX runs with the INPLACE algorithm, which avoids the full table copy but still has to read the table and sort the new keys — seconds on a small table, potentially as long as a rebuild on your biggest one, so rehearse it like any other blocking DDL rather than assuming it is quick. DROP INDEX is the genuinely cheap one, usually metadata-fast. CREATE TABLE for a new migration table, dropping an unused small table, RENAME — all fine under TOI.
The dangerous set is the physically rewriting set: ENGINE or ROW_FORMAT changes, MODIFY COLUMN that widens or retypes, anything that reports ALGORITHM=COPY. Estimate the duration first, on a restored copy with production row counts, not on a dev box holding a thousand rows. My rule of thumb: if the rehearsed duration exceeds the application's write-timeout tolerance, the DDL goes out via RSU, one node at a time, off-peak.
How does RSU work, step by step?
Rolling Schema Upgrade flips the unit of change from the cluster to the session. You connect to one node, SET SESSION wsrep_OSU_method='RSU', and run the ALTER there. Under RSU the statement executes locally only — it is not replicated — and while it runs, that node desynchronizes: it keeps receiving writesets from the cluster but queues them instead of applying, and it does not send flow control. The other two nodes keep serving normally. When the ALTER commits, the node works through its backlog, catches up, and returns to Synced. Then you repeat on the next node, and the next, until the schema has rolled everywhere.
-- on ONE node, in ONE session, in exactly this order:
SET SESSION wsrep_OSU_method = 'RSU';
ALTER TABLE orders MODIFY COLUMN status VARCHAR(32) NOT NULL;
SET SESSION wsrep_OSU_method = 'TOI';
-- before touching the next node, verify this one fully caught up:
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'; -- want: Synced
SHOW GLOBAL STATUS LIKE 'wsrep_local_recv_queue'; -- want: 0
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; -- want: full membershipTwo mechanics deserve respect. First, the desync: a node mid-RSU is receiving but not applying, so its receive queue grows for the entire ALTER duration. The same wsrep_desync machinery is what operators set manually to drain a node for maintenance without triggering flow control. Always confirm the queue drains to zero before moving on — otherwise you stack one catching-up node behind another and flow control eventually throttles the writers you were trying to protect. Second, the scope: you set wsrep_OSU_method per session, and the session scope is the safety feature. The moment your session ends, the node is back to TOI. The variable does accept global scope — SET GLOBAL wsrep_OSU_method='RSU' is valid and quietly hands RSU semantics to every future session — so the discipline is session-only, asked for explicitly every time, with an occasional SHOW GLOBAL VARIABLES to confirm nobody broke the rule.
What breaks during the RSU divergence window?
Between the first upgraded node and the last, the cluster deliberately runs two schemas at once, and this is where RSU incidents actually happen. If the application starts writing to the new column the moment node one finishes, those writesets replicate to nodes two and three — which do not have the column yet — and fail to apply. An apply failure is fatal to the node: it aborts to protect consistency, and your rolling upgrade has just become an unplanned state transfer. The contract is strict: during the window, the application may use only the intersection of the old and new schemas. Additive changes — a new nullable column, a new table, a new index — are RSU-friendly because old-schema writes apply cleanly on new-schema nodes and vice versa. Renames, drops, and type changes are not; run them as a multi-phase add-copy-swap sequence spread across separate windows.
Reads are the lesser trap: a connection landing on an upgraded node sees the new schema, one landing elsewhere sees the old, so any code that introspects columns at runtime must tolerate both answers. Keep the window short — run the nodes back to back, not one per day — and freeze deployments that touch the table until every node is back on TOI with the new schema in place.
Can you use gh-ost or pt-online-schema-change on Galera?
You can physically run them, and I keep advising teams not to. Both tools were designed around asynchronous replication: they assume DDL is a local event that flows downstream through the binlog, they throttle on measured replica lag, and they cut over with an atomic RENAME. On Galera every one of those assumptions inverts. Their internal DDL — creating the ghost table, attaching triggers, the final rename — replicates under TOI and blocks the cluster exactly like the ALTER you were trying to avoid. Trigger-based change capture doubles the certification load on the hot table. And the lag signal both tools throttle on barely exists in synchronous replication, so the safety valve they are built around measures nothing meaningful.
The native toolbox has absorbed most of their use cases anyway: instant ADD COLUMN covers the common case, ALGORITHM=INPLACE with LOCK=NONE covers much of the rest, and RSU covers the big rebuilds. If a legacy runbook insists on pt-osc against a cluster node, rehearse it on a copy first and treat the whole exercise as debt, not as the plan.
What is the decision runbook for cluster DDL?
Mine fits on an index card. Step one: classify the statement. Rehearse on a restored backup with production row counts and record the duration; instant or seconds goes out under TOI at any hour, done. Step two: anything blocking for minutes on a hot table goes RSU, off-peak — and before you start, confirm the change is additive or split it into phases until every phase is. Step three: upgrade the least-busy node first, verify wsrep_local_state_comment returns to Synced and wsrep_local_recv_queue drains to zero, then proceed node by node. Step four: hold application changes until every node reports the new schema, then deploy the code that uses it. Step five: write down what ran and when. The next person debugging a certification failure at 3am will want the DDL timeline in front of them, not in your memory.
Where MonPG fits
The cluster signals that matter during DDL are per node: local state, receive-queue depth, and cluster size, sampled often enough to catch a desync that is not draining. Disclosure, as always in this series: I work on MonPG, which monitors PostgreSQL today and does not monitor MariaDB yet. MariaDB support is coming soon and in active development — see the /mariadb-monitoring page for where it stands — and Galera health is on the shortlist of things it is being built around: node state transitions, queue growth during maintenance windows, and cluster-size drift surfaced before they page you. Until it ships, the status queries above are the toolkit. If PostgreSQL is also in your fleet, that workflow is live today — the PostgreSQL overview has the details, and the blog has more field notes like this one.