MySQL10 min read

gh-ost vs pt-osc: Field Notes on Online Schema Change in MySQL

Triggers or binlog streaming? Field notes from running both gh-ost and pt-online-schema-change in production: cut-over risk, throttling behavior, foreign key walls, and when you need neither.

Every team that operates large MySQL tables eventually hits an ALTER that native online DDL cannot do safely: a data type change on a 500 GB table, a character set conversion, a primary key redesign. The operation is COPY or a rebuilding INPLACE, the replica will lag for the entire duration, and the deploy window is not long enough. At that point you reach for an external online schema change tool, and the choice is usually between pt-online-schema-change from Percona Toolkit and gh-ost from GitHub.

I have run both in production. Both work. They fail differently, throttle differently, and cut over differently, and those differences matter more than any feature checklist. These are the notes I wish someone had handed me before my first 400 GB migration.

The shape both tools share

Strip away the details and both tools do the same dance. Create a ghost table with the new schema. Copy existing rows from the original table in chunks. Capture ongoing writes to the original table and apply them to the ghost. When the ghost has caught up, atomically swap the names so the ghost becomes the real table. Drop or keep the old table.

Everything interesting lives in step three: how do you capture ongoing writes? That single design decision drives almost every operational difference between the two tools.

Both approaches also share two unglamorous prerequisites people forget to budget for. You need free disk for a second full copy of the table plus its indexes for the duration of the migration, and you need a plan for the old table afterward — dropping a multi-hundred-gigabyte table has its own I/O cost, which is why both tools default to leaving it behind for you to remove deliberately rather than dropping it at cut-over.

pt-osc: triggers on the production table

pt-online-schema-change creates three triggers on the original table: AFTER INSERT, AFTER UPDATE, and AFTER DELETE. Each production write executes the trigger synchronously, applying the same change to the ghost table inside the same transaction. The copy process then backfills historical rows in chunks.

The synchronous part is the cost. Every write to the table now does double work for the entire duration of the migration, which on a big table can be days. Lock contention on the ghost table becomes contention on your production writes. If the migration dies, the triggers must be cleaned up. And creating the triggers in the first place requires a brief metadata lock that can queue behind long-running transactions, exactly like a native ALTER.

Triggers also interact with existing triggers. Modern pt-osc has a --preserve-triggers option, but the interaction surface is real, and I have seen teams discover trigger-order assumptions only during a migration. The flip side: because writes to the ghost are synchronous, pt-osc never falls behind on change capture. There is no replication-style lag between the original and the ghost; the copy backlog is the only thing that has to catch up.

gh-ost: binlog streaming, no triggers

gh-ost takes the trigger logic out of the write path entirely. It connects as a replica, reads row-based binlog events, and applies changes to the ghost table asynchronously. Production writes pay nothing. The tool can pause itself completely: stop copying, stop applying, wait, resume, all without touching production write latency.

The requirements follow from the design. You need binlog_format=ROW available (gh-ost can read from a replica that has row-format binlogs even if the primary uses statement format, though ROW everywhere is the sane modern default) and log_slave_updates if reading from a replica. Verify before you start:

SELECT @@binlog_format   AS binlog_format,
       @@log_slave_updates AS log_slave_updates,
       @@binlog_row_image  AS binlog_row_image;

The asynchronous design has a cost symmetrical to pt-osc's: gh-ost can fall behind. Under heavy write bursts, the binlog applier lags, and the migration simply takes longer. That is usually the right trade. A migration that takes an extra day is an inconvenience; production writes that got 30% slower for a week is an incident.

Cut-over: the riskiest part of both tools

The rename swap at the end is where I have seen every serious online-schema-change incident happen. pt-osc uses an atomic RENAME TABLE, which is clean, but the rename must acquire metadata locks on both tables, and a long-running query or transaction can stall it while everything else queues behind. gh-ost implements a more elaborate cut-over algorithm that locks the original table, confirms the applier has drained the binlog backlog, then performs the swap; it deliberately aborts and retries if the lock cannot be acquired within a timeout.

Practical guidance that has saved me repeatedly: cut over during the lowest-write period you can find, hunt down long-running transactions first, and rehearse the abort path. gh-ost's interactive commands let you postpone cut-over until a human confirms (--postpone-cut-over-flag-file), which turns the scariest moment of a week-long migration into a deliberate, staffed action instead of a 3 a.m. surprise.

Throttling: how each tool yields under load

Both tools chunk their copy phase and throttle between chunks, but the control models differ. pt-osc watches --max-load and --critical-load thresholds (Threads_running by default) and replica lag via --max-lag, pausing the copy when limits are exceeded. gh-ost throttles on replica lag, on arbitrary user-defined queries, on a flag file you can touch to pause instantly, and via a network-reachable interactive interface where you can change throttle parameters mid-flight.

That last capability is the one I now consider essential for multi-day migrations. Requirements change mid-migration: a marketing campaign lands, a batch job needs headroom, a replica gets rebuilt. With gh-ost you adjust without restarting; with pt-osc a mistake in the initial thresholds generally means killing and restarting the migration, and the copy starts over.

The foreign key wall

Foreign keys are where the decision often gets made for you. gh-ost does not support tables with foreign key constraints, in either direction: no FKs on the table being migrated, and no FKs referencing it. The rename swap would leave child tables pointing at the wrong parent, so the tool refuses.

pt-osc supports foreign keys through --alter-foreign-keys-method, with two real options. rebuild_constraints re-creates the child-table constraints, which is safe but rebuilds the children, and drop_swap drops the original table before renaming the ghost into place, which is faster but leaves a brief window where the table does not exist and is not atomic. Neither option is free; both need rehearsal. Audit the constraint graph before choosing a tool:

SELECT rc.CONSTRAINT_NAME,
       rc.TABLE_NAME       AS child_table,
       rc.REFERENCED_TABLE_NAME AS parent_table
FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS rc
WHERE rc.CONSTRAINT_SCHEMA = 'shop'
  AND (rc.TABLE_NAME = 'orders'
       OR rc.REFERENCED_TABLE_NAME = 'orders');

If this query returns rows for your target table, gh-ost is out, and your pt-osc plan needs an explicit foreign-key strategy reviewed by someone who has done it before.

When native online DDL is enough

The honest answer for MySQL 8.0/8.4 is: more often than the folklore suggests. Instant operations (adding and dropping columns, changing defaults) need no tool at all. Many INPLACE operations, including adding secondary indexes, permit concurrent DML and are fine on moderately sized tables if you can tolerate the replica applying the DDL as one long statement. The external tools earn their complexity when the operation forces a COPY, when the table is large enough that replica lag from a native INPLACE is unacceptable, or when you need the ability to pause and resume a multi-day change. I went deeper on the native algorithm matrix in MySQL online DDL vs PostgreSQL DDL locks.

One more boundary worth knowing: neither tool helps with partitioned-table management operations, and partitioning brings its own constraints — see MySQL partitioning vs PostgreSQL partitioning for that comparison.

Where MonPG fits

MonPG is a PostgreSQL monitoring platform today, and MySQL support is coming soon — not shipped, and I will not pretend otherwise. The reason we write about schema-change tooling now is that migration observability is a core requirement for the MySQL monitoring (coming soon) product: replica lag during copy phases, metadata lock queues at cut-over, throttle state, and the write-amplification cost of triggers are all things an operator should see on one screen instead of four terminals. If you also run PostgreSQL, MonPG already does this class of work there — you can start there while the MySQL side is built.