When doing migrations or failovers in MySQL, there is usually a need to do a topology change and repoint replica servers to obtain replication data from a different server.
For example, given servers {A, B, and C} and the following topology:
If you need to repoint C to be a replica of B, i.e:
You can follow the next steps:
Note: log_replica_updates should be enabled on the soon-to-be primary as it is a prerequisite for chain replication.
Note: It is assumed that both replicas only stream from Server A and there are no conflicting replication filters in place that might break replication later on.
|
1 |
1) Stop B and C<br><br>STOP REPLICA;<br><br>2) If replicas are multi-threaded, correct MTS gaps and make them single-threaded until all changes are applied. To do so, execute the following commands on BOTH nodes:<br><br>START REPLICA UNTIL SQL_AFTER_MTS_GAPS;<br>SHOW REPLICA STATUSG -- repeat this until you see "Replica_SQL_Running: No"<br>STOP REPLICA;<br>SELECT @@global.replica_parallel_workers; -- take note to restore later<br>SET GLOBAL replica_parallel_workers=0; -- disable MTS during the operations<br><br>3) Then check which is the node that is more up to date by looking at Relay_Source_Log_File and Exec_Source_Log_Pos. Run on BOTH nodes:<br><br>SHOW REPLICA STATUSG<br># Take note of Relay_Source_Log_File/Exec_Source_Log_Pos from the most up to date node.<br><br>4) Sync replicas with UNTIL. Run on the most delayed node with above outputs:<br><br>START REPLICA UNTIL SOURCE_LOG_FILE='<Relay_Source_Log_File>', SOURCE_LOG_POS=<Exec_Source_Log_Pos>;<br>SHOW REPLICA STATUSG -- repeat this until you see "Replica_SQL_Running: No"<br><br>5) If followed above steps, at this point both replicas should have the exact same data set and should be in sync at the same point in time.<br># Double check that both replicas are stopped and with the same coords as doing topology changes while replication is ongoing and with diffs coords can cause inconsistencies:<br><br>SHOW REPLICA STATUSG<br># Replica_IO_Running must be "NO" in both replicas<br># Replica_SQL_Running must be "NO" in both replicas<br># Relay_Source_Log_File must match in both replicas<br># Exec_Source_Log_Pos must match in both replicas<br><br>6) Get current coordinates from B (new intermediate primary). Execute on B:<br><br>SHOW MASTER STATUS G<br># Take note of File and Position<br><br>7) Repoint C to B. Execute on C with coords from previous step:<br><br>CHANGE REPLICATION SOURCE TO SOURCE_HOST='<ip-address-of-B>', SOURCE_LOG_FILE='<File>', SOURCE_LOG_POS='<Position>';<br><br>8) If you had disabled MTS, you should re-enable here for both B and C;<br><br>SET GLOBAL replica_parallel_workers=X; -- see output of step 2 for correct value<br><br>9) Restart replication normally. Run on both nodes:<br><br>START REPLICA; |
|
1 |
1) Stop B and C:<br>STOP REPLICA;<br><br>2) If replicas are multi-threaded, correct MTS gaps and make them single-threaded until all changes are applied. Run on BOTH nodes:<br><br>SHOW REPLICA STATUSG -- repeat this until you see "Replica_SQL_Running: No"<br>STOP REPLICA;<br>SELECT @@global.replica_parallel_workers; -- take note to restore later<br>SET GLOBAL replica_parallel_workers=0; -- disable MTS during the operations<br><br>3) Then check which is the node that is more up to date by looking at sequence numbers in Executed_Gtid_Set. Run on BOTH nodes:<br><br>SHOW REPLICA STATUSG<br># Take note of Executed_Gtid_Set with the largest sequence number. If there is a mismatch in the gtid sets it means there were either local writes or writes coming from some other server. In that case you should check data consistency between the servers, for example with pt-table-checksum . Then you need to fix gtid differences by either restoring the replica from scratch or fix errant transactions as explained on this other blogpost<br><br>4) Bring up all nodes to the same point in time. Run on node with smallest GTID sequence number;<br><br>START REPLICA UNTIL SQL_AFTER_GTIDS='<Executed_Gtid_Set>';<br>SHOW REPLICA STATUSG -- repeat this until you see "Replica_SQL_Running: No"<br><br>5) If followed above steps, at this point both replicas should have the exact same data set and should be in sync at the same point in time.<br># Double check that both replicas are stopped and with the same coords as doing topology changes while replication is ongoing and with diffs coords can cause inconsistencies:<br><br>SHOW REPLICA STATUSG<br># Replica_IO_Running must be "NO" in both replicas<br># Replica_SQL_Running must be "NO" in both replicas<br># Executed_Gtid_Set must match in both replicas<br><br>6) Now both replicas have identical data, so you can re-point C to replicate from B. Run on C:<br><br>CHANGE REPLICATION SOURCE TO SOURCE_HOST='<ip-address-of-B>'<br><br>7) If you had disabled MTS, you should re-enable here for both B and C;<br><br>SET GLOBAL replica_parallel_workers=X; -- see output of step 2 for correct value<br><br>8) Restart replication normally. Run on both nodes<br><br>START REPLICA; |
|
1 |
1) Stop replication on B and make sure B is not receiving any write activity:<br><br>STOP REPLICA;<br><br>2) Check current binary log position on B:<br><br>SHOW MASTER STATUS G<br><br>3) On C check replication until C does catch up with B. On C:<br><br>SHOW REPLICA STATUS G<br><br># For C to have catch up with B, the following conditions should be met:<br># "File" from B on step 2) should match Relay_Source_Log_File from 3)<br># "Position" from B on step2) should match Exec_Source_Log_Pos from 3)<br><br># After catchup, both servers will be in sync with the same data set.<br><br>4) Check current replication coords from B:<br><br>SHOW REPLICA STATUS G<br># Write down Relay_Source_Log_File and Exec_Source_Log_Pos from B, as we will be using this coords on C<br><br>5) Re point C to replicate from A. File and positions used should be the ones taken from B on last step: <br><br>CHANGE REPLICATION SOURCE TO SOURCE_HOST='<ip-address-of-A>', SOURCE_LOG_FILE='<File>', SOURCE_LOG_POS='<Position>'<br><br>6) Restart replication normally. Run on both nodes:<br><br>START REPLICA; |
|
1 |
1) Stop replication on B and make sure B is not receiving any write activity:<br><br>STOP REPLICA;<br><br>2) Check current binary log position on B:<br><br>SHOW MASTER STATUS G<br><br>3) On C check replication until C does catch up with B. On C:<br><br>SHOW REPLICA STATUS G<br><br># For C to have catch up with B, the following conditions should be met:<br># Executed_Gtid_Set from B step 2) should match Executed_Gtid_Set from 3)<br># After catchup, both servers will be in sync with the same data set.<br><br>4) Re point C to replicate from A:<br><br>CHANGE REPLICATION SOURCE TO SOURCE_HOST='<ip-address-of-A>'<br><br>5) Restart replication normally. Run on both nodes<br><br>START REPLICA;<br> |
Doing topology changes might seem hard at first, but with the above procedure, it should be easy and error-free! If you do not want to do the manual approach, then you can consider using tools like Orchestrator which allows for automatic failover and promotions.
Percona Distribution for MySQL is the most complete, stable, scalable, and secure, open-source MySQL solution available, delivering enterprise-grade database environments for your most critical business applications… and it’s free to use!
Resources
RELATED POSTS
