MySQL supports replicating to a slave that is one release higher. This allows us to easily upgrade our MySQL setup to a new version, by promoting the slave and pointing the application to it. However, though unsupported, there are times when the MySQL version of slave deployed is one release lower. In this scenario, if your application has been performing much better on an older version of MySQL, you would like to have a convenient option to downgrade. You can simply promote the slave to get the old performance back.
The MySQL manual says that ROW based replication can be used to replicate to a lower version, provided that no DDLs replicated are incompatible with the slave. One such incompatible command is ALTER USER which is a new feature in MySQL 5.7 and not available on 5.6. :
|
1 |
ALTER USER 'testuser'@'localhost' IDENTIFIED BY 'testuser'; |
Executing that command would break replication. Here is an example of a broken slave in non-GTID replication:
|
1 |
*************************** 1. row ***************************<br> Slave_IO_State: Waiting for master to send event<br> Master_Host: 127.0.0.1<br> Master_User: repl<br> Master_Port: 5723<br> Connect_Retry: 60<br> Master_Log_File: mysql-bin.000002<br> Read_Master_Log_Pos: 36915649<br> Relay_Log_File: mysql_sandbox5641-relay-bin.000006<br> Relay_Log_Pos: 36174552<br> Relay_Master_Log_File: mysql-bin.000002<br> Slave_IO_Running: Yes<br> Slave_SQL_Running: No<br>*** redacted ***<br> Last_Errno: 1064<br> Last_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A784' at line 1' on query. Default database: ''. Query: 'ALTER USER 'testuser'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A7846BCB8''<br> Skip_Counter: 0<br> Exec_Master_Log_Pos: 36174373<br> Relay_Log_Space: 36916179<br> Until_Condition: None<br> Until_Log_File: <br> Until_Log_Pos: 0<br> Master_SSL_Allowed: No<br>*** redacted ***<br> Seconds_Behind_Master: NULL<br>Master_SSL_Verify_Server_Cert: No<br> Last_IO_Errno: 0<br> Last_IO_Error: <br> Last_SQL_Errno: 1064<br> Last_SQL_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A784' at line 1' on query. Default database: ''. Query: 'ALTER USER 'testuser'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A7846BCB8''<br> Replicate_Ignore_Server_Ids: <br> Master_Server_Id: 1<br> Master_UUID: 00005723-0000-0000-0000-000000005723<br>*** redacted ***<br> Last_IO_Error_Timestamp: <br> Last_SQL_Error_Timestamp: 180918 22:03:40<br>*** redacted ***<br> Auto_Position: 0<br>1 row in set (0.00 sec) |
Skipping the statement does not resume replication:
|
1 |
mysql> STOP SLAVE;<br>Query OK, 0 rows affected (0.02 sec)<br>mysql> SET GLOBAL sql_slave_skip_counter=1;<br>Query OK, 0 rows affected (0.00 sec)<br>mysql> START SLAVE;<br>Query OK, 0 rows affected (0.01 sec)<br>mysql> SHOW SLAVE STATUSG |
When you check slave status, replication still isn’t fixed. To fix it, you must manually skip to the next binary log position. The current binary log (Relay_Master_Log_File) and position (Exec_Master_Log_Pos) executed are mysql-bin.000002 and 36174373 respectively. We can use mysqlbinlog on the master to determine the next position:
|
1 |
mysqlbinlog -v --base64-output=DECODE-ROWS --start-position=36174373 /ssd/sandboxes/msb_5_7_23/data/mysql-bin.000002 | head -n 30<br>/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;<br>/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;<br>DELIMITER /*!*/;<br># at 36174373<br>#180918 22:03:40 server id 1 end_log_pos 36174438 CRC32 0xc7e1e553 Anonymous_GTID last_committed=19273 sequence_number=19277 rbr_only=no<br>SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;<br># at 36174438<br>#180918 22:03:40 server id 1 end_log_pos 36174621 CRC32 0x2e5bb235 Query thread_id=563 exec_time=0 error_code=0<br>SET TIMESTAMP=1537279420/*!*/;<br>SET @@session.pseudo_thread_id=563/*!*/;<br>SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;<br>SET @@session.sql_mode=1436549152/*!*/;<br>SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;<br>/*!C latin1 *//*!*/;<br>SET @@session.character_set_client=8,@@session.collation_connection=8,@@session.collation_server=8/*!*/;<br>SET @@session.lc_time_names=0/*!*/;<br>SET @@session.collation_database=DEFAULT/*!*/;<br>ALTER USER 'testuser'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A7846BCB8'<br>/*!*/;<br># at 36174621<br>#180918 22:03:40 server id 1 end_log_pos 36174686 CRC32 0x86756b3f Anonymous_GTID last_committed=19275 sequence_number=19278 rbr_only=yes<br>/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;<br>SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;<br># at 36174686<br>#180918 22:03:40 server id 1 end_log_pos 36174760 CRC32 0x30e663f9 Query thread_id=529 exec_time=0 error_code=0<br>SET TIMESTAMP=1537279420/*!*/;<br>BEGIN<br>/*!*/;<br># at 36174760<br>#180918 22:03:40 server id 1 end_log_pos 36174819 CRC32 0x48054daf Table_map: `sbtest`.`sbtest1` mapped to number 226 |
Based on the output above, the next binary log position is 36174621. To fix the slave, run:
|
1 |
STOP SLAVE;<br>CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=36174621;<br>START SLAVE; |
Verify if the slave threads are now running by executing SHOW SLAVE STATUSG
|
1 |
Slave_IO_State: Waiting for master to send event<br> Master_Host: 127.0.0.1<br> Master_User: repl<br> Master_Port: 5723<br> Connect_Retry: 60<br> Master_Log_File: mysql-bin.000002<br> Read_Master_Log_Pos: 306841423<br> Relay_Log_File: mysql_sandbox5641-relay-bin.000002<br> Relay_Log_Pos: 190785290<br> Relay_Master_Log_File: mysql-bin.000002<br> Slave_IO_Running: Yes<br> Slave_SQL_Running: Yes<br>*** redacted *** <br> Last_Errno: 0<br> Last_Error: <br> Skip_Counter: 0<br> Exec_Master_Log_Pos: 226959625<br> Relay_Log_Space: 270667273<br> Until_Condition: None<br> Until_Log_File: <br> Until_Log_Pos: 0<br> Master_SSL_Allowed: No<br>*** redacted ***<br> Seconds_Behind_Master: 383<br>Master_SSL_Verify_Server_Cert: No<br> Last_IO_Errno: 0<br> Last_IO_Error: <br> Last_SQL_Errno: 0<br> Last_SQL_Error: <br> Replicate_Ignore_Server_Ids: <br> Master_Server_Id: 1<br> Master_UUID: 00005723-0000-0000-0000-000000005723<br> Master_Info_File: /ssd/sandboxes/msb_5_6_41/data/master.info<br> SQL_Delay: 0<br> SQL_Remaining_Delay: NULL<br> Slave_SQL_Running_State: Opening tables<br> Master_Retry_Count: 86400<br>*** redacted ***<br> Auto_Position: 0 |
To make the slave consistent with the master, execute the compatible query on the slave.
|
1 |
SET SESSION sql_log_bin = 0;<br>GRANT USAGE ON *.* TO 'testuser'@'localhost' IDENTIFIED BY 'testuser'; |
Done.
For GTID replication, in addition to injecting an empty transaction for the offending statement, you’ll need skip it by using the non-GTID solution provided above. Once running, flip it back to GTID.
Here’s an example of a broken GTID slave:
|
1 |
mysql> SHOW SLAVE STATUSG<br>*************************** 1. row ***************************<br> Slave_IO_State: Waiting for master to send event<br> Master_Host: 127.0.0.1<br> Master_User: repl<br> Master_Port: 5723<br> Connect_Retry: 60<br> Master_Log_File: mysql-bin.000003<br> Read_Master_Log_Pos: 14364967<br> Relay_Log_File: mysql_sandbox5641-relay-bin.000002<br> Relay_Log_Pos: 8630318<br> Relay_Master_Log_File: mysql-bin.000003<br> Slave_IO_Running: Yes<br> Slave_SQL_Running: No<br>*** redacted ***<br> Last_Errno: 1064<br> Last_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A784' at line 1' on query. Default database: ''. Query: 'ALTER USER 'testuser'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A7846BCB8''<br> Skip_Counter: 0<br> Exec_Master_Log_Pos: 12468343<br> Relay_Log_Space: 10527158<br> Until_Condition: None<br> Until_Log_File: <br> Until_Log_Pos: 0<br> Master_SSL_Allowed: No<br>*** redacted ***<br> Seconds_Behind_Master: NULL<br>Master_SSL_Verify_Server_Cert: No<br> Last_IO_Errno: 0<br> Last_IO_Error: <br> Last_SQL_Errno: 1064<br> Last_SQL_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A784' at line 1' on query. Default database: ''. Query: 'ALTER USER 'testuser'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A7846BCB8''<br> Replicate_Ignore_Server_Ids: <br> Master_Server_Id: 1<br> Master_UUID: 00005723-0000-0000-0000-000000005723<br> Master_Info_File: /ssd/sandboxes/msb_5_6_41/data/master.info<br> SQL_Delay: 0<br> SQL_Remaining_Delay: NULL<br> Slave_SQL_Running_State: <br> Master_Retry_Count: 86400<br> Master_Bind: <br> Last_IO_Error_Timestamp: <br> Last_SQL_Error_Timestamp: 180918 22:32:28<br> Master_SSL_Crl: <br> Master_SSL_Crlpath: <br> Retrieved_Gtid_Set: 00005723-0000-0000-0000-000000005723:2280-8530<br> Executed_Gtid_Set: 00005723-0000-0000-0000-000000005723:1-7403<br> Auto_Position: 1<br>1 row in set (0.00 sec)<br><br>mysql> SHOW GLOBAL VARIABLES LIKE 'gtid_executed';<br>+---------------+---------------------------------------------+<br>| Variable_name | Value |<br>+---------------+---------------------------------------------+<br>| gtid_executed | 00005723-0000-0000-0000-000000005723:1-7403 |<br>+---------------+---------------------------------------------+<br>1 row in set (0.00 sec) |
Since the last position executed is 7403, so you’ll need to create an empty transaction for the offending sequence 7404.
|
1 |
STOP SLAVE;<br>SET GTID_NEXT='00005723-0000-0000-0000-000000005723:7404';<br>BEGIN;<br>COMMIT;<br>SET GTID_NEXT=AUTOMATIC;<br>START SLAVE; |
Note: If you have MTS enabled, you can also get the offending GTID coordinates from Last_SQL_Error of SHOW SLAVE STATUSG
The next step is to find the next binary log position. The current binary log(Relay_Master_Log_File) and position(Exec_Master_Log_Pos) executed are mysql-bin.000003 and 12468343 respectively. We can again use mysqlbinlog on the master to determine the next position:
|
1 |
mysqlbinlog -v --base64-output=DECODE-ROWS --start-position=12468343 /ssd/sandboxes/msb_5_7_23/data/mysql-bin.000003 | head -n 30<br><br>/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;<br>/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;<br>DELIMITER /*!*/;<br># at 12468343<br>#180918 22:32:19 server id 1 end_log_pos 12468408 CRC32 0x259ee085 GTID last_committed=7400 sequence_number=7404 rbr_only=no<br>SET @@SESSION.GTID_NEXT= '00005723-0000-0000-0000-000000005723:7404'/*!*/;<br># at 12468408<br>#180918 22:32:19 server id 1 end_log_pos 12468591 CRC32 0xb349ad80 Query thread_id=142 exec_time=0 error_code=0<br>SET TIMESTAMP=1537281139/*!*/;<br>SET @@session.pseudo_thread_id=142/*!*/;<br>SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;<br>SET @@session.sql_mode=1436549152/*!*/;<br>SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;<br>/*!C latin1 *//*!*/;<br>SET @@session.character_set_client=8,@@session.collation_connection=8,@@session.collation_server=8/*!*/;<br>SET @@session.lc_time_names=0/*!*/;<br>SET @@session.collation_database=DEFAULT/*!*/;<br>ALTER USER 'testuser'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*3A2EB9C80F7239A4DE3933AE266DB76A7846BCB8'<br>/*!*/;<br># at 12468591<br>#180918 22:32:19 server id 1 end_log_pos 12468656 CRC32 0xb2019f3f GTID last_committed=7400 sequence_number=7405 rbr_only=yes<br>/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;<br>SET @@SESSION.GTID_NEXT= '00005723-0000-0000-0000-000000005723:7405'/*!*/;<br># at 12468656<br>#180918 22:32:19 server id 1 end_log_pos 12468730 CRC32 0x76b5ea6c Query thread_id=97 exec_time=0 error_code=0<br>SET TIMESTAMP=1537281139/*!*/;<br>BEGIN<br>/*!*/;<br># at 12468730<br>#180918 22:32:19 server id 1 end_log_pos 12468789 CRC32 0x48f0ba6d Table_map: `sbtest`.`sbtest8` mapped to number 115 |
The next binary log position is 36174621. To fix the slave, run:
|
1 |
STOP SLAVE;<br>CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=12468591, MASTER_AUTO_POSITION=0;<br>START SLAVE; |
Notice that I added MASTER_AUTO_POSITION=0 above to disable GTID replication for now. You can run SHOW SLAVE STATUSG to determine that MySQL is running fine:
|
1 |
mysql> SHOW SLAVE STATUSG<br>*************************** 1. row ***************************<br> Slave_IO_State: Waiting for master to send event<br> Master_Host: 127.0.0.1<br> Master_User: repl<br> Master_Port: 5723<br> Connect_Retry: 60<br> Master_Log_File: mysql-bin.000003<br> Read_Master_Log_Pos: 446194575<br> Relay_Log_File: mysql_sandbox5641-relay-bin.000002<br> Relay_Log_Pos: 12704248<br> Relay_Master_Log_File: mysql-bin.000003<br> Slave_IO_Running: Yes<br> Slave_SQL_Running: Yes<br>*** redacted ***<br> Last_Errno: 0<br> Last_Error: <br> Skip_Counter: 0<br> Exec_Master_Log_Pos: 25172522<br> Relay_Log_Space: 433726939<br> Until_Condition: None<br> Until_Log_File: <br> Until_Log_Pos: 0<br> Master_SSL_Allowed: No<br>*** redacted ***<br> Seconds_Behind_Master: 2018<br>Master_SSL_Verify_Server_Cert: No<br> Last_IO_Errno: 0<br> Last_IO_Error: <br> Last_SQL_Errno: 0<br> Last_SQL_Error: <br> Replicate_Ignore_Server_Ids: <br> Master_Server_Id: 1<br> Master_UUID: 00005723-0000-0000-0000-000000005723<br> Master_Info_File: /ssd/sandboxes/msb_5_6_41/data/master.info<br> SQL_Delay: 0<br> SQL_Remaining_Delay: NULL<br> Slave_SQL_Running_State: Reading event from the relay log<br> Master_Retry_Count: 86400<br>*** redacted ***<br> Retrieved_Gtid_Set: 00005723-0000-0000-0000-000000005723:7405-264930<br> Executed_Gtid_Set: 00005723-0000-0000-0000-000000005723:1-14947<br> Auto_Position: 0 |
Since it’s running fine you can now revert back to GTID replication:
|
1 |
STOP SLAVE;<br>CHANGE MASTER TO MASTER_AUTO_POSITION=1;<br>START SLAVE; |
Finally, to make the slave consistent with the master, execute the compatible query on the slave.
|
1 |
SET SESSION sql_log_bin = 0;<br>GRANT USAGE ON *.* TO 'testuser'@'localhost' IDENTIFIED BY 'testuser'; |
In this article, I’ve shared how to fix replication when it breaks due to an incompatible command being replicated to the slave. In fact, I’ve only identified ALTER USER as an incompatible command for 5.6. If there are other incompatible commands, please share them in the comment section. Thanks in advance.
Update:
I filed a bug at https://bugs.mysql.com/bug.php?id=92629 to verify if the errors I’ve encountered here is a bug or undocumented behavior.
Resources
RELATED POSTS
