State Snapshot Transfer (SST) is used in Percona XtraDB Cluster (PXC) when a new node joins the cluster or to resync a failed node if Incremental State Transfer (IST) is no longer available. SST is triggered automatically but there is no magic: If it is not configured properly, it will not work and new nodes will never be able to join the cluster. Let’s have a look at a few classic issues.
The donor and the joiner communicate on port 4444, and if the port is closed on one side, SST will always fail.
You will see in the error log of the donor that SST is started:
|
1 |
[...]<br>141223 16:08:48 [Note] WSREP: Node 2 (node1) requested state transfer from '*any*'. Selected 0 (node3)(SYNCED) as donor.<br>141223 16:08:48 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 6)<br>141223 16:08:48 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.<br>141223 16:08:48 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.234.101:4444/xtrabackup_sst' --auth 'sstuser:s3cret' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '04c085a1-89ca-11e4-b1b6-6b692803109b:6''<br>[...]<br> |
But then nothing happens, and some time later you will see a bunch of errors:
|
1 |
[...]<br>2014/12/23 16:09:52 socat[2965] E connect(3, AF=2 192.168.234.101:4444, 16): Connection timed out<br>WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 0 1 (20141223 16:09:52.057)<br>WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141223 16:09:52.064)<br>WSREP_SST: [INFO] Cleaning up temporary directories (20141223 16:09:52.068)<br>141223 16:09:52 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.234.101:4444/xtrabackup_sst' --auth 'sstuser:s3cret' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '04c085a1-89ca-11e4-b1b6-6b692803109b:6'<br>[...]<br> |
On the joiner side, you will see a similar sequence: SST is started, then hangs and is finally aborted:
|
1 |
[...]<br>141223 16:08:48 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 6) <br>141223 16:08:48 [Note] WSREP: Requesting state transfer: success, donor: 0<br>141223 16:08:49 [Note] WSREP: (f9560d0d, 'tcp://0.0.0.0:4567') turning message relay requesting off<br>141223 16:09:52 [Warning] WSREP: 0 (node3): State transfer to 2 (node1) failed: -32 (Broken pipe)<br>141223 16:09:52 [ERROR] WSREP: gcs/src/gcs_group.cpp:long int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():717: Will never receive state. Need to abort.<br> |
The solution is of course to make sure that the ports are open on both sides.
Sometimes you will see an error like this on the donor:
|
1 |
141223 21:03:15 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.234.102:4444/xtrabackup_sst' --auth 'sstuser:s3cretzzz' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid 'e63f38f2-8ae6-11e4-a383-46557c71f368:0''<br>[...]<br>WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (20141223 21:03:26.973)<br> |
And if you look at innobackup.backup.log:
|
1 |
41223 21:03:26 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'sstuser' (using password: YES).<br>innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 2995<br> main::mysql_connect('abort_on_error', 1) called at /usr//bin/innobackupex line 1530<br>innobackupex: Error: Failed to connect to MySQL server: DBI connect(';mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock','sstuser',...) failed: Access denied for user 'sstuser'@'localhost' (using password: YES) at /usr//bin/innobackupex line 2979<br> |
What happened?
The default SST method is xtrabackup-v2 and for it to work, you need to specify a username/password in the my.cnf file:
|
1 |
[mysqld]<br>wsrep_sst_auth=sstuser:s3cret<br> |
And you also need to create the corresponding MySQL user:
|
1 |
mysql> GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost' IDENTIFIED BY 's3cret';<br> |
So you should check that the user has been correctly created in MySQL and that wsrep_sst_auth is correctly set.
Here is another set of errors you may see in the error log of the donor:
|
1 |
141223 21:14:27 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error)<br> at gcomm/src/gcomm/datagram.hpp:unserialize():101<br>141223 21:14:30 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error)<br> at gcomm/src/gcomm/datagram.hpp:unserialize():101<br>141223 21:14:33 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error)<br> at gcomm/src/gcomm/datagram.hpp:unserialize():101<br> |
Here the issue is that you try to connect a node using Galera 2.x and a node running Galera 3.x. This can happen if you try to use a PXC 5.5 node and a PXC 5.6 node.
The right solution is probably to understand why you ended up with such inconsistent versions and make sure all nodes are using the same Percona XtraDB Cluster version and Galera version.
But if you know what you are doing, you can also instruct the node using Galera 3.x that it will communicate with Galera 2.x nodes by specifying in the my.cnf file:
|
1 |
[mysqld]<br>wsrep_provider_options="socket.checksum=1"<br> |
SST errors can have multiple reasons for occurring, and the best way to diagnose the issue is to have a look at the error log of the donor and the joiner. Galera is in general quite verbose so you can follow the progress of SST on both nodes and see where it fails. Then it is mostly about being able to interpret the error messages.
Resources
RELATED POSTS
