Emergency

2nd node unable to join cluster

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • 2nd node unable to join cluster

    I'm setting up a 3 node cluster . The first node starts up fine with bootstrap , the second node fails to join the cluster

    [root@mysql-test02 mysql]# /etc/init.d/mysql start
    MySQL (Percona XtraDB Cluster) is not running, but lock fil[FAILED]lock/subsys/mysql) exists
    Starting MySQL (Percona XtraDB Cluster)..State transfer in progress, setting sleep higher
    ..The server quit without updating PID file (/var/lib/mysql[FAILED]est02.bo3.e-dialog.com.pid).
    MySQL (Percona XtraDB Cluster) server startup failed! [FAILED]
    [root@mysql-test02 mysql]#


    Here is the relevant information from the node 2 error log -

    2014-12-03 16:02:09 51655 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '10.100.7.28' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '51655' '' '
    WSREP_SST: [INFO] Streaming with tar (20141203 16:02:10.030)
    WSREP_SST: [INFO] Using socat as streamer (20141203 16:02:10.031)
    WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner 2>>/tmp/sstl.progress | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141203 16:02:10.041)
    2014-12-03 16:02:10 51655 [Note] WSREP: Prepared SST request: xtrabackup|10.100.7.28:4444/xtrabackup_sst
    2014-12-03 16:02:10 51655 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2014-12-03 16:02:10 51655 [Note] WSREP: REPL Protocols: 6 (3, 2)
    2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed.
    2014-12-03 16:02:10 51655 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
    2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed.
    2014-12-03 16:02:10 51655 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (60c7c2b6-7b1d-11e4-9d8f-3723cbbcdb67): 1 (Operation not permitted)
    at galera/src/replicator_str.cpprepare_for_IST():456. IST will be unavailable.
    2014-12-03 16:02:10 51655 [Note] WSREP: Member 1.0 (mysql-test02.bo3.e-dialog.com) requested state transfer from '*any*'. Selected 0.0 (mysql-test01.bo3.e-dialog.com)(SYNCED) as donor.
    2014-12-03 16:02:10 51655 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3)
    2014-12-03 16:02:10 51655 [Note] WSREP: Requesting state transfer: success, donor: 0
    2014-12-03 16:02:12 51655 [Note] WSREP: (a4faecfe, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2014-12-03 16:02:13 51655 [Note] WSREP: 0.0 (mysql-test01.bo3.e-dialog.com): State transfer to 1.0 (mysql-test02.bo3.e-dialog.com) complete.
    2014-12-03 16:02:13 51655 [Note] WSREP: Member 0.0 (mysql-test01.bo3.e-dialog.com) synced with group.
    WSREP_SST: [ERROR] xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info' (20141203 16:02:13.684)
    WSREP_SST: [INFO] Contents of datadir (20141203 16:02:13.685)
    WSREP_SST: [INFO] -rw------- 1 mysql mysql 8820 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.frm
    -rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.MYD
    -rw------- 1 mysql mysql 4096 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.MYI
    -rw------- 1 mysql mysql 9582 Dec 3 13:09 /var/lib/mysql//mysql/db.frm
    -rw------- 1 mysql mysql 880 Dec 3 13:09 /var/lib/mysql//mysql/db.MYD
    -rw------- 1 mysql mysql 5120 Dec 3 13:09 /var/lib/mysql//mysql/db.MYI
    -rw------- 1 mysql mysql 10223 Dec 3 13:09 /var/lib/mysql//mysql/event.frm
    -rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/event.MYD
    .
    .
    .
    -rw-rw---- 1 mysql mysql 13518 Dec 3 13:09 /var/lib/mysql//performance_schema/table_lock_waits_summary_by_table.frm
    -rw-rw---- 1 mysql mysql 9201 Dec 3 13:09 /var/lib/mysql//performance_schema/threads.frm
    -rw-rw---- 1 mysql mysql 8676 Dec 3 13:09 /var/lib/mysql//performance_schema/users.frm
    -rw-rw---- 1 mysql mysql 0 Dec 3 16:02 /var/lib/mysql//test/db.opt (20141203 16:02:13.694)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141203 16:02:13.698)
    WSREP_SST: [INFO] Removing the sst_in_progress file (20141203 16:02:13.699)
    2014-12-03 16:02:13 51655 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '10.100.7.28' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '51655' '' : 32 (Broken pipe)
    2014-12-03 16:02:13 51655 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    2014-12-03 16:02:13 51655 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    2014-12-03 16:02:13 51655 [ERROR] Aborting





    On node 1 the the innobackup.backup.log log looks clean -

    [root@mysql-test01 mysql]# cat innobackup.backup.log

    InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
    and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

    This software is published under
    the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

    Get the latest version of Percona XtraBackup, documentation, and help resources:
    http://www.percona.com/xb/p

    141203 16:02:10 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_s ocket=/var/lib/mysql/mysql.sock' as 'sstuser' (using password: YES).
    141203 16:02:10 innobackupex: Connected to MySQL server
    141203 16:02:10 innobackupex: Executing a version check against the server...
    141203 16:02:10 innobackupex: Done.
    141203 16:02:10 innobackupex: Starting the backup operation

    IMPORTANT: Please check that the backup run completes successfully.
    At the end of a successful backup run innobackupex
    prints "completed OK!".

    innobackupex: Using server version 5.6.21-70.1-56

    innobackupex: Created backup directory /tmp

    141203 16:02:10 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --innodb_log_file_size="50331648" --innodb_data_file_path="ibdata1:12M:autoextend" --tmpdir=/tmp --extra-lsndir='/tmp' --stream=tar
    innobackupex: Waiting for ibbackup (pid=53974) to suspend
    innobackupex: Suspend file '/tmp/xtrabackup_suspended_2'

    xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
    xtrabackup: uses posix_fadvise().
    xtrabackup: cd to /var/lib/mysql
    xtrabackup: open files limit requested 0, set to 131072
    xtrabackup: using the following InnoDB configuration:
    xtrabackup: innodb_data_home_dir = ./
    xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
    xtrabackup: innodb_log_group_home_dir = ./
    xtrabackup: innodb_log_files_in_group = 2
    xtrabackup: innodb_log_file_size = 50331648
    >> log scanned up to (1626819)
    xtrabackup: Generating a list of tablespaces
    [01] Streaming ./ibdata1
    [01] ...done
    [01] Streaming ./mysql/slave_worker_info.ibd
    [01] ...done
    [01] Streaming ./mysql/slave_relay_log_info.ibd
    [01] ...done
    [01] Streaming ./mysql/innodb_table_stats.ibd
    [01] ...done
    [01] Streaming ./mysql/innodb_index_stats.ibd
    [01] ...done
    [01] Streaming ./mysql/slave_master_info.ibd
    [01] ...done
    >> log scanned up to (1626819)
    xtrabackup: Creating suspend file '/tmp/xtrabackup_suspended_2' with pid '53974'

    141203 16:02:12 innobackupex: Continuing after ibbackup has suspended
    141203 16:02:12 innobackupex: Executing LOCK TABLES FOR BACKUP...
    141203 16:02:12 innobackupex: Backup tables lock acquired

    141203 16:02:12 innobackupex: Starting to backup non-InnoDB tables and files
    innobackupex: in subdirectories of '/var/lib/mysql/'
    innobackupex: Backing up files '/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}' (74 files)
    >> log scanned up to (1626819)
    innobackupex: Backing up files '/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}' (53 files)
    141203 16:02:12 innobackupex: Finished backing up non-InnoDB tables and files

    141203 16:02:12 innobackupex: Executing LOCK BINLOG FOR BACKUP...
    141203 16:02:12 innobackupex: Executing FLUSH ENGINE LOGS...
    141203 16:02:12 innobackupex: Waiting for log copying to finish

    xtrabackup: The latest check point (for incremental): '1626819'
    xtrabackup: Stopping log copying thread.
    .>> log scanned up to (1626819)

    xtrabackup: Creating suspend file '/tmp/xtrabackup_log_copied' with pid '53974'
    xtrabackup: Transaction log of lsn (1626819) to (1626819) was copied.
    141203 16:02:13 innobackupex: Executing UNLOCK BINLOG
    141203 16:02:13 innobackupex: Executing UNLOCK TABLES
    141203 16:02:13 innobackupex: All tables unlocked

    innobackupex: Backup created in directory '/tmp'
    141203 16:02:13 innobackupex: Connection to database server closed
    innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
    141203 16:02:13 innobackupex: completed OK!
    [root@mysql-test01 mysql]#

  • #2
    Contents of /etc/my.cnf on node 2 -

    [mysqld]

    datadir=/var/lib/mysql
    user=mysql

    # Path to Galera library
    wsrep_provider=/usr/lib64/libgalera_smm.so

    # Cluster connection URL contains the IPs of node#1, node#2 and node#3
    wsrep_cluster_address=gcomm://10.100.7.27,10.100.7.28,10.100.7.29



    # In order for Galera to work correctly binlog format should be ROW
    binlog_format=ROW

    # MyISAM storage engine has only experimental support
    default_storage_engine=InnoDB

    # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
    innodb_autoinc_lock_mode=2

    # Node #2 address
    wsrep_node_address=10.100.7.28

    # SST method
    wsrep_sst_method=xtrabackup

    # Cluster name
    wsrep_cluster_name=mysql_test_cluster

    # Authentication for SST method
    wsrep_sst_auth="sstuser:s3cret"

    Comment


    • #3
      Any news on this issue? I am struggling with the same problem.

      Comment


      • #4
        No solution yet . I see from your thread that you reverted to 5.6.20 , going to try that .

        Comment


        • #5
          Scratch that . This is what worked -

          commenting out the line "wsrep_sst_method=xtrabackup" in the my.cnf file


          Comment


          • #6
            Looks similar to the answer to this thread:
            http://www.percona.com/forums/questi...in-the-cluster

            The wsrep_sst_method=xtrabackup method is apparently being depricated, so commenting out that line made it use the default (probably the xtrabackup-v2 mentioned in the above thread) which then worked.

            Comment


            • #7
              That's right! I've just tested it, commenting out the option wsrep_sst_method, it has worked finely well. It seems that the variable's default is xtrabackup-v2 as you well said.
              Last edited by wagnerbianchi; 12-30-2014, 06:19 PM.
              WB

              Comment

              Working...
              X