Announcement

Announcement Module
Collapse
No announcement yet.

XtraBackup SST failing when adding 4th node

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • XtraBackup SST failing when adding 4th node

    I currently have a 3 node cluster and was hoping to add a 4th node. The first 3 installs went rather easily though this one is having a hang up. Any help would be greatly appreciated.

    Everything is running cent 6.4 x64

    Current Nodes:
    vm02db1.domain.com
    vm03db1.domain.com <-- Started with /etc/init.d/mysql start --wsrep-cluster-address="gcomm://"
    vm04db1.domain.com

    Trying to add Node:
    vm01db1.domain.com


    my.cnf
    [mysqld]
    datadir=/Array/perconadb/data
    tmpdir=/Array/perconadb/tmp
    user=mysql
    wsrep_provider=/usr/lib64/libgalera_smm.so
    wsrep_slave_threads=8
    wsrep_cluster_address=gcomm://vm01db1.domain.com,vm02db1.domain.com,vm03db1.doma in.com,vm04db1.domain.com
    binlog_format=ROW
    default_storage_engine=InnoDB
    innodb_locks_unsafe_for_binlog=1
    innodb_autoinc_lock_mode=2
    wsrep_node_address=vm01db1.domain.com
    wsrep_sst_method=xtrabackup
    wsrep_sst_auth=sstuser:mysupersecretpasswordhere <- Thats not really the password i use

    From the DONOR (vm03db1) - This is the sync trying to kick off
    130613 10:02:03 [Note] WSREP: Flow-control interval: [32, 32]
    130613 10:02:03 [Note] WSREP: New cluster view: global state: 2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388, view# 24: Primary, number of nodes: 4, my index: 0, protocol version 2
    130613 10:02:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    130613 10:02:04 [Note] WSREP: Assign initial position for certification: 1191388, protocol version: 2
    130613 10:02:05 [Note] WSREP: Node 3 (vm01db1) requested state transfer from '*any*'. Selected 0 (vm03db1)(SYNCED) as donor.
    130613 10:02:05 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 1191388)
    130613 10:02:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

    From the DONOR (vm03db1) - This is the error I see
    130613 10:02:05 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'donor' --address 'vm01db1.domain.com:4444/xtrabackup_sst' --auth 'sstuser:mysupersecretpasswordhere' --socket '/var/lib/mysql/mysql.sock' --datadir '/Array/perconadb/data/' --defaults-file '/etc/my.cnf' --gtid '2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388''
    130613 10:02:05 [Note] WSREP: sst_donor_thread signaled with 0
    WSREP_SST: [ERROR] innobackupex finished with error: 25. Check /Array/perconadb/data//innobackup.backup.log (20130613 10:02:06.307)
    130613 10:02:06 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role 'donor' --address 'vm01db1.domain.com:4444/xtrabackup_sst' --auth 'sstuser:mysupersecretpasswordhere' --socket '/var/lib/mysql/mysql.sock' --datadir '/Array/perconadb/data/' --defaults-file '/etc/my.cnf' --gtid '2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388'
    130613 10:02:06 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'donor' --address 'vm01db1.domain.com:4444/xtrabackup_sst' --auth 'sstuser:mysupersecretpasswordhere' --socket '/var/lib/mysql/mysql.sock' --datadir '/Array/perconadb/data/' --defaults-file '/etc/my.cnf' --gtid '2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388': 22 (Invalid argument)
    130613 10:02:06 [Warning] WSREP: 0 (vm03db1): State transfer to 3 (vm01db1) failed: -1 (Operation not permitted)

    From the DONOR (vm03db1) - This is the innobackupex log
    30613 10:02:06 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_s ocket=/var/lib/mysql/mysql.sock' as 'sstuser' (using password: YES).
    130613 10:02:06 innobackupex: Connected to MySQL server
    IMPORTANT: Please check that the backup run completes successfully.
    At the end of a successful backup run innobackupex
    prints "completed OK!".

    innobackupex: Using mysql server version 5.5.30

    innobackupex: Created backup directory /tmp
    tar: -: Cannot write: Broken pipe
    tar: Error is not recoverable: exiting now
    innobackupex: Error: Failed to stream 'backup-my.cnf': Inappropriate ioctl for device at /usr//bin/innobackupex line 389.


    From the JOINER (vm01cv1) - This is the error I see
    130613 10:02:03 [Note] WSREP: State transfer required:
    Group state: 2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388
    Local state: 00000000-0000-0000-0000-000000000000:-1
    130613 10:02:03 [Note] WSREP: New cluster view: global state: 2c96604b-ca07-11e2-0800-c45d1fb1ae6f:1191388, view# 24: Primary, number of nodes: 4, my index: 3, protocol version 2
    130613 10:02:03 [Warning] WSREP: Gap in state sequence. Need state transfer.
    130613 10:02:05 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address 'vm01db1.domain.com' --auth 'sstuser:mysupersecretpasswordhere' --datadir '/Array/perconadb/data/' --defaults-file '/etc/my.cnf' --parent '3024''
    130613 10:02:05 [Note] WSREP: Prepared SST request: xtrabackup|vm01db1.domain.com:4444/xtrabackup_sst
    130613 10:02:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    130613 10:02:05 [Note] WSREP: Assign initial position for certification: 1191388, protocol version: 2
    130613 10:02:05 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (2c96604b-ca07-11e2-0800-c45d1fb1ae6f): 1 (Operation not permitted)
    at galera/src/replicator_str.cpprepare_for_IST():436. IST will be unavailable.
    130613 10:02:05 [Note] WSREP: Node 3 (vm01db1) requested state transfer from '*any*'. Selected 0 (vm03db1)(SYNCED) as donor.
    130613 10:02:05 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1191388)
    130613 10:02:05 [Note] WSREP: Requesting state transfer: success, donor: 0
    130613 10:02:06 [Warning] WSREP: 0 (vm03db1): State transfer to 3 (vm01db1) failed: -1 (Operation not permitted)
    130613 10:02:06 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.
    130613 10:02:06 [Note] WSREP: gcomm: terminating thread
    130613 10:02:06 [Note] WSREP: gcomm: joining thread
    130613 10:02:06 [Note] WSREP: gcomm: closing backend

Working...
X