Second node don't sync neigther xtrabackup nor rsync

  • Filter
  • Time
  • Show
Clear All
new posts

  • Second node don't sync neigther xtrabackup nor rsync

    Hello, everybody.
    I'm trying change from MySQL master-master to Percona multi-master setup. The first node (donor) is starting ok, but second node doesn't up. I can see their IP with show status like 'wsrep_incoming_addresses', but some timer after the second node go down.

    In my log I found some information, but isn't clear to me. I pasted below only must important info found in my log:
    removed `/database/mysql/ibdata1'
    WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20131026 12:37:17.290)
    WSREP_SST: [ERROR] xtrabackup process ended without creating '/database/mysql//xtrabackup_galera_info' (20131026 12:37:26.969)
    WSREP_SST: [INFO] Contents of datadir (20131026 12:37:26.972)
    WSREP_SST: [INFO] -rw------- 1 mysql mysql 134219048 Oct 26 12:37 /database/mysql//galera.cache
    -rw-rw---- 1 mysql mysql 104 Oct 26 12:37 /database/mysql//grastate.dat
    -rw-rw---- 1 mysql mysql 0 Oct 26 12:37 /database/mysql//sst_in_progress (20131026 12:37:26.978)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20131026 12:37:26.981)
    WSREP_SST: [INFO] Removing the sst_in_progress file (20131026 12:37:26.984)
    131026 12:37:26 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '' --auth 'replication:replication' --datadir '/database/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '24611': 32 (Broken pipe)
    131026 12:37:26 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    131026 12:37:26 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    131026 12:37:26 [ERROR] Aborting

    131026 12:37:26 [Warning] WSREP: 1 (oldVaio): State transfer to 0 (perolaNegra) failed: -1 (Operation not permitted)
    131026 12:37:26 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.

    I did 'grant all privileges on *.* to replication@'%' identified by 'replication' (that is a lab environment, I don't care with security for now), so I think the permissions are ok.
    I changed many permissions to 777 in directories and 666 in many files to avoid any restriction, but I have no idea what I can to do anymore.
    My cnf files are below:

    #Performance schema
    #performance_schema = OFF

    datadir = /database/mysql
    socket = /database/mysqld.sock
    pid_file = /database/mysqld.pid
    user = mysql
    port = 3306

    innodb_flush_log_at_trx_commit = 2
    innodb_buffer_pool_size = 2G
    innodb_buffer_pool_instances = 1
    innodb_log_file_size = 64M
    innodb_file_per_table = 1
    innodb_flush_method = O_DIRECT

    key_buffer_size = 200M
    myisam_block_size = 4096
    myisam_sort_buffer_size = 256M

    log_bin = mysql-bin
    log_error = /database/mysql-error.log
    long_query_time = 10
    slow_query_log = ON
    slow_query_log_file = /database/mysql-slow.log

    bind-address =
    tmp_table_size = 32M
    max_heap_table_size = 32M
    query_cache_type = 0
    query_cache_size = 0
    max_connections = 100
    thread_cache_size = 20
    open_files_limit = 65535
    skip_name_resolve = OFF

    binlog_format = ROW
    wsrep_provider = /usr/lib/libgalera_smm.so
    wsrep_cluster_address = gcomm:// #,,
    default_storage_engine = InnoDB
    innodb_locks_unsafe_for_binlog = 1
    innodb_autoinc_lock_mode = 2
    wsrep_node_address =
    wsrep_sst_method = rsync #xtrabackup
    wsrep_cluster_name = axiros_lab
    wsrep_sst_auth = "replication:replication"

    socket = /var/lib/mysql/mysql.sock
    port = 3306


    All configurations are the same, so I will paste just the galera configuration block:
    binlog_format = ROW
    wsrep_provider = /usr/lib/libgalera_smm.so
    wsrep_cluster_address = gcomm://,
    #wsrep_cluster_address = gcomm://
    default_storage_engine = InnoDB
    innodb_locks_unsafe_for_binlog = 1
    innodb_autoinc_lock_mode = 2
    wsrep_node_address =
    wsrep_sst_method = xtrabackup
    wsrep_cluster_name = axiros_lab
    wsrep_sst_auth = "replication:replication"

    How you can see, I'm trying avoid any problem in the first setup to see everything running. I will add a third node when my problems are solved.
    I have no firewall rules, I'm using 3 notebooks to create a cluster to test.
    My operating system are Debian Wheezy in all nodes. The packages was downloaded via percona repository.
    All notebooks and systems are 64bits.

    Please, do someone have suggestions?

  • #2
    After many hours crying and trying, I changed sst tool. With rsync the things are running now so, seems to be a bug with xtrabackup tool.
    The logs are terrible, aren't clear to understand problems. I'm feeling afraid to continue trying. In lab I passed many hours of hard work to discover this problem; I'm thinking if I have some problem in production...


    • #3
      Can you check the contents of the innobackup.backup.log on donor node for errors?


      • #4
        Sorry my delay.
        I will do in the next setup, sorry. I don't have this log because I'm running in a chroot with no rsyslog.