GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

Error while getting data from donor node

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Error while getting data from donor node

    Hello
    I've a problem in joining new node to cluster.
    here is my log :
    I tested the test_sstuser working on honor without any problem .
    but in new nodes I get these errors :

    Code:
    140531 04:01:34 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    140531 04:01:34 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.C9XMX6' --pid-file='/var/lib/mysql/master-2.test.net-recover.pid'
    2014-05-31 04:01:35 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    140531 04:01:37 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
    2014-05-31 04:01:37 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
    2014-05-31 04:01:37 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2014-05-31 04:01:37 2391 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2014-05-31 04:01:37 2391 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
    2014-05-31 04:01:37 2391 [Note] WSREP: wsrep_load(): Galera 3.5(r178) by Codership Oy <info@codership.com> loaded successfully.
    2014-05-31 04:01:37 2391 [Note] WSREP: CRC-32C: using hardware acceleration.
    2014-05-31 04:01:37 2391 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
    2014-05-31 04:01:37 2391 [Note] WSREP: Passing config to GCS: base_host = 192.168.0.2; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; proton
    2014-05-31 04:01:37 2391 [Note] WSREP: Service thread queue flushed.
    2014-05-31 04:01:37 2391 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
    2014-05-31 04:01:37 2391 [Note] WSREP: wsrep_sst_grab()
    2014-05-31 04:01:37 2391 [Note] WSREP: Start replication
    2014-05-31 04:01:37 2391 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
    2014-05-31 04:01:37 2391 [Note] WSREP: protonet asio version 0
    2014-05-31 04:01:37 2391 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
    2014-05-31 04:01:37 2391 [Note] WSREP: backend: asio
    2014-05-31 04:01:37 2391 [Note] WSREP: GMCast version 0
    2014-05-31 04:01:37 2391 [Note] WSREP: (8b703e66-e852-11e3-95d2-d2704d8c333d, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2014-05-31 04:01:37 2391 [Note] WSREP: (8b703e66-e852-11e3-95d2-d2704d8c333d, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2014-05-31 04:01:37 2391 [Note] WSREP: EVS version 0
    2014-05-31 04:01:37 2391 [Note] WSREP: PC version 0
    2014-05-31 04:01:37 2391 [Note] WSREP: gcomm: connecting to group 'cluster_test', peer '192.168.0.1:,192.168.0.2:,192.168.0.3:'
    2014-05-31 04:01:37 2391 [Warning] WSREP: (8b703e66-e852-11e3-95d2-d2704d8c333d, 'tcp://0.0.0.0:4567') address 'tcp://192.168.0.2:4567' points to own listening address, blacklisting
    2014-05-31 04:01:37 2391 [Note] WSREP: (8b703e66-e852-11e3-95d2-d2704d8c333d, 'tcp://0.0.0.0:4567') address 'tcp://192.168.0.2:4567' pointing to uuid 8b703e66-e852-11e3-95d2-d2704d8c333d is blacklisted, skipping
    2014-05-31 04:01:38 2391 [Note] WSREP: declaring dbff73e9-e82f-11e3-8554-af724dd9b3ae stable
    2014-05-31 04:01:38 2391 [Note] WSREP: Node dbff73e9-e82f-11e3-8554-af724dd9b3ae state prim
    2014-05-31 04:01:38 2391 [Note] WSREP: view(view_id(PRIM,8b703e66-e852-11e3-95d2-d2704d8c333d,12) memb {
        8b703e66-e852-11e3-95d2-d2704d8c333d,0
        dbff73e9-e82f-11e3-8554-af724dd9b3ae,0
    } joined {
    } left {
    } partitioned {
    })
    2014-05-31 04:01:38 2391 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.0.3:4567
    2014-05-31 04:01:38 2391 [Note] WSREP: gcomm: connected
    2014-05-31 04:01:38 2391 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2014-05-31 04:01:38 2391 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2014-05-31 04:01:38 2391 [Note] WSREP: Opened channel 'cluster_test'
    2014-05-31 04:01:38 2391 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
    2014-05-31 04:01:38 2391 [Note] WSREP: Waiting for SST to complete.
    2014-05-31 04:01:38 2391 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 8c097be1-e852-11e3-8b95-13d3ed4bfad0
    2014-05-31 04:01:38 2391 [Note] WSREP: STATE EXCHANGE: sent state msg: 8c097be1-e852-11e3-8b95-13d3ed4bfad0
    2014-05-31 04:01:38 2391 [Note] WSREP: STATE EXCHANGE: got state msg: 8c097be1-e852-11e3-8b95-13d3ed4bfad0 from 0 (master-2.test.net)
    2014-05-31 04:01:38 2391 [Note] WSREP: STATE EXCHANGE: got state msg: 8c097be1-e852-11e3-8b95-13d3ed4bfad0 from 1 (test.host.com)
    2014-05-31 04:01:38 2391 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 11,
        members    = 1/2 (joined/total),
        act_id     = 2,
        last_appl. = -1,
        protocols  = 0/5/2 (gcs/repl/appl),
        group UUID = 02dc9b38-e825-11e3-a67b-4bfddfde0c98
    2014-05-31 04:01:38 2391 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2)
    2014-05-31 04:01:38 2391 [Note] WSREP: State transfer required: 
        Group state: 02dc9b38-e825-11e3-a67b-4bfddfde0c98:2
        Local state: 00000000-0000-0000-0000-000000000000:-1
    2014-05-31 04:01:38 2391 [Note] WSREP: New cluster view: global state: 02dc9b38-e825-11e3-a67b-4bfddfde0c98:2, view# 12: Primary, number of nodes: 2, my index: 0, protocol version 2
    2014-05-31 04:01:38 2391 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2014-05-31 04:01:40 2391 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.0.2' --auth 'test_sstuser:123456' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '2391'  '' '
    WSREP_SST: [INFO] Streaming with xbstream (20140531 04:01:40.700)
    WSREP_SST: [INFO] Using socat as streamer (20140531 04:01:40.702)
    WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20140531 04:01:40.912)
    2014-05-31 04:01:40 2391 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.0.2:4444/xtrabackup_sst
    2014-05-31 04:01:40 2391 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2014-05-31 04:01:40 2391 [Note] WSREP: REPL Protocols: 5 (3, 1)
    2014-05-31 04:01:40 2391 [Note] WSREP: Service thread queue flushed.
    2014-05-31 04:01:40 2391 [Note] WSREP: Assign initial position for certification: 2, protocol version: 3
    2014-05-31 04:01:40 2391 [Note] WSREP: Service thread queue flushed.
    2014-05-31 04:01:40 2391 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (02dc9b38-e825-11e3-a67b-4bfddfde0c98): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
    WSREP_SST: [INFO] Evaluating timeout 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140531 04:01:40.930)
    2014/05/31 04:01:40 socat[2647] E bind(3, {AF=2 0.0.0.0:4444}, 16): Address already in use
    WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20140531 04:01:40.934)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20140531 04:01:40.936)
    WSREP_SST: [INFO] Removing the sst_in_progress file (20140531 04:01:40.937)
    2014-05-31 04:01:40 2391 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.0.2' --auth 'test_sstuser:123456' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '2391'  '' : 32 (Broken pipe)
    2014-05-31 04:01:40 2391 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    2014-05-31 04:01:40 2391 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    2014-05-31 04:01:40 2391 [ERROR] Aborting
    
    2014-05-31 04:01:40 2391 [Note] WSREP: Member 0.0 (master-2.test.net) requested state transfer from '*any*'. Selected 1.0 (test.host.com)(SYNCED) as donor.
    2014-05-31 04:01:40 2391 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2)
    2014-05-31 04:01:40 2391 [Note] WSREP: Requesting state transfer: success, donor: 1
    2014-05-31 04:01:41 2391 [Warning] WSREP: 1.0 (test.host.com): State transfer to 0.0 (master-2.test.net) failed: -32 (Broken pipe)
    2014-05-31 04:01:41 2391 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
    2014-05-31 04:01:41 2391 [Note] WSREP: gcomm: terminating thread
    2014-05-31 04:01:41 2391 [Note] WSREP: gcomm: joining thread
    2014-05-31 04:01:41 2391 [Note] WSREP: gcomm: closing backend
    2014-05-31 04:01:42 2391 [Note] WSREP: view(view_id(NON_PRIM,8b703e66-e852-11e3-95d2-d2704d8c333d,12) memb {
        8b703e66-e852-11e3-95d2-d2704d8c333d,0
    } joined {
    } left {
    } partitioned {
        dbff73e9-e82f-11e3-8554-af724dd9b3ae,0
    })
    2014-05-31 04:01:42 2391 [Note] WSREP: view((empty))
    2014-05-31 04:01:42 2391 [Note] WSREP: gcomm: closed
    2014-05-31 04:01:42 2391 [Note] WSREP: /usr/sbin/mysqld: Terminated.
    140531 04:01:42 mysqld_safe mysqld from pid file /var/lib/mysql/master-2.test.net.pid ended
    how can I solve this problem ?
    Best regards
    Ali

  • #2

    >> 2014-05-31 04:01:40 2391 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.0.2' --auth 'test_sstuser:123456' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '2391' '' : 32 (Broken pipe) >> 2014-05-31 04:01:40 2391 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. >> 2014-05-31 04:01:40 2391 [ERROR] WSREP: SST failed: 32 (Broken pipe) >> 2014-05-31 04:01:40 2391 [ERROR] Aborting I would suggest to check backup log. It should point you the exact error during SST. Check here for details on installation http://www.percona.com/doc/percona-x...ots_howto.html

    Comment

    Working...
    X