Announcement

Announcement Module
Collapse
No announcement yet.

A node doesn't start normally

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • A node doesn't start normally

    Hello,

    I have an another problem.
    I can't start my node 3, i have always this error message :
    ERROR 1047 (08S01) at line 1: Unknown command
    So why ?

    with this problem, i can't stop mysql normally, i must execute kill -9

    Thx a lot

  • #2
    There must be some problem with state snapshot transfer. It is impossible to say anything more without seeing the logs.

    Comment


    • #3
      Thx for your help.
      Here the log :

      120514 16:49:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
      120514 16:49:22 [Warning] option 'wsrep_max_ws_rows': unsigned value 67108864 adjusted to 1048576
      120514 16:49:22 [Note] Flashcache bypass: disabled
      120514 16:49:22 [Note] Flashcache setup error is : ioctl failed

      120514 16:49:22 [Note] Plugin 'FEDERATED' is disabled.
      120514 16:49:23 InnoDB: The InnoDB memory heap is disabled
      120514 16:49:23 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      120514 16:49:23 InnoDB: Compressed tables use zlib 1.2.3
      120514 16:49:23 InnoDB: Using Linux native AIO
      120514 16:49:23 InnoDB: Initializing buffer pool, size = 1.0G
      120514 16:49:23 InnoDB: Completed initialization of buffer pool
      120514 16:49:23 InnoDB: highest supported file format is Barracuda.
      InnoDB: The log sequence number in ibdata files does not match
      InnoDB: the log sequence number in the ib_logfiles!
      120514 16:49:24 InnoDB: Database was not shut down normally!
      InnoDB: Starting crash recovery.
      InnoDB: Reading tablespace information from the .ibd files...
      InnoDB: Restoring possible half-written data pages from the doublewrite
      InnoDB: buffer...
      120514 16:49:27 InnoDB: Waiting for the background threads to start
      120514 16:49:28 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 370811796
      120514 16:49:28 [Warning] 'user' entry 'root@percona-1' ignored in --skip-name-resolve mode.
      120514 16:49:28 [Warning] 'proxies_priv' entry '@ root@percona-1' ignored in --skip-name-resolve mode.
      120514 16:49:28 [Note] Event Scheduler: Loaded 0 events
      120514 16:49:28 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
      120514 16:49:29 [Note] WSREP: wsrep_load(): Galera 2.1dev(r109) by Codership Oy loaded succesfully.
      120514 16:49:29 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
      120514 16:49:29 [Note] WSREP: Passing config to GCS: gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
      120514 16:49:29 [Note] WSREP: Start replication
      120514 16:49:29 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
      120514 16:49:29 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      120514 16:49:29 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      120514 16:49:29 [Note] WSREP: protonet asio version 0
      120514 16:49:29 [Note] WSREP: backend: asio
      120514 16:49:29 [Note] WSREP: GMCast version 0
      120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      120514 16:49:29 [Note] WSREP: EVS version 0
      120514 16:49:29 [Note] WSREP: PC version 0
      120514 16:49:29 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.7.110.3:'
      120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.4:4567
      120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') turning message relay requesting off
      120514 16:49:29 [Note] WSREP: declaring e039838f-9ab6-11e1-0800-ee89c2714ab0 stable
      120514 16:49:29 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
      120514 16:49:29 [Note] WSREP: view(view_id(PRIM,02230a2a-9dd4-11e1-0800-37cde4c9bd30,7) memb {
      02230a2a-9dd4-11e1-0800-37cde4c9bd30,
      e039838f-9ab6-11e1-0800-ee89c2714ab0,
      e137a28e-9ab6-11e1-0800-0427149fd658,
      } joined {
      } left {
      } partitioned {
      })
      120514 16:49:30 [Note] WSREP: gcomm: connected
      120514 16:49:30 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      120514 16:49:30 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      120514 16:49:30 [Note] WSREP: Opened channel 'my_wsrep_cluster'
      120514 16:49:30 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
      120514 16:49:30 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '5.5.20-55-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), wsrep_23.4.r3748
      120514 16:49:30 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 02c311df-9dd4-11e1-0800-4eb11325e27f
      120514 16:49:30 [Note] WSREP: STATE EXCHANGE: sent state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f
      120514 16:49:30 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 0 (percona-3)
      120514 16:49:30 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 1 (percona1)
      120514 16:49:30 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 2 (percona-2)
      120514 16:49:30 [Note] WSREP: Quorum results:
      version = 2,
      component = PRIMARY,
      conf_id = 6,
      members = 2/3 (joined/total),
      act_id = 61951,
      last_appl. = -1,
      protocols = 0/3/1 (gcs/repl/appl),
      group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
      120514 16:49:30 [Note] WSREP: Flow-control interval: [14, 28]
      120514 16:49:30 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 61951)
      120514 16:49:30 [Note] WSREP: State transfer required:
      Group state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951
      Local state: 00000000-0000-0000-0000-000000000000:-1
      120514 16:49:30 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 7: Primary, number of nodes: 3, my index: 0, protocol version 1
      120514 16:49:30 [Warning] WSREP: Gap in state sequence. Need state transfer.
      120514 16:49:32 [Note] WSREP: SST kill local trx: 10
      120514 16:49:32 [Note] WSREP: Prepared SST request: mysqldump|10.7.110.5:3306
      120514 16:49:32 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      120514 16:49:32 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
      120514 16:49:32 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (134487b3-8f74-11e1-0800-14b30cfd2465): 1 (Operation not permitted)
      at galera/src/replicator_str.cpprepare_for_IST():429. IST will be unavailable.
      120514 16:49:32 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
      120514 16:49:32 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 61951)
      120514 16:49:32 [Note] WSREP: Requesting state transfer: success, donor: 1
      120514 16:51:03 [Note] /usr/sbin/mysqld: Normal shutdown

      120514 16:51:03 [Note] WSREP: Stop replication
      120514 16:51:03 [Note] WSREP: Closing send monitor...
      120514 16:51:03 [Note] WSREP: Closed send monitor.
      120514 16:51:03 [Note] WSREP: gcomm: terminating thread
      120514 16:51:03 [Note] WSREP: gcomm: joining thread
      120514 16:51:03 [Note] WSREP: gcomm: closing backend
      120514 16:51:03 [Note] WSREP: view(view_id(NON_PRIM,02230a2a-9dd4-11e1-0800-37cde4c9bd30,7 ) memb {
      02230a2a-9dd4-11e1-0800-37cde4c9bd30,
      } joined {
      } left {
      } partitioned {
      e039838f-9ab6-11e1-0800-ee89c2714ab0,
      e137a28e-9ab6-11e1-0800-0427149fd658,
      })
      120514 16:51:03 [Note] WSREP: view((empty))
      120514 16:51:03 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      120514 16:51:03 [Note] WSREP: gcomm: closed
      120514 16:51:03 [Note] WSREP: Flow-control interval: [8, 16]
      120514 16:51:03 [Note] WSREP: Received NON-PRIMARY.
      120514 16:51:03 [Note] WSREP: Shifting JOINER -> OPEN (TO: 61951)
      120514 16:51:03 [Note] WSREP: Received self-leave message.
      120514 16:51:03 [Note] WSREP: Flow-control interval: [0, 0]
      120514 16:51:03 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      120514 16:51:03 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 61951)
      120514 16:51:03 [Note] WSREP: RECV thread exiting 0: Success
      120514 16:51:03 [Note] WSREP: recv_thread() joined.
      120514 16:51:03 [Note] WSREP: Closing slave action queue.
      120514 16:51:05 [Note] WSREP: SST kill local trx: 85
      120514 16:51:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
      120514 16:51:58 [Warning] option 'wsrep_max_ws_rows': unsigned value 67108864 adjusted to 1048576
      120514 16:51:58 [Note] Flashcache bypass: disabled
      120514 16:51:58 [Note] Flashcache setup error is : ioctl failed

      120514 16:51:58 [Note] Plugin 'FEDERATED' is disabled.
      120514 16:51:58 InnoDB: The InnoDB memory heap is disabled
      120514 16:51:58 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      120514 16:51:58 InnoDB: Compressed tables use zlib 1.2.3
      120514 16:51:58 InnoDB: Using Linux native AIO
      120514 16:51:58 InnoDB: Initializing buffer pool, size = 1.0G
      120514 16:51:58 InnoDB: Completed initialization of buffer pool
      120514 16:51:58 InnoDB: highest supported file format is Barracuda.
      InnoDB: The log sequence number in ibdata files does not match
      InnoDB: the log sequence number in the ib_logfiles!
      120514 16:51:58 InnoDB: Database was not shut down normally!
      InnoDB: Starting crash recovery.
      InnoDB: Reading tablespace information from the .ibd files...
      InnoDB: Restoring possible half-written data pages from the doublewrite
      InnoDB: buffer...
      120514 16:51:59 InnoDB: Waiting for the background threads to start
      120514 16:52:00 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 741028919
      120514 16:52:00 [Warning] 'user' entry 'root@percona-1' ignored in --skip-name-resolve mode.
      120514 16:52:00 [Warning] 'proxies_priv' entry '@ root@percona-1' ignored in --skip-name-resolve mode.
      120514 16:52:00 [Note] Event Scheduler: Loaded 0 events
      120514 16:52:00 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
      120514 16:52:00 [Note] WSREP: wsrep_load(): Galera 2.1dev(r109) by Codership Oy loaded succesfully.
      120514 16:52:01 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
      120514 16:52:01 [Note] WSREP: Passing config to GCS: gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
      120514 16:52:01 [Note] WSREP: Start replication
      120514 16:52:01 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
      120514 16:52:01 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      120514 16:52:01 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      120514 16:52:01 [Note] WSREP: protonet asio version 0
      120514 16:52:01 [Note] WSREP: backend: asio
      120514 16:52:01 [Note] WSREP: GMCast version 0
      120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      120514 16:52:01 [Note] WSREP: EVS version 0
      120514 16:52:01 [Note] WSREP: PC version 0
      120514 16:52:01 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.7.110.3:'
      120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.4:4567
      120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') turning message relay requesting off
      120514 16:52:01 [Note] WSREP: declaring e039838f-9ab6-11e1-0800-ee89c2714ab0 stable
      120514 16:52:01 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
      120514 16:52:01 [Note] WSREP: view(view_id(PRIM,5c9ad119-9dd4-11e1-0800-dea8fc9be39a,9) memb {
      5c9ad119-9dd4-11e1-0800-dea8fc9be39a,
      e039838f-9ab6-11e1-0800-ee89c2714ab0,
      e137a28e-9ab6-11e1-0800-0427149fd658,
      } joined {
      } left {
      } partitioned {
      })
      120514 16:52:01 [Note] WSREP: gcomm: connected
      120514 16:52:01 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      120514 16:52:01 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      120514 16:52:01 [Note] WSREP: Opened channel 'my_wsrep_cluster'
      120514 16:52:01 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
      120514 16:52:01 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '5.5.20-55-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), wsrep_23.4.r3748
      120514 16:52:01 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8
      120514 16:52:01 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8
      120514 16:52:01 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 0 (percona-3)
      120514 16:52:01 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 1 (percona1)
      120514 16:52:01 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 2 (percona-2)
      120514 16:52:01 [Note] WSREP: Quorum results:
      version = 2,
      component = PRIMARY,
      conf_id = 8,
      members = 2/3 (joined/total),
      act_id = 61951,
      last_appl. = -1,
      protocols = 0/3/1 (gcs/repl/appl),
      group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
      120514 16:52:01 [Note] WSREP: Flow-control interval: [14, 28]
      120514 16:52:01 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 61951)
      120514 16:52:01 [Note] WSREP: State transfer required:
      Group state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951
      Local state: 00000000-0000-0000-0000-000000000000:-1
      120514 16:52:01 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 9: Primary, number of nodes: 3, my index: 0, protocol version 1
      120514 16:52:01 [Warning] WSREP: Gap in state sequence. Need state transfer.
      120514 16:52:03 [Note] WSREP: Prepared SST request: mysqldump|10.7.110.5:3306
      120514 16:52:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      120514 16:52:03 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
      120514 16:52:03 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (134487b3-8f74-11e1-0800-14b30cfd2465): 1 (Operation not permitted)
      at galera/src/replicator_str.cpprepare_for_IST():429. IST will be unavailable.
      120514 16:52:03 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
      120514 16:52:03 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 61951)
      120514 16:52:03 [Note] WSREP: Requesting state transfer: success, donor: 1

      Comment


      • #4
        this the first node's log :


        120514 16:48:39 [Note] WSREP: declaring 02230a2a-9dd4-11e1-0800-37cde4c9bd30 stable
        120514 16:48:39 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
        120514 16:48:39 [Note] WSREP: view(view_id(PRIM,02230a2a-9dd4-11e1-0800-37cde4c9bd30,7) memb {
        02230a2a-9dd4-11e1-0800-37cde4c9bd30,
        e039838f-9ab6-11e1-0800-ee89c2714ab0,
        e137a28e-9ab6-11e1-0800-0427149fd658,
        } joined {
        } left {
        } partitioned {
        })
        120514 16:48:39 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
        120514 16:48:39 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
        120514 16:48:40 [Note] WSREP: STATE EXCHANGE: sent state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f
        120514 16:48:40 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 0 (percona-3)
        120514 16:48:40 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 1 (percona1)
        120514 16:48:40 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 2 (percona-2)
        120514 16:48:40 [Note] WSREP: Quorum results:
        version = 2,
        component = PRIMARY,
        conf_id = 6,
        members = 2/3 (joined/total),
        act_id = 61951,
        last_appl. = 0,
        protocols = 0/3/1 (gcs/repl/appl),
        group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
        120514 16:48:40 [Note] WSREP: Flow-control interval: [14, 28]
        120514 16:48:40 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 7: Primary, number of nodes: 3, my index: 1, protocol version 1
        120514 16:48:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        120514 16:48:40 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
        120514 16:48:42 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
        120514 16:48:42 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 61951)
        120514 16:48:42 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        hostname: Name or service not known
        hostname: Name or service not known
        ERROR 1010 (HY000) at line 451: Error dropping database (can't rmdir './communautaire', errno: 39)
        mysqldump: Got errno 32 on write
        120514 16:49:42 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
        120514 16:49:42 [ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
        hostname: Name or service not known
        hostname: Name or service not known
        120514 16:50:13 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
        120514 16:50:13 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.5:4567
        120514 16:50:13 [Note] WSREP: view(view_id(PRIM,e039838f-9ab6-11e1-0800-ee89c2714ab0, memb {
        e039838f-9ab6-11e1-0800-ee89c2714ab0,
        e137a28e-9ab6-11e1-0800-0427149fd658,
        } joined {
        } left {
        } partitioned {
        02230a2a-9dd4-11e1-0800-37cde4c9bd30,
        })
        120514 16:50:13 [Note] WSREP: forgetting 02230a2a-9dd4-11e1-0800-37cde4c9bd30 (tcp://10.7.110.5:4567)
        120514 16:50:13 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting off
        120514 16:50:13 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
        120514 16:50:13 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1c7238d2-9dd4-11e1-0800-2cea06775603
        120514 16:50:13 [Note] WSREP: STATE EXCHANGE: sent state msg: 1c7238d2-9dd4-11e1-0800-2cea06775603
        120514 16:50:13 [Note] WSREP: STATE EXCHANGE: got state msg: 1c7238d2-9dd4-11e1-0800-2cea06775603 from 0 (percona1)
        120514 16:50:13 [Note] WSREP: STATE EXCHANGE: got state msg: 1c7238d2-9dd4-11e1-0800-2cea06775603 from 1 (percona-2)
        120514 16:50:13 [Note] WSREP: Quorum results:
        version = 2,
        component = PRIMARY,
        conf_id = 7,
        members = 2/2 (joined/total),
        act_id = 61951,
        last_appl. = 0,
        protocols = 0/3/1 (gcs/repl/appl),
        group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
        120514 16:50:13 [Note] WSREP: Flow-control interval: [12, 23]
        120514 16:50:18 [Note] WSREP: cleaning up 02230a2a-9dd4-11e1-0800-37cde4c9bd30 (tcp://10.7.110.5:4567)
        ERROR 1010 (HY000) at line 451: Error dropping database (can't rmdir './communautaire', errno: 39)
        mysqldump: Got errno 32 on write
        120514 16:50:37 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
        120514 16:50:37 [ERROR] WSREP: Try 2/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
        hostname: Name or service not known
        hostname: Name or service not known
        ERROR 2003 (HY000): Can't connect to MySQL server on '10.7.110.5' (4)
        120514 16:50:48 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
        120514 16:50:48 [ERROR] WSREP: Try 3/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
        120514 16:50:49 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 8: Primary, number of nodes: 2, my index: 0, protocol version 1
        120514 16:50:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        120514 16:50:49 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
        120514 16:50:49 [Warning] WSREP: Could not find peer: 02230a2a-9dd4-11e1-0800-37cde4c9bd30
        120514 16:50:49 [Warning] WSREP: 0 (percona1): State transfer to -1 (left the group) failed: -1 (Operation not permitted)
        120514 16:50:49 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 61951)
        120514 16:50:49 [Note] WSREP: Member 0 (percona1) synced with group.
        120514 16:50:49 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 61951)
        120514 16:50:49 [Note] WSREP: Synchronized with group, ready for connections
        120514 16:50:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        120514 16:51:11 [Note] WSREP: declaring 5c9ad119-9dd4-11e1-0800-dea8fc9be39a stable
        120514 16:51:11 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
        120514 16:51:11 [Note] WSREP: view(view_id(PRIM,5c9ad119-9dd4-11e1-0800-dea8fc9be39a,9) memb {
        5c9ad119-9dd4-11e1-0800-dea8fc9be39a,
        e039838f-9ab6-11e1-0800-ee89c2714ab0,
        e137a28e-9ab6-11e1-0800-0427149fd658,
        } joined {
        } left {
        } partitioned {
        })
        120514 16:51:11 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
        120514 16:51:11 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
        120514 16:51:11 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8
        120514 16:51:11 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 0 (percona-3)
        120514 16:51:11 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 1 (percona1)
        120514 16:51:11 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 2 (percona-2)
        120514 16:51:11 [Note] WSREP: Quorum results:
        version = 2,
        component = PRIMARY,
        conf_id = 8,
        members = 2/3 (joined/total),
        act_id = 61951,
        last_appl. = 0,
        protocols = 0/3/1 (gcs/repl/appl),
        group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
        120514 16:51:11 [Note] WSREP: Flow-control interval: [14, 28]
        120514 16:51:11 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 9: Primary, number of nodes: 3, my index: 1, protocol version 1
        120514 16:51:11 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        120514 16:51:11 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
        120514 16:51:13 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
        120514 16:51:13 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 61951)
        120514 16:51:13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        hostname: Name or service not known
        hostname: Name or service not known
        ERROR120514 16:51:34 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.5:4567
        2013 (HY000) at line 365: Lost connection to MySQL server during query
        mysqldump: Got errno 32 on write
        120514 16:51:34 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
        120514 16:51:34 [ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
        hostname: Name or service not known
        hostname: Name or service not known
        ERROR 2003 (HY000): Can't connect to MySQL server on '10.7.110.5' (111)
        120514 16:51:35 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
        120514 16:51:35 [ERROR] WSREP: Try 2/3: 'wsrep_sst_mysqldump 'sst' 'sstpass123' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
        120514 16:51:35 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') reconnecting to 5c9ad119-9dd4-11e1-0800-dea8fc9be39a (tcp://10.7.110.5:4567), attempt 0
        hostname: Name or service not known
        hostname: Name or service not known
        ERROR 2003 (HY000): Can't connect to MySQL server on '10.7.110.5' (111)
        120514 16:51:36 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
        120514 16:51:36 [ERROR] WSREP: Try 3/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
        120514 16:51:38 [Note] WSREP: evs:roto(e039838f-9ab6-11e1-0800-ee89c2714ab0, GATHER, view_id(REG,5c9ad119-9dd4-11e1-0800-dea8fc9be39a,9)) suspecting node: 5c9ad119-9dd4-11e1-0800-dea8fc9be39a
        120514 16:51:38 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
        120514 16:51:38 [Warning] WSREP: 1 (percona1): State transfer to 0 (percona-3) failed: -1 (Operation not permitted)
        120514 16:51:38 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 61951)
        120514 16:51:38 [Note] WSREP: view(view_id(PRIM,e039838f-9ab6-11e1-0800-ee89c2714ab0,10) memb {
        e039838f-9ab6-11e1-0800-ee89c2714ab0,
        e137a28e-9ab6-11e1-0800-0427149fd658,
        } joined {
        } left {
        } partitioned {
        5c9ad119-9dd4-11e1-0800-dea8fc9be39a,
        })
        120514 16:51:38 [Note] WSREP: forgetting 5c9ad119-9dd4-11e1-0800-dea8fc9be39a (tcp://10.7.110.5:4567)
        120514 16:51:38 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting off
        120514 16:51:38 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
        120514 16:51:38 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0
        120514 16:51:38 [Warning] WSREP: SYNC message from member 0 in non-primary configuration. Ignored.
        120514 16:51:38 [Note] WSREP: STATE EXCHANGE: sent state msg: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0
        120514 16:51:39 [Note] WSREP: STATE EXCHANGE: got state msg: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0 from 0 (percona1)
        120514 16:51:39 [Note] WSREP: STATE EXCHANGE: got state msg: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0 from 1 (percona-2)
        120514 16:51:39 [Note] WSREP: Quorum results:
        version = 2,
        component = PRIMARY,
        conf_id = 9,
        members = 2/2 (joined/total),
        act_id = 61951,
        last_appl. = 0,
        protocols = 0/3/1 (gcs/repl/appl),
        group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
        120514 16:51:39 [Note] WSREP: Flow-control interval: [12, 23]
        120514 16:51:39 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 10: Primary, number of nodes: 2, my index: 0, protocol version 1
        120514 16:51:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        120514 16:51:39 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
        120514 16:51:39 [Note] WSREP: Member 0 (percona1) synced with group.
        120514 16:51:39 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 61951)
        120514 16:51:39 [Note] WSREP: Synchronized with group, ready for connections
        120514 16:51:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        120514 16:51:44 [Note] WSREP: cleaning up 5c9ad119-9dd4-11e1-0800-dea8fc9be39a (tcp://10.7.110.5:4567)

        Comment


        • #5
          azigui wrote on Tue, 15 May 2012 19:12
          120514 16:49:32 [Note] WSREP: Requesting state transfer: success, donor: 1
          120514 16:51:03 [Note] /usr/sbin/mysqld: Normal shutdown
          Who initiated this shutdown?

          Also from donor log it follows that 10.7.110.5 has a corrupt data directory. You need to fix that first:
          Quote:
          120514 16:48:42 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
          hostname: Name or service not known
          hostname: Name or service not known
          ERROR 1010 (HY000) at line 451: Error dropping database (can't rmdir './communautaire', errno: 39)
          mysqldump: Got errno 32 on write
          The easiest way to do this is to use xtrabackup or rsync based SST.

          Comment


          • #6
            Hello,

            Thx a lot for your response.
            I have tested with xtrabackup, the firt time is working
            But now it doesn't work, so how to configure it

            Best regards

            Azigui

            Comment


            • #7
              I resolve it, you must put in my.cnf file in [MYSQLD] section :
              datadir=/var/lib/mysql
              tmpdir=/tmp
              user=mysql

              Be carrefull, you can probably obtain a start/stop failed, so you must update password for mysql user debian-syst-main.
              This password is written in debian.cnf file

              Comment

              Working...
              X