There must be some problem with state snapshot transfer. It is impossible to say anything more without seeing the logs.

Thx for your help.
Here the log :

120514 16:49:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120514 16:49:22 [Warning] option 'wsrep_max_ws_rows': unsigned value 67108864 adjusted to 1048576
120514 16:49:22 [Note] Flashcache bypass: disabled
120514 16:49:22 [Note] Flashcache setup error is : ioctl failed

120514 16:49:22 [Note] Plugin 'FEDERATED' is disabled.
120514 16:49:23 InnoDB: The InnoDB memory heap is disabled
120514 16:49:23 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120514 16:49:23 InnoDB: Compressed tables use zlib 1.2.3
120514 16:49:23 InnoDB: Using Linux native AIO
120514 16:49:23 InnoDB: Initializing buffer pool, size = 1.0G
120514 16:49:23 InnoDB: Completed initialization of buffer pool
120514 16:49:23 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120514 16:49:24 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
120514 16:49:27 InnoDB: Waiting for the background threads to start
120514 16:49:28 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 370811796
120514 16:49:28 [Warning] 'user' entry 'root@percona-1' ignored in --skip-name-resolve mode.
120514 16:49:28 [Warning] 'proxies_priv' entry '@ root@percona-1' ignored in --skip-name-resolve mode.
120514 16:49:28 [Note] Event Scheduler: Loaded 0 events
120514 16:49:28 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
120514 16:49:29 [Note] WSREP: wsrep_load(): Galera 2.1dev(r109) by Codership Oy loaded succesfully.
120514 16:49:29 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
120514 16:49:29 [Note] WSREP: Passing config to GCS: gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
120514 16:49:29 [Note] WSREP: Start replication
120514 16:49:29 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
120514 16:49:29 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
120514 16:49:29 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
120514 16:49:29 [Note] WSREP: protonet asio version 0
120514 16:49:29 [Note] WSREP: backend: asio
120514 16:49:29 [Note] WSREP: GMCast version 0
120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
120514 16:49:29 [Note] WSREP: EVS version 0
120514 16:49:29 [Note] WSREP: PC version 0
120514 16:49:29 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.7.110.3:'
120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.4:4567
120514 16:49:29 [Note] WSREP: (02230a2a-9dd4-11e1-0800-37cde4c9bd30, 'tcp://0.0.0.0:4567') turning message relay requesting off
120514 16:49:29 [Note] WSREP: declaring e039838f-9ab6-11e1-0800-ee89c2714ab0 stable
120514 16:49:29 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
120514 16:49:29 [Note] WSREP: view(view_id(PRIM,02230a2a-9dd4-11e1-0800-37cde4c9bd30,7) memb {
02230a2a-9dd4-11e1-0800-37cde4c9bd30,
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
} joined {
} left {
} partitioned {
})
120514 16:49:30 [Note] WSREP: gcomm: connected
120514 16:49:30 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
120514 16:49:30 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
120514 16:49:30 [Note] WSREP: Opened channel 'my_wsrep_cluster'
120514 16:49:30 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
120514 16:49:30 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.20-55-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), wsrep_23.4.r3748
120514 16:49:30 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 02c311df-9dd4-11e1-0800-4eb11325e27f
120514 16:49:30 [Note] WSREP: STATE EXCHANGE: sent state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f
120514 16:49:30 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 0 (percona-3)
120514 16:49:30 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 1 (percona1)
120514 16:49:30 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 2 (percona-2)
120514 16:49:30 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 6,
members = 2/3 (joined/total),
act_id = 61951,
last_appl. = -1,
protocols = 0/3/1 (gcs/repl/appl),
group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
120514 16:49:30 [Note] WSREP: Flow-control interval: [14, 28]
120514 16:49:30 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 61951)
120514 16:49:30 [Note] WSREP: State transfer required:
Group state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951
Local state: 00000000-0000-0000-0000-000000000000:-1
120514 16:49:30 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 7: Primary, number of nodes: 3, my index: 0, protocol version 1
120514 16:49:30 [Warning] WSREP: Gap in state sequence. Need state transfer.
120514 16:49:32 [Note] WSREP: SST kill local trx: 10
120514 16:49:32 [Note] WSREP: Prepared SST request: mysqldump|10.7.110.5:3306
120514 16:49:32 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:49:32 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
120514 16:49:32 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (134487b3-8f74-11e1-0800-14b30cfd2465): 1 (Operation not permitted)
at galera/src/replicator_str.cpprepare_for_IST():429. IST will be unavailable.
120514 16:49:32 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
120514 16:49:32 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 61951)
120514 16:49:32 [Note] WSREP: Requesting state transfer: success, donor: 1
120514 16:51:03 [Note] /usr/sbin/mysqld: Normal shutdown

120514 16:51:03 [Note] WSREP: Stop replication
120514 16:51:03 [Note] WSREP: Closing send monitor...
120514 16:51:03 [Note] WSREP: Closed send monitor.
120514 16:51:03 [Note] WSREP: gcomm: terminating thread
120514 16:51:03 [Note] WSREP: gcomm: joining thread
120514 16:51:03 [Note] WSREP: gcomm: closing backend
120514 16:51:03 [Note] WSREP: view(view_id(NON_PRIM,02230a2a-9dd4-11e1-0800-37cde4c9bd30,7 ) memb {
02230a2a-9dd4-11e1-0800-37cde4c9bd30,
} joined {
} left {
} partitioned {
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
})
120514 16:51:03 [Note] WSREP: view((empty))
120514 16:51:03 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
120514 16:51:03 [Note] WSREP: gcomm: closed
120514 16:51:03 [Note] WSREP: Flow-control interval: [8, 16]
120514 16:51:03 [Note] WSREP: Received NON-PRIMARY.
120514 16:51:03 [Note] WSREP: Shifting JOINER -> OPEN (TO: 61951)
120514 16:51:03 [Note] WSREP: Received self-leave message.
120514 16:51:03 [Note] WSREP: Flow-control interval: [0, 0]
120514 16:51:03 [Note] WSREP: Received SELF-LEAVE. Closing connection.
120514 16:51:03 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 61951)
120514 16:51:03 [Note] WSREP: RECV thread exiting 0: Success
120514 16:51:03 [Note] WSREP: recv_thread() joined.
120514 16:51:03 [Note] WSREP: Closing slave action queue.
120514 16:51:05 [Note] WSREP: SST kill local trx: 85
120514 16:51:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120514 16:51:58 [Warning] option 'wsrep_max_ws_rows': unsigned value 67108864 adjusted to 1048576
120514 16:51:58 [Note] Flashcache bypass: disabled
120514 16:51:58 [Note] Flashcache setup error is : ioctl failed

120514 16:51:58 [Note] Plugin 'FEDERATED' is disabled.
120514 16:51:58 InnoDB: The InnoDB memory heap is disabled
120514 16:51:58 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120514 16:51:58 InnoDB: Compressed tables use zlib 1.2.3
120514 16:51:58 InnoDB: Using Linux native AIO
120514 16:51:58 InnoDB: Initializing buffer pool, size = 1.0G
120514 16:51:58 InnoDB: Completed initialization of buffer pool
120514 16:51:58 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120514 16:51:58 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
120514 16:51:59 InnoDB: Waiting for the background threads to start
120514 16:52:00 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 741028919
120514 16:52:00 [Warning] 'user' entry 'root@percona-1' ignored in --skip-name-resolve mode.
120514 16:52:00 [Warning] 'proxies_priv' entry '@ root@percona-1' ignored in --skip-name-resolve mode.
120514 16:52:00 [Note] Event Scheduler: Loaded 0 events
120514 16:52:00 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
120514 16:52:00 [Note] WSREP: wsrep_load(): Galera 2.1dev(r109) by Codership Oy loaded succesfully.
120514 16:52:01 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
120514 16:52:01 [Note] WSREP: Passing config to GCS: gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
120514 16:52:01 [Note] WSREP: Start replication
120514 16:52:01 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
120514 16:52:01 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
120514 16:52:01 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
120514 16:52:01 [Note] WSREP: protonet asio version 0
120514 16:52:01 [Note] WSREP: backend: asio
120514 16:52:01 [Note] WSREP: GMCast version 0
120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
120514 16:52:01 [Note] WSREP: EVS version 0
120514 16:52:01 [Note] WSREP: PC version 0
120514 16:52:01 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.7.110.3:'
120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.4:4567
120514 16:52:01 [Note] WSREP: (5c9ad119-9dd4-11e1-0800-dea8fc9be39a, 'tcp://0.0.0.0:4567') turning message relay requesting off
120514 16:52:01 [Note] WSREP: declaring e039838f-9ab6-11e1-0800-ee89c2714ab0 stable
120514 16:52:01 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
120514 16:52:01 [Note] WSREP: view(view_id(PRIM,5c9ad119-9dd4-11e1-0800-dea8fc9be39a,9) memb {
5c9ad119-9dd4-11e1-0800-dea8fc9be39a,
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
} joined {
} left {
} partitioned {
})
120514 16:52:01 [Note] WSREP: gcomm: connected
120514 16:52:01 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
120514 16:52:01 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
120514 16:52:01 [Note] WSREP: Opened channel 'my_wsrep_cluster'
120514 16:52:01 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
120514 16:52:01 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.20-55-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), wsrep_23.4.r3748
120514 16:52:01 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8
120514 16:52:01 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8
120514 16:52:01 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 0 (percona-3)
120514 16:52:01 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 1 (percona1)
120514 16:52:01 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 2 (percona-2)
120514 16:52:01 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 8,
members = 2/3 (joined/total),
act_id = 61951,
last_appl. = -1,
protocols = 0/3/1 (gcs/repl/appl),
group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
120514 16:52:01 [Note] WSREP: Flow-control interval: [14, 28]
120514 16:52:01 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 61951)
120514 16:52:01 [Note] WSREP: State transfer required:
Group state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951
Local state: 00000000-0000-0000-0000-000000000000:-1
120514 16:52:01 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 9: Primary, number of nodes: 3, my index: 0, protocol version 1
120514 16:52:01 [Warning] WSREP: Gap in state sequence. Need state transfer.
120514 16:52:03 [Note] WSREP: Prepared SST request: mysqldump|10.7.110.5:3306
120514 16:52:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:52:03 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
120514 16:52:03 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (134487b3-8f74-11e1-0800-14b30cfd2465): 1 (Operation not permitted)
at galera/src/replicator_str.cpprepare_for_IST():429. IST will be unavailable.
120514 16:52:03 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
120514 16:52:03 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 61951)
120514 16:52:03 [Note] WSREP: Requesting state transfer: success, donor: 1

this the first node's log :


120514 16:48:39 [Note] WSREP: declaring 02230a2a-9dd4-11e1-0800-37cde4c9bd30 stable
120514 16:48:39 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
120514 16:48:39 [Note] WSREP: view(view_id(PRIM,02230a2a-9dd4-11e1-0800-37cde4c9bd30,7) memb {
02230a2a-9dd4-11e1-0800-37cde4c9bd30,
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
} joined {
} left {
} partitioned {
})
120514 16:48:39 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
120514 16:48:39 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
120514 16:48:40 [Note] WSREP: STATE EXCHANGE: sent state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f
120514 16:48:40 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 0 (percona-3)
120514 16:48:40 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 1 (percona1)
120514 16:48:40 [Note] WSREP: STATE EXCHANGE: got state msg: 02c311df-9dd4-11e1-0800-4eb11325e27f from 2 (percona-2)
120514 16:48:40 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 6,
members = 2/3 (joined/total),
act_id = 61951,
last_appl. = 0,
protocols = 0/3/1 (gcs/repl/appl),
group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
120514 16:48:40 [Note] WSREP: Flow-control interval: [14, 28]
120514 16:48:40 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 7: Primary, number of nodes: 3, my index: 1, protocol version 1
120514 16:48:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:48:40 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
120514 16:48:42 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
120514 16:48:42 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 61951)
120514 16:48:42 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
hostname: Name or service not known
hostname: Name or service not known
ERROR 1010 (HY000) at line 451: Error dropping database (can't rmdir './communautaire', errno: 39)
mysqldump: Got errno 32 on write
120514 16:49:42 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
120514 16:49:42 [ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
hostname: Name or service not known
hostname: Name or service not known
120514 16:50:13 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
120514 16:50:13 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.5:4567
120514 16:50:13 [Note] WSREP: view(view_id(PRIM,e039838f-9ab6-11e1-0800-ee89c2714ab0, memb {
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
} joined {
} left {
} partitioned {
02230a2a-9dd4-11e1-0800-37cde4c9bd30,
})
120514 16:50:13 [Note] WSREP: forgetting 02230a2a-9dd4-11e1-0800-37cde4c9bd30 (tcp://10.7.110.5:4567)
120514 16:50:13 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting off
120514 16:50:13 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
120514 16:50:13 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1c7238d2-9dd4-11e1-0800-2cea06775603
120514 16:50:13 [Note] WSREP: STATE EXCHANGE: sent state msg: 1c7238d2-9dd4-11e1-0800-2cea06775603
120514 16:50:13 [Note] WSREP: STATE EXCHANGE: got state msg: 1c7238d2-9dd4-11e1-0800-2cea06775603 from 0 (percona1)
120514 16:50:13 [Note] WSREP: STATE EXCHANGE: got state msg: 1c7238d2-9dd4-11e1-0800-2cea06775603 from 1 (percona-2)
120514 16:50:13 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 7,
members = 2/2 (joined/total),
act_id = 61951,
last_appl. = 0,
protocols = 0/3/1 (gcs/repl/appl),
group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
120514 16:50:13 [Note] WSREP: Flow-control interval: [12, 23]
120514 16:50:18 [Note] WSREP: cleaning up 02230a2a-9dd4-11e1-0800-37cde4c9bd30 (tcp://10.7.110.5:4567)
ERROR 1010 (HY000) at line 451: Error dropping database (can't rmdir './communautaire', errno: 39)
mysqldump: Got errno 32 on write
120514 16:50:37 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
120514 16:50:37 [ERROR] WSREP: Try 2/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
hostname: Name or service not known
hostname: Name or service not known
ERROR 2003 (HY000): Can't connect to MySQL server on '10.7.110.5' (4)
120514 16:50:48 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
120514 16:50:48 [ERROR] WSREP: Try 3/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
120514 16:50:49 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 8: Primary, number of nodes: 2, my index: 0, protocol version 1
120514 16:50:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:50:49 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
120514 16:50:49 [Warning] WSREP: Could not find peer: 02230a2a-9dd4-11e1-0800-37cde4c9bd30
120514 16:50:49 [Warning] WSREP: 0 (percona1): State transfer to -1 (left the group) failed: -1 (Operation not permitted)
120514 16:50:49 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 61951)
120514 16:50:49 [Note] WSREP: Member 0 (percona1) synced with group.
120514 16:50:49 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 61951)
120514 16:50:49 [Note] WSREP: Synchronized with group, ready for connections
120514 16:50:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:51:11 [Note] WSREP: declaring 5c9ad119-9dd4-11e1-0800-dea8fc9be39a stable
120514 16:51:11 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
120514 16:51:11 [Note] WSREP: view(view_id(PRIM,5c9ad119-9dd4-11e1-0800-dea8fc9be39a,9) memb {
5c9ad119-9dd4-11e1-0800-dea8fc9be39a,
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
} joined {
} left {
} partitioned {
})
120514 16:51:11 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
120514 16:51:11 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
120514 16:51:11 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8
120514 16:51:11 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 0 (percona-3)
120514 16:51:11 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 1 (percona1)
120514 16:51:11 [Note] WSREP: STATE EXCHANGE: got state msg: 5ce79cdb-9dd4-11e1-0800-1742273f8cf8 from 2 (percona-2)
120514 16:51:11 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 8,
members = 2/3 (joined/total),
act_id = 61951,
last_appl. = 0,
protocols = 0/3/1 (gcs/repl/appl),
group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
120514 16:51:11 [Note] WSREP: Flow-control interval: [14, 28]
120514 16:51:11 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 9: Primary, number of nodes: 3, my index: 1, protocol version 1
120514 16:51:11 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:51:11 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
120514 16:51:13 [Note] WSREP: Node 0 (percona-3) requested state transfer from '*any*'. Selected 1 (percona1)(SYNCED) as donor.
120514 16:51:13 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 61951)
120514 16:51:13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
hostname: Name or service not known
hostname: Name or service not known
ERROR120514 16:51:34 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.7.110.5:4567
2013 (HY000) at line 365: Lost connection to MySQL server during query
mysqldump: Got errno 32 on write
120514 16:51:34 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
120514 16:51:34 [ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
hostname: Name or service not known
hostname: Name or service not known
ERROR 2003 (HY000): Can't connect to MySQL server on '10.7.110.5' (111)
120514 16:51:35 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
120514 16:51:35 [ERROR] WSREP: Try 2/3: 'wsrep_sst_mysqldump 'sst' 'sstpass123' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
120514 16:51:35 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') reconnecting to 5c9ad119-9dd4-11e1-0800-dea8fc9be39a (tcp://10.7.110.5:4567), attempt 0
hostname: Name or service not known
hostname: Name or service not known
ERROR 2003 (HY000): Can't connect to MySQL server on '10.7.110.5' (111)
120514 16:51:36 [ERROR] WSREP: Process completed with error: wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0': 1 (Operation not permitted)
120514 16:51:36 [ERROR] WSREP: Try 3/3: 'wsrep_sst_mysqldump '***' '********' '10.7.110.5' '3306' '3306' '134487b3-8f74-11e1-0800-14b30cfd2465' '61951' '0'' failed: 1 (Operation not permitted)
120514 16:51:38 [Note] WSREP: evs:roto(e039838f-9ab6-11e1-0800-ee89c2714ab0, GATHER, view_id(REG,5c9ad119-9dd4-11e1-0800-dea8fc9be39a,9)) suspecting node: 5c9ad119-9dd4-11e1-0800-dea8fc9be39a
120514 16:51:38 [Note] WSREP: declaring e137a28e-9ab6-11e1-0800-0427149fd658 stable
120514 16:51:38 [Warning] WSREP: 1 (percona1): State transfer to 0 (percona-3) failed: -1 (Operation not permitted)
120514 16:51:38 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 61951)
120514 16:51:38 [Note] WSREP: view(view_id(PRIM,e039838f-9ab6-11e1-0800-ee89c2714ab0,10) memb {
e039838f-9ab6-11e1-0800-ee89c2714ab0,
e137a28e-9ab6-11e1-0800-0427149fd658,
} joined {
} left {
} partitioned {
5c9ad119-9dd4-11e1-0800-dea8fc9be39a,
})
120514 16:51:38 [Note] WSREP: forgetting 5c9ad119-9dd4-11e1-0800-dea8fc9be39a (tcp://10.7.110.5:4567)
120514 16:51:38 [Note] WSREP: (e039838f-9ab6-11e1-0800-ee89c2714ab0, 'tcp://0.0.0.0:4567') turning message relay requesting off
120514 16:51:38 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
120514 16:51:38 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0
120514 16:51:38 [Warning] WSREP: SYNC message from member 0 in non-primary configuration. Ignored.
120514 16:51:38 [Note] WSREP: STATE EXCHANGE: sent state msg: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0
120514 16:51:39 [Note] WSREP: STATE EXCHANGE: got state msg: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0 from 0 (percona1)
120514 16:51:39 [Note] WSREP: STATE EXCHANGE: got state msg: 4f7c18c2-9dd4-11e1-0800-feb26eca28b0 from 1 (percona-2)
120514 16:51:39 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 9,
members = 2/2 (joined/total),
act_id = 61951,
last_appl. = 0,
protocols = 0/3/1 (gcs/repl/appl),
group UUID = 134487b3-8f74-11e1-0800-14b30cfd2465
120514 16:51:39 [Note] WSREP: Flow-control interval: [12, 23]
120514 16:51:39 [Note] WSREP: New cluster view: global state: 134487b3-8f74-11e1-0800-14b30cfd2465:61951, view# 10: Primary, number of nodes: 2, my index: 0, protocol version 1
120514 16:51:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:51:39 [Note] WSREP: Assign initial position for certification: 61951, protocol version: 2
120514 16:51:39 [Note] WSREP: Member 0 (percona1) synced with group.
120514 16:51:39 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 61951)
120514 16:51:39 [Note] WSREP: Synchronized with group, ready for connections
120514 16:51:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120514 16:51:44 [Note] WSREP: cleaning up 5c9ad119-9dd4-11e1-0800-dea8fc9be39a (tcp://10.7.110.5:4567)

azigui wrote on Tue, 15 May 2012 19:12Who initiated this shutdown?

Also from donor log it follows that 10.7.110.5 has a corrupt data directory. You need to fix that first:
Quote:The easiest way to do this is to use xtrabackup or rsync based SST.

Hello,

Thx a lot for your response.
I have tested with xtrabackup, the firt time is working
But now it doesn't work, so how to configure it

Best regards

Azigui

I resolve it, you must put in my.cnf file in [MYSQLD] section :
datadir=/var/lib/mysql
tmpdir=/tmp
user=mysql

Be carrefull, you can probably obtain a start/stop failed, so you must update password for mysql user debian-syst-main.
This password is written in debian.cnf file