GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

Cluster failed, can you provide any insight?

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Cluster failed, can you provide any insight?

    We had an influx of connections/queries , ~2000 in a matter of 5 seconds, which resulted in one of the servers dropping out of the cluster 10.19.0.44.

    We are currently running version: 5.5.31-23.7.5.438.rhel6.x86_64
    I know there is a new version out but after reading over the change logs I only saw a couple changes in 5.5.34-23.7.6 that could be semi related.

    Attempting to check the status of mysql resulted in the following error on all nodes still online resulted with the following message:

    mysql> show global status;
    ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

    I have the wsrep_cluster_address set as follows:
    wsrep_cluster_address=gcomm://10.19.0.40,10.19.0.41,10.19.0.42,10.19.0.43,10.19. 0.44,10.19.0.45

    One possibly unrelated configuration question I have is, should I not include the servers own IP in the wsrep_cluster_address variable, I looked all over but could not find any documentation to allow me to come to a conclusion either way. During testing, I found that it would detect that it was itself, and then would skip that node, however is this bad form, should you include the the servers own IP in the cluster address or not?


    At 15:51:45 the issue began, at 16:05:37 I reinitialized the cluster using db6. I stopped mysql and restarted with it using the following command:
    /etc/init.d/mysql start --wsrep_cluster_address=gcomm://


    At this point db6 was the primary, db4 was offline, db1,db2,db5 all joined db6

    I had to stop db3 and restart db3 and db4 which both performed an SST from db6, and everything was back up and running normally



    What I am looking to figure out is what cause the servers to no longer work and result in a Lock wait timeout exceeded when checking the status.

    There are no log entries up to the point where the influx of connections occurred
    The logs presented are from the server 10.19.0.42

  • #2
    Code:
    131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    
    ...
    
    131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.43:4567 tcp://10.19.0.44:4567 tcp://10.19.0.45:4567
    131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    
    ...
    
    131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    131107 15:51:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567), attempt 0
    131107 15:51:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
    131107 15:51:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
    131107 15:51:47 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.85633S), skipping check
    131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    
    ...
    
    131107 15:51:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
    131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    
    ...
    
    131107 15:51:52 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    131107 15:51:52 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
    131107 15:51:52 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
    131107 15:51:52 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
    131107 15:51:52 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
    131107 15:51:52 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
    131107 15:51:53 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
    131107 15:51:53 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,264) memb {
    27013c03-3740-11e3-aec6-4a0d195491f5,
    644c6f70-4716-11e3-9900-c2dfa5a8512f,
    a21cd90d-373e-11e3-b146-560f037992a6,
    c089c14e-3738-11e3-877c-8aa6e988bded,
    } joined {
    } left {
    } partitioned {
    b5387adf-4564-11e3-9ffe-f38d17a91a12,
    b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
    })
    131107 15:51:53 [Note] WSREP: forgetting b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567)
    131107 15:51:53 [Note] WSREP: deleting entry tcp://10.19.0.44:4567
    131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    
    ...
    
    131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    131107 15:51:53 [Note] WSREP: forgetting b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
    131107 15:51:53 [Note] WSREP: deleting entry tcp://10.19.0.43:4567
    131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
    
    ...

    Comment


    • #3
      Code:
      131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
      131107 15:51:53 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 4
      131107 15:51:53 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      131107 15:51:53 [Note] WSREP: STATE EXCHANGE: sent state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771
      131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 0 db1
      131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 3 db4
      131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 1 db5
      131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 2 db2
      131107 15:51:53 [Note] WSREP: Quorum results:
      version = 2,
      component = PRIMARY,
      conf_id = 232,
      members = 4/4 (joined/total),
      act_id = 8175717,
      last_appl. = 8175401,
      protocols = 0/4/2 (gcs/repl/appl),
      group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
      131107 15:51:53 [Note] WSREP: Flow-control interval: [32, 32]
      131107 15:51:53 [Warning] WSREP: gcs_caused() returned -1 (Operation not permitted)
      
      ...
      
      131107 15:51:53 [Warning] WSREP: gcs_caused() returned -1 (Operation not permitted)
      131107 15:51:53 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175717, view# 233: Primary, number of nodes: 4, my index: 2, protocol version 2
      131107 15:51:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      131107 15:51:53 [Note] WSREP: Assign initial position for certification: 8175717, protocol version: 2
      131107 15:51:59 [Note] WSREP: cleaning up b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
      131107 15:51:59 [Note] WSREP: cleaning up b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567)
      131107 15:51:53 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175717, view# 233: Primary, number of nodes: 4, my index: 2, protocol version 2
      131107 15:51:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      131107 15:51:53 [Note] WSREP: Assign initial position for certification: 8175717, protocol version: 2
      131107 15:51:59 [Note] WSREP:  cleaning up b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
      131107 15:51:59 [Note] WSREP:  cleaning up b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567)
      131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.45:4567
      131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
      131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
      131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.43:4567
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      
      ...
      
      131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
      131107 15:52:27 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
      131107 15:52:27 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
      131107 15:52:27 [Note] WSREP: declaring b5387adf-4564-11e3-9ffe-f38d17a91a12 stable
      131107 15:52:27 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
      131107 15:52:28 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
      131107 15:52:29 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
      131107 15:52:29 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
      131107 15:52:29 [Note] WSREP: declaring b5387adf-4564-11e3-9ffe-f38d17a91a12 stable
      131107 15:52:29 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
      131107 15:52:29 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
      131107 15:52:30 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
      131107 15:52:30 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266) memb {
              27013c03-3740-11e3-aec6-4a0d195491f5,
              644c6f70-4716-11e3-9900-c2dfa5a8512f,
              a21cd90d-373e-11e3-b146-560f037992a6,
              b5387adf-4564-11e3-9ffe-f38d17a91a12,
              b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
              c089c14e-3738-11e3-877c-8aa6e988bded,
      } joined {
      } left {
      } partitioned {
      })
      131107 15:52:30 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 6
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: sent state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 0 (db1)
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 1 (db5)
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 2 (db2)
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 5 (db6)
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 3 (db4)
      131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 4 (db3)
      131107 15:52:30 [Note] WSREP: Quorum results:
              version    = 2,
              component  = PRIMARY,
              conf_id    = 233,
              members    = 4/6 (joined/total),
              act_id     = 8175755,
              last_appl. = 8175401,
              protocols  = 0/4/2 (gcs/repl/appl),
              group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b

      Comment


      • #4
        Code:
        131107 15:52:30 [Note] WSREP: Flow-control interval: [39, 39]
        131107 15:52:30 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175755, view# 234: Primary, number of nodes: 6, my index: 2, protocol version 2
        131107 15:52:30 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
        131107 15:52:30 [Note] WSREP: Assign initial position for certification: 8175755, protocol version: 2
        131107 15:52:33 [Note] WSREP: Node 3 (db4) requested state transfer from 'db6'. Selected 5 (db6)(SYNCED) as donor.
        131107 15:52:35 [Note] WSREP: 5 (db6): State transfer to 3 (db4) complete.
        131107 15:52:35 [Note] WSREP: Member 5 (db6) synced with group.
        131107 15:52:35 [Note] WSREP: 3 (db4): State transfer from 5 (db6) complete.
        131107 15:52:35 [Note] WSREP: Member 3 (db4) synced with group.
        131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        
        ...
        
        131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.43:4567
        131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        
        ...
        
        131107 15:53:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567), attempt 0
        131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
        131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        
        ...
        
        131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:04 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:04 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
        131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
        131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
        131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
        131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
        131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
        131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        
        ...
        
        131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
        131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
        131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
        131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        
        ...
        
        131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        131107 15:53:09 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
        131107 15:53:09 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
        131107 15:53:09 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
        131107 15:53:10 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
        
        ...

        Comment


        • #5
          Code:
          131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
          131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
          131107 15:53:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:15 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,267)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
          131107 15:53:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          
          ...
          
          131107 15:53:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:15 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,267)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
          131107 15:53:16 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          
          ...
          
          131107 15:53:16 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
          131107 15:53:16 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,267) memb {
                  27013c03-3740-11e3-aec6-4a0d195491f5,
                  a21cd90d-373e-11e3-b146-560f037992a6,
                  c089c14e-3738-11e3-877c-8aa6e988bded,
          } joined {
          } left {
          } partitioned {
                  644c6f70-4716-11e3-9900-c2dfa5a8512f,
                  b5387adf-4564-11e3-9ffe-f38d17a91a12,
                  b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
          })
          131107 15:53:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3
          131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
          }} in state TRANS
          131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
          }} in state TRANS
          131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
          }} in state TRANS
          131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
                  b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
                  c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
          }} in state TRANS
          131107 15:53:16 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
          131107 15:53:16 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
          131107 15:53:16 [Note] WSREP: Flow-control interval: [28, 28]
          131107 15:53:16 [Note] WSREP: Received NON-PRIMARY.
          131107 15:53:16 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8175767)
          131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
          131107 15:53:16 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version 2
          131107 15:53:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
          131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
          
          ...
          
          131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
          131107 15:53:16 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,270) memb {
                  27013c03-3740-11e3-aec6-4a0d195491f5,
                  a21cd90d-373e-11e3-b146-560f037992a6,
                  c089c14e-3738-11e3-877c-8aa6e988bded,
          } joined {
          } left {
          } partitioned {
                  644c6f70-4716-11e3-9900-c2dfa5a8512f,
                  b5387adf-4564-11e3-9ffe-f38d17a91a12,
                  b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
          })

          Comment


          • #6
            Code:
            131107 15:53:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3
            131107 15:53:16 [Note] WSREP: Flow-control interval: [28, 28]
            131107 15:53:16 [Note] WSREP: Received NON-PRIMARY.
            131107 15:53:16 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version 2
            131107 15:53:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            131107 15:53:16 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
            131107 15:53:16 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
            131107 15:53:16 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
            131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:18 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,271) memb {
                    27013c03-3740-11e3-aec6-4a0d195491f5,
                    644c6f70-4716-11e3-9900-c2dfa5a8512f,
                    a21cd90d-373e-11e3-b146-560f037992a6,
                    c089c14e-3738-11e3-877c-8aa6e988bded,
            } joined {
            } left {
            } partitioned {
                    b5387adf-4564-11e3-9ffe-f38d17a91a12,
                    b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
            })
            131107 15:53:18 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 4
            131107 15:53:18 [Note] WSREP: Flow-control interval: [32, 32]
            131107 15:53:18 [Note] WSREP: Received NON-PRIMARY.
            131107 15:53:18 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 4, my index: 2, protocol version 2
            131107 15:53:18 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            131107 15:53:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
            131107 15:53:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:22 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:22 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
            131107 15:53:22 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
            131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
            131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
            131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:27 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
            131107 15:53:27 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
            131107 15:53:27 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
            131107 15:53:27 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
            131107 15:53:27 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,272) memb {
                    27013c03-3740-11e3-aec6-4a0d195491f5,
                    644c6f70-4716-11e3-9900-c2dfa5a8512f,
                    a21cd90d-373e-11e3-b146-560f037992a6,
                    b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
                    c089c14e-3738-11e3-877c-8aa6e988bded,
            } joined {
            } left {
            } partitioned {
                    b5387adf-4564-11e3-9ffe-f38d17a91a12,
            })
            131107 15:53:27 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 5
            131107 15:53:27 [Note] WSREP: Flow-control interval: [36, 36]
            131107 15:53:27 [Note] WSREP: Received NON-PRIMARY.
            131107 15:53:27 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 5, my index: 2, protocol version 2
            131107 15:53:27 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            131107 15:53:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
            131107 15:53:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:29 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:51 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:52 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...
            
            131107 15:53:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            131107 15:53:56 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
            131107 15:53:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
            
            ...

            Comment


            • #7
              Code:
              131107 15:54:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:03 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:06 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 30
              131107 15:54:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:13 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:21 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:25 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:35 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:36 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:47 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:57 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 60
              131107 15:54:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:54:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:54:58 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:54:59 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:02 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:09 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:09 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:12 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:17 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:20 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:20 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:27 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:30 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:32 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...
              
              131107 15:55:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              131107 15:55:32 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
              131107 15:55:33 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
              
              ...

              Comment


              • #8
                Code:
                131107 15:55:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:55:42 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:55:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 90
                
                ...
                
                131107 15:55:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:55:42 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:55:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:55:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:55:53 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:55:54 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:04 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:15 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:18 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:23 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:26 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 120
                131107 15:56:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:35 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:36 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:37 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:49 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:51 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:54 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:56:56 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:56:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:56:59 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:57:00 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:57:00 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:57:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:57:06 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:57:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:57:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:57:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:57:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 150
                131107 15:57:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...
                
                131107 15:57:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                131107 15:57:22 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                131107 15:57:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                
                ...

                Comment


                • #9
                  Code:
                  131107 15:57:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:57:33 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:57:33 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:57:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:57:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:57:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:57:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:57:44 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:57:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:57:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:57:46 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:57:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:57:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:57:55 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:57:55 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:57:55 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:57:56 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:57:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 180
                  131107 15:57:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:01 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:05 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:15 [Note] WSREP: remote endpoint tcp://10.19.0.44:4567 changed identity b5387adf-4564-11e3-9ffe-f38d17a91a12 -> 526e2b4e-47ef-11e3-8c28-d21f3034d100
                  131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
                  131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:16 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
                  131107 15:58:16 [Note] WSREP: declaring 526e2b4e-47ef-11e3-8c28-d21f3034d100 stable
                  131107 15:58:16 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
                  131107 15:58:16 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
                  131107 15:58:16 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
                  131107 15:58:16 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,273) memb {
                          27013c03-3740-11e3-aec6-4a0d195491f5,
                          526e2b4e-47ef-11e3-8c28-d21f3034d100,
                          644c6f70-4716-11e3-9900-c2dfa5a8512f,
                          a21cd90d-373e-11e3-b146-560f037992a6,
                          b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
                          c089c14e-3738-11e3-877c-8aa6e988bded,
                  } joined {
                  } left {
                  } partitioned {
                          b5387adf-4564-11e3-9ffe-f38d17a91a12,
                  })
                  131107 15:58:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 3, memb_num = 6
                  131107 15:58:16 [Note] WSREP: Flow-control interval: [39, 39]
                  131107 15:58:16 [Note] WSREP: Received NON-PRIMARY.
                  131107 15:58:16 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 6, my index: 3, protocol version 2
                  131107 15:58:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
                  131107 15:58:17 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:26 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:38 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                  131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...
                  
                  131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
                  131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                  
                  ...

                  Comment


                  • #10
                    Code:
                    131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:58:46 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
                    131107 15:58:46 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
                    131107 15:58:46 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
                    131107 15:58:46 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
                    131107 15:58:46 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,274) memb {
                            27013c03-3740-11e3-aec6-4a0d195491f5,
                            644c6f70-4716-11e3-9900-c2dfa5a8512f,
                            a21cd90d-373e-11e3-b146-560f037992a6,
                            b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
                            c089c14e-3738-11e3-877c-8aa6e988bded,
                    } joined {
                    } left {
                    } partitioned {
                            526e2b4e-47ef-11e3-8c28-d21f3034d100,
                            b5387adf-4564-11e3-9ffe-f38d17a91a12,
                    })
                    131107 15:58:46 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 5
                    131107 15:58:46 [Note] WSREP: Flow-control interval: [36, 36]
                    131107 15:58:46 [Note] WSREP: Received NON-PRIMARY.
                    131107 15:58:46 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 5, my index: 2, protocol version 2
                    131107 15:58:46 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
                    131107 15:58:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567), attempt 0
                    131107 15:58:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:58:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:58:50 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:58:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:00 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:00 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:02 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:07 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:17 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:22 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:22 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:30 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567), attempt 30
                    131107 15:59:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:32 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:33 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:33 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:37 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:44 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:47 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 15:59:54 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 15:59:55 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 15:59:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 16:00:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 16:00:05 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                    131107 16:00:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    
                    ...
                    
                    131107 16:00:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                    131107 16:00:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)

                    Comment


                    • #11
                      Code:
                      131107 16:00:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567), attempt 60
                      131107 16:00:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:23 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:29 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:38 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:38 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:43 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:49 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:53 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      131107 16:00:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:00:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:00:58 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
                      
                      131107 16:05:37 [Note] WSREP: remote endpoint tcp://10.19.0.40:4567 changed identity c089c14e-3738-11e3-877c-8aa6e988bded -> 59fea0f7-47f0-11e3-b16f-2b76a5181294
                      131107 16:05:37 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:05:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:05:39 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
                      131107 16:05:39 [Note] WSREP: declaring 59fea0f7-47f0-11e3-b16f-2b76a5181294 stable
                      131107 16:05:39 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
                      131107 16:05:39 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
                      131107 16:05:39 [Note] WSREP: Node 59fea0f7-47f0-11e3-b16f-2b76a5181294 state prim
                      131107 16:05:39 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,277) memb {
                              27013c03-3740-11e3-aec6-4a0d195491f5,
                              59fea0f7-47f0-11e3-b16f-2b76a5181294,
                              644c6f70-4716-11e3-9900-c2dfa5a8512f,
                              a21cd90d-373e-11e3-b146-560f037992a6,
                              b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
                      } joined {
                      } left {
                      } partitioned {
                              526e2b4e-47ef-11e3-8c28-d21f3034d100,
                              b5387adf-4564-11e3-9ffe-f38d17a91a12,
                              c089c14e-3738-11e3-877c-8aa6e988bded,
                      })
                      131107 16:05:39 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 5
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
                      131107 16:05:39 [Note] WSREP: forgetting 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567)
                      131107 16:05:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...
                      
                      131107 16:05:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      131107 16:05:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: sent state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 0 (db1)
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 1 (db6)
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 2 (db5)
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 3 (db2)
                      131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 4 (db3)
                      131107 16:05:39 [Note] WSREP: Quorum results:
                              version    = 2,
                              component  = PRIMARY,
                              conf_id    = 1,
                              members    = 4/5 (joined/total),
                              act_id     = 8175767,
                              last_appl. = 8175401,
                              protocols  = 0/4/2 (gcs/repl/appl),
                              group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
                      131107 16:05:39 [Note] WSREP: Flow-control interval: [36, 36]
                      131107 16:05:39 [Note] WSREP: Restored state OPEN -> SYNCED (8175767)
                      131107 16:05:39 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# 2: Primary, number of nodes: 5, my index: 3, protocol version 2
                      131107 16:05:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
                      131107 16:05:39 [Note] WSREP: Assign initial position for certification: 8175767, protocol version: 2
                      131107 16:05:39 [Note] WSREP: Synchronized with group, ready for connections
                      131107 16:05:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
                      131107 16:05:39 [Note] WSREP: Node 4 (db3) requested state transfer from 'db6'. Selected 1 (db6)(SYNCED) as donor.
                      131107 16:05:40 [Warning] WSREP: 1 (db6): State transfer to 4 (db3) failed: -125 (Operation canceled)
                      131107 16:05:40 [Note] WSREP: Member 1 (db6) synced with group.
                      131107 16:05:40 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
                      131107 16:05:40 [Note] WSREP: declaring 59fea0f7-47f0-11e3-b16f-2b76a5181294 stable
                      131107 16:05:40 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
                      131107 16:05:40 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
                      131107 16:05:40 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,278) memb {
                              27013c03-3740-11e3-aec6-4a0d195491f5,
                              59fea0f7-47f0-11e3-b16f-2b76a5181294,
                              644c6f70-4716-11e3-9900-c2dfa5a8512f,
                              a21cd90d-373e-11e3-b146-560f037992a6,
                      } joined {
                      } left {
                      } partitioned {
                              b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
                      })
                      131107 16:05:40 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 4
                      131107 16:05:40 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
                      131107 16:05:40 [Note] WSREP: forgetting 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567)
                      131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                      
                      ...

                      Comment


                      • #12
                        Code:
                        131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        131107 16:05:40 [Note] WSREP: forgetting b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
                        131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        
                        ...
                        
                        131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        131107 16:05:40 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359
                        131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 0 (db1)
                        131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 1 (db6)
                        131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 2 (db5)
                        131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 3 (db2)
                        131107 16:05:40 [Note] WSREP: Quorum results:
                                version    = 2,
                                component  = PRIMARY,
                                conf_id    = 2,
                                members    = 4/4 (joined/total),
                                act_id     = 8175767,
                                last_appl. = 0,
                                protocols  = 0/4/2 (gcs/repl/appl),
                                group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
                        131107 16:05:40 [Note] WSREP: Flow-control interval: [32, 32]
                        131107 16:05:40 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# 3: Primary, number of nodes: 4, my index: 3, protocol version 2
                        131107 16:05:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
                        131107 16:05:40 [Note] WSREP: Assign initial position for certification: 8175767, protocol version: 2
                        131107 16:05:45 [Note] WSREP:  cleaning up b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
                        131107 16:05:45 [Note] WSREP:  cleaning up 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567)
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        
                        ...
                        
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        
                        ...
                        
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
                        131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
                        
                        ...
                         
                        131107 16:17:08 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
                        131107 16:17:08 [Note] WSREP: declaring 59fea0f7-47f0-11e3-b16f-2b76a5181294 stable
                        131107 16:17:08 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
                        131107 16:17:08 [Note] WSREP: declaring f5b78e53-47f1-11e3-b8c6-6f4c302349ca stable
                        131107 16:17:08 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
                        131107 16:17:08 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,279) memb {
                                27013c03-3740-11e3-aec6-4a0d195491f5,
                                59fea0f7-47f0-11e3-b16f-2b76a5181294,
                                644c6f70-4716-11e3-9900-c2dfa5a8512f,
                                a21cd90d-373e-11e3-b146-560f037992a6,
                                f5b78e53-47f1-11e3-b8c6-6f4c302349ca,
                        } joined {
                        } left {
                        } partitioned {
                        })
                        131107 16:17:08 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 5
                        131107 16:17:08 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
                        131107 16:17:08 [Note] WSREP: STATE EXCHANGE: sent state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d
                        131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 0 (db1)
                        131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 1 (db6)
                        131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 2 (db5)
                        131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 3 (db2)
                        131107 16:17:09 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 4 (db4)
                        131107 16:17:09 [Note] WSREP: Quorum results:
                                version    = 2,
                                component  = PRIMARY,
                                conf_id    = 3,
                                members    = 4/5 (joined/total),
                                act_id     = 8176745,
                                last_appl. = 0,
                                protocols  = 0/4/2 (gcs/repl/appl),
                                group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
                        131107 16:17:09 [Note] WSREP: Flow-control interval: [36, 36]
                        131107 16:17:09 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8176745, view# 4: Primary, number of nodes: 5, my index: 3, protocol version 2
                        131107 16:17:09 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
                        131107 16:17:09 [Note] WSREP: Assign initial position for certification: 8176745, protocol version: 2
                        131107 16:17:11 [Note] WSREP: Node 4 (db4) requested state transfer from 'db6'. Selected 1 (db6)(SYNCED) as donor.
                        131107 16:19:06 [Note] WSREP: 1 (db6): State transfer to 4 (db4) complete.
                        131107 16:19:06 [Note] WSREP: 4 (db6): State transfer from 1 (db6) complete.
                        131107 16:19:06 [Note] WSREP: Member 1 (db6) synced with group.
                        131107 16:19:20 [Note] WSREP: Member 4 (db4) synced with group.

                        Comment

                        Working...
                        X