GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

update pxc node from 5.6.16 to 5.6.19

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • update pxc node from 5.6.16 to 5.6.19

    on RHEL 6
    Percona-XtraDB-Cluster-server-56-5.6.15-25.5.759.rhel6.x86_64
    (with Percona-XtraDB-Cluster-galera-3-3.5-1.216.rhel6.x86_64)
    is running a 3 node cluster fine

    upgrading one node to
    Percona-XtraDB-Cluster-shared-56-5.6.19-25.6.824.el6.x86_64
    (and to Percona-XtraDB-Cluster-galera-3-3.6-1.3190.rhel6.x86_6)
    breaks it, SST is working but then:


    2014-08-11 16:25:45 32720 [Note] WSREP: Member 2.0 (xdb8071) requested state transfer from 'xdb8069'. Selected 1.0 (xdb8069)(SYNCED) as donor.
    2014-08-11 16:25:45 32720 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 6558)
    2014-08-11 16:25:45 32720 [Note] WSREP: Requesting state transfer: success, donor: 1
    WSREP_SST: [INFO] Proceeding with SST (20140811 16:25:45.992)
    WSREP_SST: [INFO] Cleaning the existing datadir (20140811 16:25:45.995)
    removed `/data1/mysql/data/gvwstate.dat'
    WSREP_SST: [INFO] Cleaning the binlog directory /data/mysql/logs as well (20140811 16:25:46.017)
    rm: cannot remove `/data/mysql/logs/*.index': No such file or directory
    WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140811 16:25:46.024)
    2014-08-11 16:26:26 32720 [Note] WSREP: 1.0 (xdb8069): State transfer to 2.0 (xdb8071) complete.
    2014-08-11 16:26:26 32720 [Note] WSREP: Member 1.0 (xdb8069) synced with group.
    ls: cannot access mysql-binlog-8071.*: No such file or directory
    WSREP_SST: [INFO] Preparing the backup at /data1/mysql/data/ (20140811 16:26:26.548)
    WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20140811 16:26:26.551)
    WSREP_SST: [INFO] Total time on joiner: 0 seconds (20140811 16:26:35.265)
    WSREP_SST: [INFO] Removing the sst_in_progress file (20140811 16:26:35.268)
    2014-08-11 16:26:35 32720 [Note] WSREP: SST complete, seqno: 6558
    2014-08-11 16:26:35 32720 [Note] Plugin 'FEDERATED' is disabled.
    2014-08-11 16:26:35 32720 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2014-08-11 16:26:35 32720 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-08-11 16:26:35 32720 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-08-11 16:26:35 32720 [Note] InnoDB: Compressed tables use zlib 1.2.3
    2014-08-11 16:26:35 32720 [Note] InnoDB: Using Linux native AIO
    2014-08-11 16:26:35 32720 [Note] InnoDB: Using CPU crc32 instructions
    2014-08-11 16:26:35 32720 [Note] InnoDB: Initializing buffer pool, size = 39.1G
    2014-08-11 16:26:37 32720 [Note] InnoDB: Completed initialization of buffer pool
    2014-08-11 16:26:37 32720 [Note] InnoDB: Highest supported file format is Barracuda.
    2014-08-11 16:26:38 32720 [Note] InnoDB: 128 rollback segment(s) are active.
    2014-08-11 16:26:38 32720 [Note] InnoDB: Waiting for purge to start
    2014-08-11 16:26:38 32720 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 3816493669
    2014-08-11 16:26:38 32720 [ERROR] Aborting

    2014-08-11 16:26:40 32720 [Note] WSREP: Closing send monitor...
    2014-08-11 16:26:40 32720 [Note] WSREP: Closed send monitor.
    ...
    2014-08-11 16:26:44 32720 [Note] InnoDB: Shutdown completed; log sequence number 3816496970


    log of the donor node:

    2014-08-11 16:26:26 3587 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    WSREP_SST: [INFO] Total time on donor: 0 seconds (20140811 16:26:26.530)
    WSREP_SST: [INFO] Cleaning up temporary directories (20140811 16:26:26.536)
    2014-08-11 16:26:40 3587 [Note] WSREP: declaring 409fe6e9-0da2-11e4-a8dc-f6c52645ad47 stable
    2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, 'tcp://0.0.0.0:4567') address 'tcp://10.64.218.40:4567' pointing to uuid 5df674bc-0da2-11e4-8264-afe0e5387779 is blacklisted, s
    kipping
    2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, 'tcp://0.0.0.0:4567') address 'tcp://10.64.218.40:4567' pointing to uuid 5df674bc-0da2-11e4-8264-afe0e5387779 is blacklisted, s
    kipping
    2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.64.218.42:4567
    2014-08-11 16:26:40 3587 [Note] WSREP: Node 409fe6e9-0da2-11e4-a8dc-f6c52645ad47 state prim
    2014-08-11 16:26:40 3587 [Note] WSREP: view(view_id(PRIM,409fe6e9-0da2-11e4-a8dc-f6c52645ad47,34) memb {
    409fe6e9-0da2-11e4-a8dc-f6c52645ad47,0
    5df674bc-0da2-11e4-8264-afe0e5387779,0
    } joined {
    } left {
    } partitioned {
    5eb5d970-2163-11e4-ad72-7bd016470afd,0
    })
    2014-08-11 16:26:40 3587 [Note] WSREP: forgetting 5eb5d970-2163-11e4-ad72-7bd016470afd (tcp://10.64.218.42:4567)
    2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, 'tcp://0.0.0.0:4567') address 'tcp://10.64.218.40:4567' pointing to uuid 5df674bc-0da2-11e4-8264-afe0e5387779 is blacklisted, s
    kipping
    2014-08-11 16:26:40 3587 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
    2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
    2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: sent state msg: 82731576-2163-11e4-9a53-87f667275baa
    2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: got state msg: 82731576-2163-11e4-9a53-87f667275baa from 0 (xdb8070)
    2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: got state msg: 82731576-2163-11e4-9a53-87f667275baa from 1 (xdb8069)
    2014-08-11 16:26:40 3587 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 32,
    members = 2/2 (joined/total),
    act_id = 6558,
    last_appl. = 6527,
    protocols = 0/5/2 (gcs/repl/appl),
    group UUID = fd641b3d-0d07-11e4-bb59-bfa063c9be5f
    2014-08-11 16:26:40 3587 [Note] WSREP: Flow-control interval: [23, 23]
    2014-08-11 16:26:40 3587 [Note] WSREP: New cluster view: global state: fd641b3d-0d07-11e4-bb59-bfa063c9be5f:6558, view# 33: Primary, number of nodes: 2, my index: 1, protocol version 2
    2014-08-11 16:26:40 3587 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2014-08-11 16:26:40 3587 [Note] WSREP: REPL Protocols: 5 (3, 1)
    2014-08-11 16:26:40 3587 [Note] WSREP: Service thread queue flushed.
    2014-08-11 16:26:40 3587 [Note] WSREP: Assign initial position for certification: 6558, protocol version: 3
    2014-08-11 16:26:40 3587 [Note] WSREP: Service thread queue flushed.
    2014-08-11 16:26:40 3587 [Warning] WSREP: Releasing seqno 6558 before 6559 was assigned.
    2014-08-11 16:26:45 3587 [Note] WSREP: cleaning up 5eb5d970-2163-11e4-ad72-7bd016470afd (tcp://10.64.218.42:4567)

    yum downgrade to 5.6.15. and node is up again.

    I tried:
    rm -rf in the datadir to force an SST, didn't help.

    Have I done it wrong? Is there a need to upgrade galera first on all node or something like this, or is 5.6.19 broken?

  • #2
    Hello
    I have set up a second Cluster on different maschines, 5.6.19 installed.
    I copied a backup (made with xtrabackup) from the 5.6.15 Cluster and tried to bootstrap that copy

    -------------error.log
    140814 17:42:41 mysqld_safe Starting mysqld daemon with databases from /data1/mysql/data
    140814 17:42:41 mysqld_safe Skipping wsrep-recover for empty datadir: /data1/mysql/data
    140814 17:42:41 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
    2014-08-14 17:42:42 0 [Warning] Using unique option prefix character_set_client instead of character-set-client-handshake is deprecated and will be removed in a future release. Pl
    ease use the full name instead.
    2014-08-14 17:42:42 0 [Warning] /usr/sbin/mysqld: ignoring option '--character-set-client-handshake' due to invalid value 'utf8'
    2014-08-14 17:42:42 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
    2014-08-14 17:42:42 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details
    ).
    2014-08-14 17:42:42 24446 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
    2014-08-14 17:42:42 24446 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2014-08-14 17:42:42 24446 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
    2014-08-14 17:42:42 24446 [Note] WSREP: wsrep_load(): Galera 3.6(r3a949e6) by Codership Oy <info@codership.com> loaded successfully.
    2014-08-14 17:42:42 24446 [Note] WSREP: CRC-32C: using hardware acceleration.
    2014-08-14 17:42:42 24446 [Warning] WSREP: Could not open saved state file for reading: /data1/mysql/data//grastate.dat
    2014-08-14 17:42:42 24446 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
    2014-08-14 17:42:42 24446 [Note] WSREP: Passing config to GCS: base_host = 10.64.224.70; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S;
    evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.u
    ser_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data1/mysql/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data1/mysql/data//galera.cac
    he; gcache.page_size = 128M; gcache.size = 256M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle
    = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.chec
    ksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S;
    2014-08-14 17:42:42 24446 [Note] WSREP: Service thread queue flushed.
    2014-08-14 17:42:42 24446 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
    2014-08-14 17:42:42 24446 [Note] WSREP: wsrep_sst_grab()
    2014-08-14 17:42:42 24446 [Note] WSREP: Start replication
    2014-08-14 17:42:42 24446 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
    2014-08-14 17:42:42 24446 [Note] WSREP: protonet asio version 0
    2014-08-14 17:42:42 24446 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
    2014-08-14 17:42:42 24446 [Note] WSREP: backend: asio
    2014-08-14 17:42:42 24446 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)
    2014-08-14 17:42:42 24446 [Note] WSREP: restore pc from disk failed
    2014-08-14 17:42:42 24446 [Note] WSREP: GMCast version 0
    2014-08-14 17:42:42 24446 [Note] WSREP: (a0d899d6, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2014-08-14 17:42:42 24446 [Note] WSREP: (a0d899d6, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2014-08-14 17:42:42 24446 [Note] WSREP: EVS version 0
    2014-08-14 17:42:42 24446 [Note] WSREP: PC version 0
    2014-08-14 17:42:42 24446 [Note] WSREP: gcomm: connecting to group 'stage_sp1_xdb_1', peer ''
    2014-08-14 17:42:42 24446 [Note] WSREP: start_prim is enabled, turn off pc_recovery
    2014-08-14 17:42:42 24446 [Note] WSREP: Node a0d899d6 state prim
    2014-08-14 17:42:42 24446 [Note] WSREP: save pc into disk
    2014-08-14 17:42:42 24446 [Note] WSREP: gcomm: connected
    2014-08-14 17:42:42 24446 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2014-08-14 17:42:42 24446 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2014-08-14 17:42:42 24446 [Note] WSREP: Opened channel 'stage_sp1_xdb_1'
    2014-08-14 17:42:42 24446 [Note] WSREP: Waiting for SST to complete.
    2014-08-14 17:42:42 24446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
    2014-08-14 17:42:42 24446 [Note] WSREP: Starting new group from scratch: a0d977c2-23c9-11e4-8b41-b707239bd1c3
    2014-08-14 17:42:42 24446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a0d989bb-23c9-11e4-8481-d33e3e29b875
    2014-08-14 17:42:42 24446 [Note] WSREP: STATE EXCHANGE: sent state msg: a0d989bb-23c9-11e4-8481-d33e3e29b875
    2014-08-14 17:42:42 24446 [Note] WSREP: STATE EXCHANGE: got state msg: a0d989bb-23c9-11e4-8481-d33e3e29b875 from 0 (xdb8284)
    2014-08-14 17:42:42 24446 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 0,
    members = 1/1 (joined/total),
    act_id = 0,
    last_appl. = -1,
    protocols = 0/6/3 (gcs/repl/appl),
    group UUID = a0d977c2-23c9-11e4-8b41-b707239bd1c3
    2014-08-14 17:42:42 24446 [Note] WSREP: Flow-control interval: [16, 16]
    2014-08-14 17:42:42 24446 [Note] WSREP: Restored state OPEN -> JOINED (0)
    2014-08-14 17:42:42 24446 [Note] WSREP: Member 0.0 (xdb8284) synced with group.
    2014-08-14 17:42:42 24446 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
    2014-08-14 17:42:42 24446 [Note] WSREP: New cluster view: global state: a0d977c2-23c9-11e4-8b41-b707239bd1c3:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
    2014-08-14 17:42:42 24446 [Note] WSREP: SST complete, seqno: 0
    2014-08-14 17:42:42 24446 [Note] Plugin 'FEDERATED' is disabled.
    2014-08-14 17:42:42 24446 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2014-08-14 17:42:42 24446 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-08-14 17:42:42 24446 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-08-14 17:42:42 24446 [Note] InnoDB: Compressed tables use zlib 1.2.3
    2014-08-14 17:42:42 24446 [Note] InnoDB: Using Linux native AIO
    2014-08-14 17:42:42 24446 [Note] InnoDB: Using CPU crc32 instructions
    2014-08-14 17:42:42 24446 [Note] InnoDB: Initializing buffer pool, size = 7.8G
    2014-08-14 17:42:42 24446 [Note] InnoDB: Completed initialization of buffer pool
    2014-08-14 17:42:42 24446 [Note] InnoDB: Highest supported file format is Barracuda.
    2014-08-14 17:42:43 24446 [Note] InnoDB: 128 rollback segment(s) are active.
    2014-08-14 17:42:43 24446 [Note] InnoDB: Waiting for purge to start
    2014-08-14 17:42:43 24446 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 6281825897
    2014-08-14 17:42:43 24446 [ERROR] Aborting

    yum downgrade Percona-XtraDB-Cluster-client-56-5.6.15 Percona-XtraDB-Cluster-galera-3-3.5 Percona-XtraDB-Cluster-shared-56-5.6.15 Percona-XtraDB-Cluster-server-56-5.6.15

    /etc/init.d/mysql start
    ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
    Starting MySQL (Percona XtraDB Cluster)..... SUCCESS!

    I didn't searched for the changelog yet, did I miss something about 5.6.19?

    I first tried mysql_install_db, but this got this also:
    2014-08-14 17:42:43 24446 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 6281825897
    2014-08-14 17:42:43 24446 [ERROR] Aborting

    Seems like a bug to me, but as this version is out for weeks now, I wonder I can't find anything about it.

    Comment


    • #3
      Code:
      2014-08-14 17:42:42 24446 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
      is the key here. Removing hardcoded lc-messages-dir path from my.cnf will help.

      Comment

      Working...
      X