Buy Percona ServicesBuy Now!

Unable to start XtraDB cluster after host failure

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Unable to start XtraDB cluster after host failure

    Unable to start XtraDB cluster after host failure.

    Hi folks,

    After some power issue in DC all hosts with cluster guests were down.
    Now unable to start mysqld service on each node. Here is an output from the syslog

    Dec 22 14:22:16 percona-1.cloud.dimanetz.net systemd[1]: Starting Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap...
    -- Subject: Unit mysql@bootstrap.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman.../systemd-devel
    --
    -- Unit mysql@bootstrap.service has begun starting up.
    Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysql-systemd[12421]: State transfer in progress, setting sleep higher
    Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.359286Z mysqld_safe Logging to '/var/log/mysqld.log'.
    Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.360737Z mysqld_safe Logging to '/var/log/mysqld.log'.
    Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.376960Z mysqld_safe Starting mysqld daemon with databases from /data/mysqldata
    Dec 22 14:22:16 percona-1.cloud.dimanetz.net mysqld_safe[12420]: 2017-12-22T13:22:16.389082Z mysqld_safe WSREP: Running position recovery with --log_error='/data/mysqldata/wsrep_recovery.xB4hbS' --
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12421]: /usr/bin/mysql-systemd: line 140: kill: (12420) - No such process
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12421]: ERROR! mysqld_safe with PID 12420 has already exited: FAILURE
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net systemd[1]: mysql@bootstrap.service: control process exited, code=exited status=1
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12922]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12922]: ERROR! mysql already dead
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net systemd[1]: mysql@bootstrap.service: control process exited, code=exited status=2
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12953]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net mysql-systemd[12953]: WARNING: mysql may be already dead
    Dec 22 14:22:26 percona-1.cloud.dimanetz.net systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
    -- Subject: Unit mysql@bootstrap.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman.../systemd-devel
    --
    -- Unit mysql@bootstrap.service has failed.
    --
    -- The result is failed.

    Tried to bootstrap cluster - got same errors. Any new lines in /var/log/mysqld.log.

    ll /var/run/mysqld/
    total 0

    Same at all 3 nodes.

    # GALERA saved state
    version: 2.1
    uuid: 6a3ccb37-c7ff-11e7-9ce1-16e4a152fdf7
    seqno: -1
    safe_to_bootstrap: 0

Working...
X