Emergency

Can't start Percona Cluster (mysql) after "self-leave"

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Can't start Percona Cluster (mysql) after "self-leave"

    Hello,

    Doesn't matter whatever I do I can't start Percona Node 2 in my 2 node cluster.... (yes, I know that I should not run 2 nodes in a cluster but that's what it is for now)
    RHEL7, latest Percona 5.7.

    Your time and help are much appropriated. This is a UAT setup but if all goes well it will go into PROD...

    This setup was running fine until one day it did that

    Code:
    2017-11-17T11:55:13.044889Z 4 [Note] WSREP: Current view of cluster as seen by this node
    view (view_id(NON_PRIM,5fc8c4bc,4)
    memb {
            8567ecca,0
            }
    joined {
            }
    left {
            }
    partitioned {
            5fc8c4bc,0
            }
    )
    2017-11-17T11:55:13.044957Z 4 [Note] WSREP: Current view of cluster as seen by this node
    view ((empty))
    2017-11-17T11:55:13.045119Z 4 [Note] WSREP: gcomm: closed
    2017-11-17T11:55:13.045186Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2017-11-17T11:55:13.045220Z 0 [Note] WSREP: Flow-control interval: [100, 100]
    2017-11-17T11:55:13.045225Z 0 [Note] WSREP: Trying to continue unpaused monitor
    2017-11-17T11:55:13.045227Z 0 [Note] WSREP: Received NON-PRIMARY.
    2017-11-17T11:55:13.045230Z 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8725)
    2017-11-17T11:55:13.045250Z 0 [Note] WSREP: Received self-leave message.
    2017-11-17T11:55:13.045254Z 0 [Note] WSREP: Flow-control interval: [0, 0]
    2017-11-17T11:55:13.045257Z 0 [Note] WSREP: Trying to continue unpaused monitor
    2017-11-17T11:55:13.045259Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
    2017-11-17T11:55:13.045261Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 8725)
    2017-11-17T11:55:13.045284Z 0 [Note] WSREP: RECV thread exiting 0: Success
    2017-11-17T11:55:13.045345Z 4 [Note] WSREP: recv_thread() joined.
    2017-11-17T11:55:13.045353Z 4 [Note] WSREP: Closing replication queue.
    2017-11-17T11:55:13.045357Z 4 [Note] WSREP: Closing slave action queue.
    2017-11-17T11:55:13.045756Z 4 [Note] WSREP: /usr/sbin/mysqld: Terminated.


    Nothing gets logged to the /var/log/mysql.log

    Here is my error message:

    Code:
    -- Unit mysql.service has begun starting up.
    Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.619589Z mysqld_safe Logging to '/var/log/mysqld-safe.log'.
    Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.623103Z mysqld_safe Logging to '/var/log/mysqld-safe.log'.
    Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.649402Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.665308Z mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.pZnH0L' --pid-file='/var/lib/mysql/server-02-recover.pid'
    Nov 24 13:38:31 S-02 mysql-systemd[126040]: /usr/bin/mysql-systemd: line 140: kill: (126039) - No such process
    Nov 24 13:38:31 S-02 mysql-systemd[126040]: ERROR! mysqld_safe with PID 126039 has already exited: FAILURE
    Nov 24 13:38:31 S-02 systemd[1]: mysql.service: control process exited, code=exited status=1
    Nov 24 13:38:31 S-02 mysql-systemd[126701]: WARNING: mysql pid file /var/lib/mysql/mysqld.pid empty or not readable
    Nov 24 13:38:31 S-02 mysql-systemd[126701]: ERROR! mysql already dead
    Nov 24 13:38:31 S-02 systemd[1]: mysql.service: control process exited, code=exited status=2
    Nov 24 13:38:31 S-02 mysql-systemd[126732]: WARNING: mysql pid file /var/lib/mysql/mysqld.pid empty or not readable
    Nov 24 13:38:31 S-02 mysql-systemd[126732]: WARNING: mysql may be already dead
    Nov 24 13:38:31 S-02 systemd[1]: Failed to start Percona XtraDB Cluster.
    -- Subject: Unit mysql.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has failed.
    --
    -- The result is failed.
    Nov 24 13:38:31 S-02 systemd[1]: Unit mysql.service entered failed state.
    Nov 24 13:38:31 S-02 systemd[1]: mysql.service failed.
    Nov 24 13:38:31 S-02 polkitd[622]: Unregistered Authentication Agent for unix-process:125992:169691973 (system bus name :1.22171, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)


    And that's what in wsrep_recovery:

    Code:
    2017-11-24T13:38:27.847539Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2017-11-24T13:38:27.849040Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-17-57-log) starting as process 126662 ...
    2017-11-24T13:38:27.852025Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=S-02-bin' to avoid this problem.
    2017-11-24T13:38:27.856882Z 0 [Note] InnoDB: PUNCH HOLE support available
    [...CUT...]
    2017-11-24T13:38:28.523347Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.19-17 started; log sequence number 126001947
    2017-11-24T13:38:28.523365Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
    2017-11-24T13:38:28.523658Z 0 [Note] Plugin 'FEDERATED' is disabled.
    2017-11-24T13:38:28.549281Z 0 [Note] WSREP: Recovered position: 601f3fef-c30e-11e7-81ab-f27164665fa9:8724
    2017-11-24T13:38:28.549313Z 0 [Note] Binlog end
    2017-11-24T13:38:28.554438Z 0 [Note] Shutting down plugin 'validate_password'
    2017-11-24T13:38:28.554458Z 0 [Note] Shutting down plugin 'ngram'
    2017-11-24T13:38:28.554460Z 0 [Note] Shutting down plugin 'partition'
    2017-11-24T13:38:28.554462Z 0 [Note] Shutting down plugin 'BLACKHOLE'
    2017-11-24T13:38:28.554464Z 0 [Note] Shutting down plugin 'ARCHIVE'
    2017-11-24T13:38:28.554466Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
    2017-11-24T13:38:28.554468Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
    2017-11-24T13:38:28.554469Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
    2017-11-24T13:38:28.554471Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
    [...CUT...]
    2017-11-24T13:38:28.554520Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
    2017-11-24T13:38:28.554522Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
    2017-11-24T13:38:28.554524Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
    2017-11-24T13:38:28.554525Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
    2017-11-24T13:38:28.554527Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
    2017-11-24T13:38:28.554528Z 0 [Note] Shutting down plugin 'InnoDB'
    2017-11-24T13:38:28.554636Z 0 [Note] InnoDB: FTS optimize thread exiting.
    2017-11-24T13:38:28.554738Z 0 [Note] InnoDB: Starting shutdown...
    2017-11-24T13:38:29.656074Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
    2017-11-24T13:38:30.568578Z 0 [Note] InnoDB: Shutdown completed; log sequence number 126001966
    2017-11-24T13:38:30.570358Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
    2017-11-24T13:38:30.570371Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
    2017-11-24T13:38:30.570420Z 0 [Note] Shutting down plugin 'MEMORY'
    2017-11-24T13:38:30.570424Z 0 [Note] Shutting down plugin 'CSV'
    2017-11-24T13:38:30.570428Z 0 [Note] Shutting down plugin 'MyISAM'
    2017-11-24T13:38:30.570457Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
    2017-11-24T13:38:30.570459Z 0 [Note] Shutting down plugin 'sha256_password'
    2017-11-24T13:38:30.570483Z 0 [Note] Shutting down plugin 'mysql_native_password'
    2017-11-24T13:38:30.570485Z 0 [Note] Shutting down plugin 'wsrep'
    2017-11-24T13:38:30.570658Z 0 [Note] Shutting down plugin 'binlog'
    2017-11-24T13:38:30.570889Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

  • #2
    Hello,

    I'm getting nowhere with this - any ideas would be more than appropriated.

    I think that I should at leas get something in the log files? Or maybe in the journal etc?

    Thnaks
    Chris

    Comment

    Working...
    X