Emergency

Percona-xtradb-cluster-57 fails start upg 5.7.18-29.20 -> 5.7.19-29.22. All nodes

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Percona-xtradb-cluster-57 fails start upg 5.7.18-29.20 -> 5.7.19-29.22. All nodes

    Percona XTRADB Cluester is down. It running for last 2 months without any issue.
    After upgrade, stopped and does not start.

    Disable systemctl and tried mannually

    /usr/sbin/mysqld --user=mysql --wsrep_recover --log-error=/tmp/start.tmp.LGZkh4w6HH.log

    Log:
    2017-09-29T13:20:08.867241Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp se$
    2017-09-29T13:20:08.868441Z 0 [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choo$
    2017-09-29T13:20:08.868467Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-17-57-log) starting as process 1455 ...
    2017-09-29T13:20:08.870109Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break wh$
    2017-09-29T13:20:08.871155Z 0 [Note] InnoDB: PUNCH HOLE support available
    2017-09-29T13:20:08.871167Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2017-09-29T13:20:08.871171Z 0 [Note] InnoDB: Uses event mutexes
    2017-09-29T13:20:08.871173Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2017-09-29T13:20:08.871176Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2017-09-29T13:20:08.871178Z 0 [Note] InnoDB: Using Linux native AIO
    2017-09-29T13:20:08.871358Z 0 [Note] InnoDB: Number of pools: 1
    2017-09-29T13:20:08.871445Z 0 [Note] InnoDB: Using CPU crc32 instructions
    2017-09-29T13:20:08.872478Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
    2017-09-29T13:20:08.875897Z 0 [Note] InnoDB: Completed initialization of buffer pool
    2017-09-29T13:20:08.877668Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See th$
    2017-09-29T13:20:08.888771Z 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
    2017-09-29T13:20:08.889147Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
    2017-09-29T13:20:09.193426Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes
    2017-09-29T13:20:09.202630Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
    2017-09-29T13:20:09.202687Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
    2017-09-29T13:20:09.336173Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
    2017-09-29T13:20:09.336916Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
    2017-09-29T13:20:09.336926Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
    2017-09-29T13:20:09.337460Z 0 [Note] InnoDB: Waiting for purge to start
    2017-09-29T13:20:09.387631Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.19-17 started; log sequence number 157903695
    2017-09-29T13:20:09.387668Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
    2017-09-29T13:20:09.387891Z 0 [Note] Plugin 'FEDERATED' is disabled.
    2017-09-29T13:20:09.581026Z 0 [Note] WSREP: Recovered position: dcb3260d-5d9a-11e7-b715-cf577114ed32:13578
    2017-09-29T13:20:09.581047Z 0 [Note] Binlog end
    2017-09-29T13:20:09.626804Z 0 [Note] Shutting down plugin 'ngram'
    2017-09-29T13:20:09.626831Z 0 [Note] Shutting down plugin 'partition'
    2017-09-29T13:20:09.626835Z 0 [Note] Shutting down plugin 'BLACKHOLE'
    2017-09-29T13:20:09.626838Z 0 [Note] Shutting down plugin 'ARCHIVE'
    2017-09-29T13:20:09.626840Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
    2017-09-29T13:20:09.626842Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
    2017-09-29T13:20:09.626844Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
    2017-09-29T13:20:09.626846Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
    2017-09-29T13:20:09.626848Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
    2017-09-29T13:20:09.626851Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
    2017-09-29T13:20:09.626853Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
    2017-09-29T13:20:09.626854Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
    2017-09-29T13:20:09.626856Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
    2017-09-29T13:20:09.626868Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
    2017-09-29T13:20:09.626870Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
    2017-09-29T13:20:09.626872Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
    2017-09-29T13:20:09.626874Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
    2017-09-29T13:20:09.626876Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
    2017-09-29T13:20:09.626878Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
    2017-09-29T13:20:09.626880Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
    2017-09-29T13:20:09.626882Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
    2017-09-29T13:20:09.626884Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
    2017-09-29T13:20:09.626886Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
    2017-09-29T13:20:09.626888Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
    2017-09-29T13:20:09.626890Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
    2017-09-29T13:20:09.626892Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
    2017-09-29T13:20:09.626894Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
    2017-09-29T13:20:09.626896Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
    2017-09-29T13:20:09.626898Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
    2017-09-29T13:20:09.626900Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
    2017-09-29T13:20:09.626902Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
    2017-09-29T13:20:09.626904Z 0 [Note] Shutting down plugin 'INNODB_CMP'
    2017-09-29T13:20:09.626906Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
    2017-09-29T13:20:09.626908Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
    2017-09-29T13:20:09.626910Z 0 [Note] Shutting down plugin 'INNODB_TRX'
    2017-09-29T13:20:09.626912Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
    2017-09-29T13:20:09.626914Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
    2017-09-29T13:20:09.626916Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
    2017-09-29T13:20:09.626918Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
    2017-09-29T13:20:09.626921Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
    2017-09-29T13:20:09.626923Z 0 [Note] Shutting down plugin 'InnoDB'
    2017-09-29T13:20:09.626978Z 0 [Note] InnoDB: FTS optimize thread exiting.
    2017-09-29T13:20:09.627092Z 0 [Note] InnoDB: Starting shutdown...
    2017-09-29T13:20:10.528446Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
    2017-09-29T13:20:11.581864Z 0 [Note] InnoDB: Shutdown completed; log sequence number 157903714
    2017-09-29T13:20:11.583248Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
    2017-09-29T13:20:11.583258Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
    2017-09-29T13:20:11.583262Z 0 [Note] Shutting down plugin 'CSV'
    2017-09-29T13:20:11.583265Z 0 [Note] Shutting down plugin 'MyISAM'
    2017-09-29T13:20:11.583279Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
    2017-09-29T13:20:11.583312Z 0 [Note] Shutting down plugin 'MEMORY'
    2017-09-29T13:20:11.583315Z 0 [Note] Shutting down plugin 'sha256_password'
    2017-09-29T13:20:11.583317Z 0 [Note] Shutting down plugin 'mysql_native_password'
    2017-09-29T13:20:11.583319Z 0 [Note] Shutting down plugin 'wsrep'
    2017-09-29T13:20:11.583512Z 0 [Note] Shutting down plugin 'binlog'
    2017-09-29T13:20:11.583661Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

    Same configuration that was perfectly working.

  • #2
    Hi

    I think I had the same issue. Same package versions on CentOS 7.
    It did not start up for me either. Since I had a working cluster, I just dropped the data from this servers and let it replicate from the cluster. This worked, even after a server reboot.

    mv /var/lib/mysql /var/lib/mysql-bck
    mkdir /var/lib/mysql
    chown mysql:mysql /var/lib/mysql
    restorecon -Rv /var/lib/mysql
    systemctl start mysql

    # Monitor it with
    tail -f /var/log/mysqld.log
    How did you solve this?

    G.

    Comment

    Working...
    X