Announcement

Announcement Module
Collapse
No announcement yet.

Primary XtraDB crashed

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Primary XtraDB crashed

    Hi,

    One of the members of the custer crashed unexpectedly. Log file as follows:

    …..
    131124 5:28:14 [Note] WSREP: (a89f58b5-3fc0-11e3-88e2-42a6d4c067ed, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://192.168.133.137:4567
    05:28:14 UTC - mysqld got signal 11 ;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.
    We will try our best to scrape up some info that will hopefully help
    diagnose the problem, but since we have already crashed,
    something is definitely wrong and this may fail.
    Please help us make Percona Server better by reporting any
    bugs at http://bugs.percona.com/

    key_buffer_size=8388608
    read_buffer_size=131072
    max_used_connections=13
    max_threads=153
    thread_count=2
    connection_count=2
    It is possible that mysqld could use up to
    key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 343054 K bytes of memory
    Hope that's ok; if not, decrease some variables in the equation.

    Thread pointer: 0x0
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    stack_bottom = 0 thread_stack 0x40000
    /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d8405]
    /usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6b0804]
    /lib64/libpthread.so.0[0x348940f500]
    /usr/lib64/libgalera_smm.so(_ZN4asio6detail8write_opINS_3ssl6 streamINS_19basic_stream_socketINS_2ip3tcpENS_21st ream_socket_serviceIS6_EEEENS2_14stream_serviceEEE N5boost5arrayINS_12const_bufferELm2EEENS0_14transf er_all_tENSC_3_bi6bind_tIvNSC_4_mfi3mf2IvN5gcomm13 AsioTcpSocketERKNS_10error_codeEmEENSH_5list3INSH_ 5valueINSC_10shared_ptrISM_EEEEPFNSC_3argILi1EEEvE PFNSW_ILi2EEEvEEEEEEclESP_mi+0x1d1)[0x7f5e4d618f61]
    /usr/lib64/libgalera_smm.so(_ZN4asio3ssl6detail22openssl_stre am_service10io_handlerINS_19basic_stream_socketINS _2ip3tcpENS_21stream_socket_serviceIS6_EEEENS_6det ail8write_opINS0_6streamIS9_NS0_14stream_serviceEE EN5boost5arrayINS_12const_bufferELm2EEENSA_14trans fer_all_tENSF_3_bi6bind_tIvNSF_4_mfi3mf2IvN5gcomm1 3AsioTcpSocketERKNS_10error_codeEmEENSK_5list3INSK _5valueINSF_10shared_ptrISP_EEEEPFNSF_3argILi1EEEv EPFNSZ_ILi2EEEvEEEEEEEE12handler_implESS_m+0x15)[0x7f5e4d619235]
    /usr/lib64/libgalera_smm.so(_ZN4asio3ssl6detail22openssl_stre am_service12base_handlerINS_19basic_stream_socketI NS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEE7do_ funcERKNS_10error_codeEm+0x18)[0x7f5e4d60e428]
    /usr/lib64/libgalera_smm.so(_ZN4asio3ssl6detail17openssl_oper ationINS_19basic_stream_socketINS_2ip3tcpENS_21str eam_socket_serviceIS5_EEEEE18async_user_handlerENS _10error_codeEi+0x27)[0x7f5e4d60e367]
    /usr/lib64/libgalera_smm.so(_ZNK5boost9function2IiRKN4asio10e rror_codeEiEclES4_i+0x18)[0x7f5e4d60e3d8]
    /usr/lib64/libgalera_smm.so(_ZN4asio3ssl6detail17openssl_oper ationINS_19basic_stream_socketINS_2ip3tcpENS_21str eam_socket_serviceIS5_EEEEE19async_write_handlerEb iRKNS_10error_codeEm+0x6e)[0x7f5e4d60e9ae]
    /usr/lib64/libgalera_smm.so(_ZN4asio6detail14strand_service8d ispatchINS0_7binder2IN5boost3_bi6bind_tIvNS4_4_mfi 3mf4IvNS_3ssl6detail17openssl_operationINS_19basic _stream_socketINS_2ip3tcpENS_21stream_socket_servi ceISE_EEEEEEbiRKNS_10error_codeEmEENS5_5list5INS5_ 5valueIPSI_EENSO_IbEENSO_IiEEPFNS4_3argILi1EEEvEPF NST_ILi2EEEvEEEEESJ_mEEEEvRPNS1_11strand_implET_+0 xc9)[0x7f5e4d60bd79]
    /usr/lib64/libgalera_smm.so(_ZN27asio_handler_invoke_helpers6 invokeIN4asio6detail17rewrapped_handlerINS2_7binde r2INS2_8write_opINS1_19basic_stream_socketINS1_2ip 3tcpENS1_21stream_socket_serviceIS8_EEEENS1_17muta ble_buffers_1ENS2_14transfer_all_tENS2_15wrapped_h andlerINS1_10io_service6strandEN5boost3_bi6bind_tI vNSH_4_mfi3mf4IvNS1_3ssl6detail17openssl_operation ISB_EEbiRKNS1_10error_codeEmEENSI_5list5INSI_5valu eIPSP_EENSV_IbEENSV_IiEEPFNSH_3argILi1EEEvEPFNS10_ ILi2EEEvEEEEEEEEESQ_mEES18_EES1C_EEvRKT_RT0_+0x174 )[0x7f5e4d611184]
    /usr/lib64/libgalera_smm.so(_ZN4asio6detail14strand_service8d ispatchINS0_17rewrapped_handlerINS0_7binder2INS0_8 write_opINS_19basic_stream_socketINS_2ip3tcpENS_21 stream_socket_serviceIS8_EEEENS_17mutable_buffers_ 1ENS0_14transfer_all_tENS0_15wrapped_handlerINS_10 io_service6strandEN5boost3_bi6bind_tIvNSH_4_mfi3mf 4IvNS_3ssl6detail17openssl_operationISB_EEbiRKNS_1 0error_codeEmEENSI_5list5INSI_5valueIPSP_EENSV_IbE ENSV_IiEEPFNSH_3argILi1EEEvEPFNS10_ILi2EEEvEEEEEEE EESQ_mEES18_EEEEvRPNS1_11strand_implET_+0x492)[0x7f5e4d6134f2]
    /usr/lib64/libgalera_smm.so(_ZN4asio6detail23reactive_socket_ send_opINS_17mutable_buffers_1ENS0_8write_opINS_19 basic_stream_socketINS_2ip3tcpENS_21stream_socket_ serviceIS6_EEEES2_NS0_14transfer_all_tENS0_15wrapp ed_handlerINS_10io_service6strandEN5boost3_bi6bind _tIvNSE_4_mfi3mf4IvNS_3ssl6detail17openssl_operati onIS9_EEbiRKNS_10error_codeEmEENSF_5list5INSF_5val ueIPSM_EENSS_IbEENSS_IiEEPFNSE_3argILi1EEEvEPFNSX_ ILi2EEEvEEEEEEEEEE11do_completeEPNS0_15task_io_ser viceEPNS0_25task_io_service_operationESN_m+0x1d5)[0x7f5e4d613775]
    /usr/lib64/libgalera_smm.so(_ZN4asio6detail15task_io_service3 runERNS_10error_codeE+0x459)[0x7f5e4d62b009]
    /usr/lib64/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_lo opERKN2gu8datetime6PeriodE+0x1d6)[0x7f5e4d622f56]
    /usr/lib64/libgalera_smm.so(_ZN9GCommConn3runEv+0x57)[0x7f5e4d63d5d7]
    /usr/lib64/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x9)[0x7f5e4d640ea9]
    /lib64/libpthread.so.0[0x3489407851]
    /lib64/libc.so.6(clone+0x6d)[0x3488ce894d]
    You may download the Percona Server operations manual by visiting
    http://www.percona.com/software/percona-server/. You may find information
    in the manual which will help you identify the cause of the crash.
    131124 05:28:14 mysqld_safe Number of processes running now: 0
    131124 05:28:14 mysqld_safe WSREP: not restarting wsrep node automatically
    131124 05:28:14 mysqld_safe mysqld from pid file /var/lib/mysql/primary.mysql.com.pid ended
    [root@primary mysql]#

    [root@primary mysql]# service mysqld start
    mysqld: unrecognized service
    [root@primary mysql]# service mysql start
    ERROR! MySQL (Percona XtraDB Cluster) is not running, but PID file exists
    [root@primary mysql]# service mysql restart
    Shutting down MySQL (Percona XtraDB Cluster) ERROR! MySQL (Percona XtraDB Cluster) server process #14393 is not running!
    Starting MySQL (Percona XtraDB Cluster)...... SUCCESS!
    [root@primary mysql]#

    Any idea why could it happened?

    Thanks

  • #2
    After the restart all looks good:

    [root@primary mysql]# tail -n 50 primary.mysql.com.err
    131127 10:38:05 [Note] WSREP: New cluster view: global state: 9f9985f1-345e-11e3-bb4d-ee083db0a078:52822, view# 55: Primary, number of nodes: 3, my index: 2, protocol version 2
    131127 10:38:05 [Warning] WSREP: Gap in state sequence. Need state transfer.
    131127 10:38:07 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '192.168.133.136' --auth 'sstuser:Mucha3G698QqWkR' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2893''
    WSREP_SST: [INFO] Streaming with xbstream (20131127 10:38:07.364)
    WSREP_SST: [INFO] Using socat as streamer (20131127 10:38:07.370)
    WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20131127 10:38:07.406)
    131127 10:38:07 [Note] WSREP: Prepared SST request: xtrabackup|192.168.133.136:4444/xtrabackup_sst
    131127 10:38:07 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    131127 10:38:07 [Note] WSREP: Assign initial position for certification: 52822, protocol version: 2
    131127 10:38:07 [Note] WSREP: IST receiver using ssl
    131127 10:38:07 [Note] WSREP: Prepared IST receiver, listening at: ssl://192.168.133.136:4568
    131127 10:38:07 [Note] WSREP: Node 2 (primary.mysql.com) requested state transfer from '*any*'. Selected 0 (secondary.mysql.com)(SYNCED) as donor.
    131127 10:38:07 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 52822)
    131127 10:38:07 [Note] WSREP: Requesting state transfer: success, donor: 0
    WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20131127 10:38:07.861)
    WSREP_SST: [INFO] Total time on joiner: 0 seconds (20131127 10:38:07.869)
    WSREP_SST: [INFO] Removing the sst_in_progress file (20131127 10:38:07.875)
    131127 10:38:07 [Note] WSREP: SST complete, seqno: 41803
    131127 10:38:07 [Note] Plugin 'FEDERATED' is disabled.
    131127 10:38:07 InnoDB: The InnoDB memory heap is disabled
    131127 10:38:07 InnoDB: Mutexes and rw_locks use GCC atomic builtins
    131127 10:38:07 InnoDB: Compressed tables use zlib 1.2.3
    131127 10:38:07 InnoDB: Using Linux native AIO
    131127 10:38:07 InnoDB: Initializing buffer pool, size = 128.0M
    131127 10:38:07 InnoDB: Completed initialization of buffer pool
    131127 10:38:07 InnoDB: highest supported file format is Barracuda.
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    131127 10:38:07 InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    131127 10:38:08 InnoDB: Waiting for the background threads to start
    131127 10:38:09 Percona XtraDB (http://www.percona.com) 5.5.33-rel31.0 started; log sequence number 51365190
    131127 10:38:09 [Note] Event Scheduler: Loaded 0 events
    131127 10:38:09 [Note] WSREP: Signalling provider to continue.
    131127 10:38:09 [Note] WSREP: SST received: 9f9985f1-345e-11e3-bb4d-ee083db0a078:41803
    131127 10:38:09 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.5.33' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), wsrep_23.7.6.r3915
    131127 10:38:09 [Note] WSREP: Receiving IST: 11019 writesets, seqnos 41803-52822
    131127 10:38:17 [Note] WSREP: IST received: 9f9985f1-345e-11e3-bb4d-ee083db0a078:52822
    131127 10:38:17 [Note] WSREP: 0 (secondary.mysql.com): State transfer to 2 (primary.mysql.com) complete.
    131127 10:38:17 [Note] WSREP: 2 (primary.mysql.com): State transfer from 0 (secondary.mysql.com) complete.
    131127 10:38:17 [Note] WSREP: Shifting JOINER -> JOINED (TO: 52822)
    131127 10:38:17 [Note] WSREP: Member 0 (secondary.mysql.com) synced with group.
    131127 10:38:17 [Note] WSREP: Member 2 (primary.mysql.com) synced with group.
    131127 10:38:17 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 52822)
    131127 10:38:17 [Note] WSREP: Synchronized with group, ready for connections
    131127 10:38:17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    [root@primary mysql]# cat /etc/my.cnf
    [mysqld]

    ssl-cipher=DHE-RSA-AES256-SHA
    ssl-ca=/etc/mysql/ca-cert-mysql-client.pem
    ssl-cert=/etc/mysql/server-cert-mysql-client.pem
    ssl-key=/etc/mysql/server-key-mysql-client.pem

    datadir=/var/lib/mysql
    user=mysql

    # Max connections
    max_connect_errors=1844674407370954751
    # SSL configuration
    wsrep_provider_options="socket.ssl_cert=/etc/mysql/cert.pem; socket.ssl_key=/etc/mysql/key.pem"

    # Path to Galera library
    wsrep_provider=/usr/lib64/libgalera_smm.so

    # Cluster connection URL contains the IPs of nodes
    wsrep_cluster_address=gcomm://192.168.133.136,192.168.133.137,192.168.132.4

    # In order for Galera to work correctly binlog format should be ROW
    binlog_format=ROW

    # MyISAM storage engine has only experimental support
    default_storage_engine=InnoDB

    # This is a recommended tuning variable for performance
    innodb_locks_unsafe_for_binlog=1

    # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
    innodb_autoinc_lock_mode=2

    # Node #1 address
    wsrep_node_address=192.168.133.136

    # SST method
    wsrep_sst_method=xtrabackup

    # Cluster name
    wsrep_cluster_name=mysql_cluster

    # Authentication for SST method
    wsrep_sst_auth="sstuser:Mucha3G698QqWkR"
    [root@primary mysql]#

    Comment


    • #3
      Thanks for sharing this, evrynet. Glad it is resolved.
      Is this an emergency? Get immediate assistance from Percona Support 24/7. Click here.

      Comment


      • #4
        Well it's not really resolved because we don't know why it happened…. so why it happened?

        Comment

        Working...
        X