GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

Percona Server 5.5.16 crashing

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Percona Server 5.5.16 crashing

    All,

    We are using Percona Server 5.5.16 in our production servers. When we are running below query Mysql get restarted automatically

    select count(distinct entry_id) from db1.t1 where pid = xxx;

    The error log is as follows that we saw during Mysql restart

    Thread pointer: 0x2ab116833d90
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    stack_bottom = 0x533c20e0 thread_stack 0x40000
    /usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7ce749]
    /usr/sbin/mysqld(handle_segfault+0x379)[0x500de9]
    /lib64/libpthread.so.0[0x3f7220ebe0]
    /lib64/libc.so.6(gsignal+0x35)[0x3f71a30285]
    /lib64/libc.so.6(abort+0x110)[0x3f71a31d30]
    /usr/sbin/mysqld[0x8a6ec0]
    /usr/sbin/mysqld[0x8542b6]
    /usr/sbin/mysqld[0x855f57]
    /usr/sbin/mysqld[0x82d797]
    /usr/sbin/mysqld(_ZN26QUICK_GROUP_MIN_MAX_SELECT11next_prefi xEv+0x12c)[0x75c28c]
    /usr/sbin/mysqld(_ZN26QUICK_GROUP_MIN_MAX_SELECT8get_nextEv+ 0x30)[0x75ccd0]
    /usr/sbin/mysqld[0x76f2c4]
    /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x66)[0x5a6dd6]
    /usr/sbin/mysqld[0x5a9070]
    /usr/sbin/mysqld(_ZN4JOIN4execEv+0x12a8)[0x5b8128]
    /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTj R4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resul tP18st_select_lex_unitP13st_select_lex+0x1a9)[0x5b9589]
    /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_result m+0x1ce)[0x5b9ede]
    /usr/sbin/mysqld[0x57727d]
    /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2505)[0x579c55]
    /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x38 c)[0x57dadc]
    /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3 THDPcj+0x148e)[0x57ef8e]
    /usr/sbin/mysqld(_Z10do_commandP3THD+0x106)[0x57f326]
    /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x111)[0x61a0a1]
    /usr/sbin/mysqld(handle_one_connection+0x54)[0x61a194]
    /lib64/libpthread.so.0[0x3f7220677d]
    /lib64/libc.so.6(clone+0x6d)[0x3f71ad325d]


    130528 23:14:41 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:15:07 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:15:22 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:15:45 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:16:09 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:17:19 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:17:25 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:17:50 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:18:03 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction
    130528 23:18:25 [ERROR] Failed to write to mysql.slow_log: Deadlock found when trying to get lock; try restarting transaction

  • #2
    @mchauhan75,

    Can you post the full log starting prior to the crash and after MySQL restarted.
    Our documentation has a lot of answers about common questions on Percona software, have you checked there before posting that question here? http://www.percona.com/forums/core/i...lies/smile.png

    Join us at the annual Percona Live MySQL Users Conference - http://www.percona.com/live/mysql-conference-2014/

    Comment


    • #3
      Here is the full log


      120720 22:56:38 mysqld_safe mysqld from pid file /var/lib/mysql/db01.egnyte.com.pid ended
      120720 23:30:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
      120720 23:30:55 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
      120720 23:30:55 [Note] Flashcache bypass: disabled
      120720 23:30:55 [Note] Flashcache setup error is : ioctl failed

      120720 23:30:55 [Note] Plugin 'FEDERATED' is disabled.
      120720 23:30:55 InnoDB: The InnoDB memory heap is disabled
      120720 23:30:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      120720 23:30:55 InnoDB: Compressed tables use zlib 1.2.3
      120720 23:30:55 InnoDB: Using Linux native AIO
      120720 23:30:55 InnoDB: Initializing buffer pool, size = 22.0G
      120720 23:30:56 InnoDB: Completed initialization of buffer pool
      120720 23:30:56 InnoDB: highest supported file format is Barracuda.
      120720 23:30:57 InnoDB: Waiting for the background threads to start
      120720 23:30:58 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 33382526481
      ^G/usr/sbin/mysqld: File '/mnt/Storage/logs/mysql-bin.000259' not found (Errcode: 2)
      120720 23:30:59 [ERROR] Failed to open log (file '/mnt/Storage/logs/mysql-bin.000259', errno 2)
      120720 23:30:59 [ERROR] Could not open log file
      120720 23:30:59 [ERROR] Can't init tc log
      120720 23:30:59 [ERROR] Aborting

      120720 23:30:59 InnoDB: Starting shutdown...
      120720 23:31:03 InnoDB: Shutdown completed; log sequence number 33382526481
      120720 23:31:03 [Note] /usr/sbin/mysqld: Shutdown complete

      120720 23:31:04 mysqld_safe mysqld from pid file /var/lib/mysql/db01.egnyte.com.pid ended
      120720 23:39:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
      120720 23:39:43 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
      120720 23:39:43 [Note] Flashcache bypass: disabled
      120720 23:39:43 [Note] Flashcache setup error is : ioctl failed

      120720 23:39:43 [Note] Plugin 'FEDERATED' is disabled.
      120720 23:39:43 InnoDB: The InnoDB memory heap is disabled
      120720 23:39:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      120720 23:39:43 InnoDB: Compressed tables use zlib 1.2.3
      120720 23:39:43 InnoDB: Using Linux native AIO
      120720 23:39:43 InnoDB: Initializing buffer pool, size = 22.0G
      120720 23:39:44 InnoDB: Completed initialization of buffer pool
      120720 23:39:44 InnoDB: highest supported file format is Barracuda.
      120720 23:39:46 InnoDB: Waiting for the background threads to start
      120720 23:39:47 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 33382526481
      ^G/usr/sbin/mysqld: File '/mnt/Storage/logs/mysql-bin.000259' not found (Errcode: 2)
      120720 23:39:47 [ERROR] Failed to open log (file '/mnt/Storage/logs/mysql-bin.000259', errno 2)
      120720 23:39:47 [ERROR] Could not open log file
      120720 23:39:47 [ERROR] Can't init tc log
      120720 23:39:47 [ERROR] Aborting

      120720 23:39:47 InnoDB: Starting shutdown...
      120720 23:39:52 InnoDB: Shutdown completed; log sequence number 33382526481
      120720 23:39:52 [Note] /usr/sbin/mysqld: Shutdown complete

      nstead.
      120720 23:41:26 [Note] Flashcache bypass: disabled
      120720 23:41:26 [Note] Flashcache setup error is : ioctl failed

      120720 23:41:26 [Note] Plugin 'FEDERATED' is disabled.
      120720 23:41:26 InnoDB: The InnoDB memory heap is disabled
      120720 23:41:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      120720 23:41:26 InnoDB: Compressed tables use zlib 1.2.3
      120720 23:41:26 InnoDB: Using Linux native AIO
      120720 23:41:26 InnoDB: Initializing buffer pool, size = 22.0G
      120720 23:41:27 InnoDB: Completed initialization of buffer pool
      120720 23:41:27 InnoDB: highest supported file format is Barracuda.
      120720 23:41:28 InnoDB: Waiting for the background threads to start
      120720 23:41:29 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 33382526481
      120720 23:41:29 [Note] Recovering after a crash using /logs/mysql-bin
      120720 23:41:29 [Note] Starting crash recovery...
      120720 23:41:29 [Note] Crash recovery finished.

      Comment


      • #4
        You could try to edit index file and start mysql again
        This error will pop up when binary logging is enabled, and someone thought it was a good idea to remove binary logs from the filesystem: /usr/sbin/mysqld:

        Comment


        • #5
          Hi mchauhan75,
          The second log fragment you pasted is much older one - 120720 vs 130528 you had segfault error in. So I guess it's totally unrelated to this crash.

          I would check if there is no disk space problem on this server. Also check the db1.t1 table - it may be corrupted and thus triggering the crash.

          Another thing is that there has been many bug fixed since 5.5.16, I'd suggest you upgrade to latest Percona Server (5.5.31 now). Of course after you check the disk space and table I mentioned, as well as if there are another issues shown in recent error log.

          Comment

          Working...
          X