October 20, 2014

Logging Deadlock errors

The principal source of information for InnoDB diagnostics is the output of SHOW ENGINE INNODB STATUS but there are some sections that are not very useful. For example, LATEST DETECTED DEADLOCK only shows, as the name implies, the latest error detected. If you have 100 deadlocks per minute you will be able to see only the latest one and that is not very helpful when you are debugging your application code. Continuing my series of blog posts about Percona Toolkit, in this case I’m going to talk about pt-deadlock-logger.

pt-deadlock-logger

The tool is very simple and useful. It monitors the output of SHOW ENGINE INNODB STATUS and log the new deadlocks to a file or to a table that we can later review. Let’s see how it works with an example.

The basic command to use is:

Without any more options the tool will start to show on STDOUT the deadlocks that are happening on our database. This tool is usually run for a long period of time that you can specify with –run-time so sometimes make sense to store the information on a file (–log) or on a table (–dest) for later review. In this example we are going to store the deadlocks information on a table.

First we have to create the destination table:

Now, we can start to log the deadlocks in our database:

After some time, we can check the deadlock information in our table:

We can see the two queries that caused the deadlock with the information like user, hostname, thread id and which one was the victim of the deadlock. You can group by server and timestamp to get all events that correspond to the same deadlock.

Another trick, with infrastructures of more than one server is possible to use the tool to have a central repository with all the deadlocks. We just need to run the tool on every node and configure it to store logs on a centralized database.

In MySQL 5.6 will be possible to log all deadlocks on error log using innodb_print_all_deadlocks parameter.

About Miguel Angel Nieto

Miguel joined Percona in October 2011. He has worked as a System Administrator for a Free Software consultant and in the supporting area of the biggest hosting company in Spain. His current focus is improving MySQL and helping the community of Free Software to grow. Miguel's roles inside Percona are Senior Support Engineer and Manager of EMEA Support Team.

Comments

  1. sbester says:

    MySQL 5.6 logs all deadlocks but I think it’s still in an unstructured format?
    ( http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_print_all_deadlocks )

  2. sbester, good one. I didn’t know about that parameter on 5.6. Seems that it logs the deadlocks on error log with the same format that uses for show engine innodb. I’m going to update the blog post.

  3. Cédric says:

    MONyog also logs deadlocks : http://www.webyog.com/en/monyog_feature_list.php
    “pt-deadlock-logger” is a nice tool and that works but what bothers me a bit (just a bit) with the “–daemonize” option of the Percona Toolkit is the need to ensure that the process run all the time (don’t forget to relaunch the tool after a server reboot for example).

  4. Ricardo says:

    This is not working in Ubuntu 12.04, getting a perl error:
    Argument “” isn’t numeric in numeric comparison () at /usr/bin/pt-deadlock-logger line 2145.

  5. Colin says:

    Here is a script that will record full deadlock details in separate files: https://gist.github.com/colinmollenhour/6578ac005074bd2f992e

Speak Your Mind

*