October 25, 2014

SystemTap – DTrace for Linux ?

Since DTrace was released for Solaris I am missing it on Linux systems… It can’t be included in Linux by the same reason why ZFS can’t be – it’s licensing issue. Both ZFS and DTrace are under CDDL, which is incompatible with GPL. So you can see DTrace and ZFS on Solaris, FreeBSD, MacOS, but not on Linux.

However I follow the project SystemTap for couple of years (it was started in 2005), which is supposed to provide similar to DTrace functionality.

Why I am interested in this tool, because there is no simple way under Linux to profile not CPU-bound load (for CPU-bound there is OProfile, see for example
http://mysqlinsights.blogspot.com/2009/08/oprofile-for-io-bound-apps.html). I.e. for IO-bound or for mutex contention problems OProfile is not that useful.

SystemTap is included in RedHat 5 releases, but I was not able to get it running even in CentOS 5.3 (it crashed and hung every so often). Latest updated RedHat 5.4 promised some more fixes to SystemTap, so I decided to give it more try as soon as I got RedHat 5.4 on hands.

Surprising, but now it runs much more stable. I was able to get profiling of kernel and system calls.
Here is simple script to show IO activity per disk per process (well, it is similar to iotop, but iotop is not available in RedHat / CentOS)

with output like this

This example maybe is simple, but the point is that there is rich scripting language with tons
of probes you can intersect ( kernel functions, FS drivers functions, any other drives and modules)

What else I see very useful in SystemTap it can work in userspace. That is you can use it to profile your and any application that has -debuginfo packages ( all -debuginfo for standard RedHat RPMS you can download from RedHat FTP), but basically it is info you get compiling with

.

Well, there seems another war story going on. To profile userspace application with SystemTap your kernel should be patches with

patch, which fortunately is included in RedHat based kernels, but not included in vanilla kernel yet. So I am not sure if you can get userspace profiling running in another distributives.

There is quite simple script that I tried to hack around MySQL ®

with output which I get running simple SELECT against InnoDB table:

Again, this case is maybe too simple, but basically you can intersect internal MySQL function and script (measure time, count of call, statistics) what you what. I did not figure out yet how to intersect C++ style function (i.e.

), so there is area to investigate.

So I am going to play with it more and do some useful scripting to get profiling of MySQL.

And it seems SystemTap can re-use DTrace probes available in application, as you may know DTrace-probes were added into MySQL 5.4, so interesting how it works.

I should mention that there is second alternative of DTrace… It’s …. DTrace port. Looking on blog it seems one-man project and currently author is fighting with resolving userspace issues. I gave to this a try, but on my current RedHat 5.4 after several runs I got “Kernel panic”, so it’s enough for now.

About Vadim Tkachenko

Vadim leads Percona's development group, which produces Percona Clould Tools, the Percona Server, Percona XraDB Cluster and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.

Comments

  1. herodiade says:

    > there is no simple way under Linux to profile not CPU-bound load
    > [...]
    > I.e. for IO-bound or for mutex contention problems OProfile is not that useful.

    True, esp. when emphasing “simple”, and when compared to DTrace.
    But, beside SystemTap a couple of tools migth still give some help here:
    * blktrace , for I/O
    * the very recent “mutrace”, to track mutex contention (see http://0pointer.de/blog/projects/mutrace.html ).

    the point of this comment was to introduce mutrace indeed ;)

  2. CoolCold says:

    I’m not really sure (using debian), but it looks like iotop should work now on rhel – http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5

  3. To get the real name of a C++ function you could use objdump:

    # objdump -t /usr/local/mysql/libexec/mysqld | grep index_read | grep ha_innoba |grep -v index_read_
    08269c80 g F .text 000002a9 _ZN11ha_innobase10index_readEPcPKcj16ha_rkey_function

    Unless it’s stripped:

    # file /usr/local/mysql/libexec/mysqld
    /usr/local/mysql/libexec/mysqld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
    # strip /usr/local/mysql/libexec/mysqld.stip
    # file /usr/local/mysql/libexec/mysqld.stip
    /usr/local/mysql/libexec/mysqld.stip: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
    # objdump -t /usr/local/mysql/libexec/mysqld.stip

    /usr/local/mysql/libexec/mysqld.stip: file format elf32-i386

    SYMBOL TABLE:
    no symbols

  4. otoburb says:

    DTrace for Linux looks like it’s being worked on here:

    http://www.crisp.demon.co.uk/blog/index.html

  5. Vadim says:

    otoburb,

    I mention that in the post. It caused kernel panic when I tried it.

  6. popgo says:

    Linux port of DTrace
    June 2009
    Paul D. Fox
    fox@crisp.demon.co.uk
    http://www.twitter.com/crispeditor
    http://www.crisp.demon.co.uk

    Blog – latest news and stuff about the dtrace project:
    http://www.crisp.demon.co.uk/blog/
    Download dtrace tarballs for linux here:
    ftp://crisp.dynalias.com/pub/release/website/dtrace

  7. Vadim says:

    popgo,

    I mentioned that tool at the end of post.
    Unfortunately when I tried it I got crashed server with Kernel panic error,
    so I would wait sometime before trying it in production.

  8. Tony says:

    how did you get this working on Centos?

  9. Vadim says:

    Tony,

    Centos 5.5 comes with stable 1.1 release of systemtap

  10. Tony says:

    Okay, but how did you get mysql compiled with systemtap support?

  11. Vadim says:

    Tony,

    You need only debug symbols to have that working. It is true for any package not only mysql.
    You can get debug symbols if you compile with CFLASG=”-g” CXXFLAGS=”-g”. Or alternative – there
    are available debug symbols packages almost for any software in CentOS distribution.

  12. Tony says:

    I see, well i quick “yum search mysql” yields no debug packages.. is there a repo i need to enable?

  13. Vadim says:

    Tony,

    You should install appropriate RPM from http://debuginfo.centos.org/5/x86_64/ manually .

  14. Tony says:

    Ah cool, thanks for the quick replies, this site looks like a great resource for mysql tips – *bookmarks*

  15. Anto says:

    if I/O is your concern, you could look into blktrace and related tools

    http://brick.kernel.dk/snaps/

    the blkiomon command (part of the same package) seems very useful in particular, example:

    # blktrace /dev/sda -a issue -a complete -w 60 -o – | blkiomon -I 4 -h -

  16. Bob Gu says:

    While I am was doing some test with SystemTap, I noticed two problems. Not sure, if there is anyone else ran into the same problems.

    1. The client session will be disconnected (MySQL server has gone away) after the SystemTap script is started or terminated.
    test@localhost [(none)]>select now();
    +———————+
    | now() |
    +———————+
    | 2011-07-27 19:58:39 |
    +———————+
    1 row in set (0.00 sec)

    test@localhost [(none)]>select now();
    ERROR 2006 (HY000): MySQL server has gone away
    No connection. Trying to reconnect…
    Connection id: 7
    Current database: *** NONE ***

    +———————+
    | now() |
    +———————+
    | 2011-07-27 19:59:04 |
    +———————+
    1 row in set (0.00 sec)

    2. When running the SystemTap script, command “alter table xxx truncate partition yyy” might hang. But if we terminate the running SystemTap script, the command will continue and success.

Speak Your Mind

*