EmergencyEMERGENCY? Get 24/7 Help Now!

How to obtain the “LES” (Last Executed Statement) from an Optimized Core Dump?

Posted on:



Share Button

Ever ran into a situation where you saw “some important variable you really needed to know about=<optimized out>” while debugging? Let’s look at an example:

It happens to all of us. This “issue” is seen when using optimized (release) binaries: debug symbols have been stripped out. In non-optimized binaries, the query would (in most cases) have shown directly in the backtrace. Sidenote: “in most cases”: sometimes you have to swap threads (or use: ‘thread apply all bt’ or ‘thread <nr>’ then ‘bt’) before you can view the actual crashing statement: gdb may have incorrectly analyzed which thread caused the crash. Inspecting the error log (which also contains a stack trace) may also help in such cases.

So… maybe you were testing a C program with three threads executing highly concurrent DML statements (where each thread executes one particular type of statement) and you are stuck as to which statement is causing your program to crash.. Or you just ran your latest and greatest RQG grammar and found this nifty crashing bug, only to find out you’ll be stuck with hours or even days of “grammar simplification”..

Not so! Read on…

If we did know the “LES” – or Last Executed Statement – in our RQG case for example, we could bring up the server with a copy of the test run’s datadir and try to re-execute the crashing statement to see if it crashes again – a plausible thing to happen. So, how do we make this happen if the query is not directly visible from the stack trace?

Let’s fire up gdb and see what we find. (We will assume the source code used for compiling is available at the same location as where it was at compilation time)

First we inspect the mysql_execute_command frame:

Looking at line 3138 (or by inspecting the stack trace above), we may expect the query to be present in the thd variable. Looking further, we can see that thd is being referenced in all frames, and that it is passed to handle_select. Let’s see what it contains:

Ok, so it’s a pointer to an array’s memory address. Using * before the address allows us to see the contents of the thd array:

Great! The query is showing in there: “INSERT INTO testdb_N . t1_temp1_N SELECT * FROM test.table10_int_autoinc”.

Instead of using a numeric memory address, we could also use:

Gdb knows the variable type, so we don’t need the typecast to “(THD *)”.

Or, better yet, now that we know where the query string is hiding (query_string > string > str), we can use:

In the future.

Conclusion: the Last Executed Statement was found in an optimized core dump by checking the contents of the thd array in the mysql_execute_command frame. By simply examining “surrounding” variables we’ve been able to find the information we needed. You can try this technique yourself the next time you need to find out exactly which query is causing you trouble!

Share Button

Roel Van de Paar

Roel leads Percona's QA team. Before coming to Percona, he contributed significantly to the QA infrastructure at Oracle. Roel has a varied background in IT, backed up by many industry leading certifications. He also enjoys time with God, his wife and 5 children, or heading into nature.

Insight for DBAs, Insight for Developers, MySQL


Leave a Reply

Percona’s widely read Percona Data Performance blog highlights our expertise in enterprise-class software, support, consulting and managed services solutions for both MySQL® and MongoDB® across traditional and cloud-based platforms. The decades of experience represented by our consultants is found daily in numerous and relevant blog posts.

Besides specific database help, the blog also provides notices on upcoming events and webinars.
Want to get weekly updates listing the latest blog posts? Subscribe to our blog now! Submit your email address below and we’ll send you an update every Friday at 1pm ET.

No, thank you. Please do not ask me again.