Diagnosing MySQL Crashes on RHEL with GDB: How to Identify the Database, Table, and Query Involved

When troubleshooting a MySQL crash, having only the error log is rarely enough to pinpoint the exact root cause. To truly understand what happened, we need to go deeper—into the memory state of the process at the moment it crashed. That’s where GDB, the GNU Debugger, comes in. GDB lets us inspect a core dump (a memory snapshot taken at crash time) and navigate through the function calls, variables, and internal MySQL structures as they existed right before failure.

In this post, we’ll walk through how to set up the right debug environment using Percona Server for MySQL with symbols, load a core dump, and extract the exact query, database, and table that caused the crash—step by step.

Step 0: Get the Build ID from the MySQL error log and OS version

When MySQL crashes due to an internal error, such as a failed assertion, corruption, or invalid memory access, it writes detailed information to the error log. Among that output is a key value:

Why is the Build ID important?

The Build ID uniquely identifies the exact binary that was running at the time of the crash. It is used to:

• Ensure that the binary you use during debugging matches exactly what was running on the crashed server.
  
• Allow gdb to resolve symbols from the core file correctly, such as function names, file names, and line numbers.
  
• Avoid mismatches that would lead to incomplete or incorrect debugging output.

If the binary and the core dump are even slightly mismatched (e.g., a minor patch difference or rebuild), your analysis may fail or point to the wrong location.

What else do we need?

We also need to confirm the operating system version on which MySQL was running. This helps us choose the right base image and debug packages.

In our case, the OS information was collected using pt-summary from the Percona Toolkit:

From this, we confirmed:

• The OS is AlmaLinux 9.5, a RHEL 9-compatible distribution.
  
• The customer also confirmed that MySQL was installed via RPMs, not from source or tarballs.
  

With this information, we could recreate a similar environment using Red Hat UBI9 + Percona RPMs.

Step 1: Launch a RHEL 9 (UBI9) debug container

To safely inspect the core dump and reproduce the runtime environment, we use a Docker container based on Red Hat’s official UBI9 (Universal Base Image).

This mounts your local crash directory into the container and runs the base image in detached mode.

Then, connect to it:

Now you’re inside a clean RHEL 9 container where you can install exactly the same version of MySQL + debug symbols.

Step 2: Install MySQL debug binaries and GDB

Inside the container, install Percona’s debug builds and modern GDB tools:

Then install the packages:

What each package does:

• percona-server-server: the exact MySQL server binary that crashed
  
• percona-server-debuginfo: symbols and debug metadata for that binary
  
• devtoolset-11-gdb: a modern version of GDB (with better C++ support)
  
• file, vim, etc.: standard tools for inspection
  

This setup mirrors the environment of the production server, but safely and with tooling.

Step 3: Confirm Build ID matches the core dump

Now, verify that the binary you just installed is identical to the one that produced the core file. Below is an example of the expected output:

Explanation:

• eu-unstrip from elfutils reads the Build ID from the core dump itself
  
• file reads the Build ID from the binary you just installed
  
• Both must match the BuildID[sha1]=… from the error log
  

If they don’t match, GDB won’t show function names or source code. If they do match, you’ll get full symbolized backtraces.

Step 4: Launch GDB with debug info and Debuginfod

Now we’re ready to inspect the core:

Explanation:

• file /usr/sbin/mysqld: loads the binary you installed
  
• core /mnt/…: loads the actual crash core file
  
• debuginfod: enables automatic download of source code (if available)
  
• thread-events off: disables verbose thread events in GDB output
  
• safe-path: avoids security restrictions on file loading
  

Once inside GDB, you’ll have access to a fully symbolized debug session—with real function names, line numbers, and source info.

Step 5: Extract the query and schema from the core dump

Goal:

Find out what query was running and in which schema when the crash occurred.

Step-by-step:

1. Show the stack trace:

This prints all function calls that led to the crash. This is the full stack trace:

The stack trace is read from bottom up. This means the thread started in frame 49, and it went all the way up until the crash. We can see that Percona Server raised the signal to the OS in frame 4. In our case, since we want the details about the table structure we will look into the 41:

Then we will switch to that frame:

This command jumps to frame 41. This specific frame corresponds to the mysql_execute_command() function, which is responsible for executing SQL commands (e.g., SELECT, INSERT, DELETE). The output confirms the source location:

2. Dump the THD (Thread Handle) structure

This is where the magic happens. thd is a rich structure that includes:

• m_query_string.str — the raw SQL query text.
• m_db.str — the active database.

And a lot more information.

From this, you can extract the exact SQL query that triggered the crash:

This shows that a DELETE statement using a LEFT JOIN was involved, and the active database was database_x.

With this analysis, we now know:

• The exact SQL command that caused the crash
• Which database it was executed in

This step is often enough to correlate with logs, find the responsible application code, or try to reproduce the issue and report a bug if necessary.

Final thoughts

This workflow helped us diagnose a production crash caused by a query. Using GDB, we extracted the full query and schema, even though some information was unavailable in the logs.

This method is essential for:

• Root cause analysis
• Matching issues with known bugs (e.g., PS-9226)
• Creating reproducible test cases
• Supporting advanced crash diagnostics in Percona Support or engineering teams

This blog post aims to highlight some of what GDB can do for core dump analysis. GDB is a powerful tool that enables deep-level debugging, helping to identify and reproduce complex issues.

If you ever run into crashes, Percona Support can help you investigate and resolve them.