Resolving MongoDB Stack Traces

MongoDB Stack TracesWhen a MongoDB server crashes you will usually find what is called a “stack trace” in its log file. But what is it and what purpose does it have? Let’s simulate a simple crash so we can dig into it.

Crashing a test server

In a test setup with a freshly installed MongoDB server, we connect to it and create some test data:

On a separate connection to the server, we change the ownership of the MongoDB data files, so the mongod user will no longer have access to them:

Going back to the mongo session, we try to add a new record and it fails, as expected:

Looking at the error log we confirm the server has crashed, leaving a stack trace (also called a “backtrace”) behind:

But what can we infer from these somewhat cryptic lines full of hexadecimal content?

Inspecting the MongoDB stack trace

In the bottom of the stack trace, we can see a list of function names and addresses. Note the resolution of most functions worked reasonably well in the example above; the mongod binary used by our test server is not stripped of symbols (if yours is you will need to install the respective debugsymbols/debuginfo package and use the mongod binary provided by it to resolve the stack trace):

This means we can actually extract them from the mongod binary with the help of a tool such as nm, from GNU Development Tools:

Function names appear all mangled though:

We can use another tool from that same toolkit, c++filt, to get them straightened, for example:

In fact, we can process the whole stack trace with c++filt all at once…

… and get it fully demangled, with C++ function and method names easily recognizable now:

While easily reproducible, this was not a very interesting example: the change in the database files ownership caused WiredTiger to crash upon the insert without leaving much trace behind. Let’s have a look at another one.

A more realistic example

Despite being somewhat old, bug SERVER-13751 (mongod crash on geo nearSphere query) provides a realistic yet easy to reproduce example of a simple routine that crashed MongoDB 2.6.0 (this bug is, in fact, a duplicate of SERVER-13666, but it provides a simpler test case). Here’s how to get to it.

1) First, we download these old binaries and start a MongoDB server:

2) In a second terminal window, we connect to the MongoDB server we just started and run a more simplified version of the routine described in the bug, which consists of creating a 2dsphere index and querying for a point described with invalid coordinates:

Now when we look back at the first terminal we find the server has crashed, leaving the following stack trace:

Processing  the stack trace with c++filt we get:

This particular mongod binary is stripped of symbols:

So in order to resolve the stack trace, we need to first obtain one that is not:

We can now extract the actual function names from the addresses between the brackets using addr2line (option “f” provides the function name and if we also use “i” we get the preceding ones as well if the main one was inline; option “C” provides some extent of demangling, similar to c++filt):

One of the greatest values of working with Open Source software is being able to have a direct look at this exact piece of code, which translates to https://github.com/mongodb/mongo/blob/v2.6/src/third_party/s2/s2cap.cc#L35 :

Note that the actual fix for this bug didn’t come from modifying this function, which is being re-used from a third-party (another beauty of working with Open Source!), but in making sure the arguments that are being passed to it (which compose the point’s coordinates) are validated beforehand.

There is a home for bugs

If you ever run into a MongoDB server crash, I hope this little set of instructions can serve as a reference in helping you make sense of the stack trace that will (hopefully) have been left behind. You can then search for bugs at https://jira.mongodb.org if you’re running a MongoDB server, or at https://jira.percona.com/projects/PSMDB if you’re running Percona Server for MongoDB. If you can’t find a bug that matches your crash, please consider filing a new one; providing a clear stack trace alongside the exact binary version you’re using is a must. If you are able to reproduce the problem at will and can provide a reproducible test case as well, like the ones we showed above, that will not only make the life of our developers easier, it also increases the likelihood of getting the bug fixed much, much faster.

Learn more about Percona Server for MongoDB

Share this post

Comment (1)

  • RajeshRam Reply

    thanks for the information guys. Sometimes you might not find a particular app that you are looking for in the apple app store. But I’m sure there are other ways to get a certain type of iOS apps. So today I will tell you about App valley which is a great third-party app where you can get much more than what you will get in the Apple App store. https://appvalleyvip.com/

    July 25, 2019 at 3:34 am

Leave a Reply