The hidepid Options for /proc and Percona XtraDB Cluster Don’t Play Well Together

hidepidIn this blog, we’ll look at how the hidepid options for /proc and Percona XtraDB Cluster can fight with one another.

One of the things I like about consulting at Percona is the opportunity to be exposed to unusual problems. I recently worked with a customer having issues getting SST to work with Percona XtraDB Cluster. A simple problem you would think. After four hours of debugging, my general feeling was that nothing made sense.

I added a bash trace to the SST script and it claimed MySQL died prematurely:

At the same time, from the MySQL error log MySQL was complaining the SST script died:

The Solution

Clearly, something odd was at play. But what? At that point, I decided to try a few operations with the mysql user. Finally, I stumbled onto something:

There are way more than 100 processes on these servers, so, why can’t the mysql user see them? Of course, the SST script monitors the state of its parent process using “ps”. Look at the bash trace above: 11244 is the mysqld pid. After a little Googling exercise, I found this blog post about the /proc hidepid mount option. Of course, the customer was using this option:

I removed the hidepid option using remount, and set hidepid=0 on all the nodes:

This simple command solved the issue. The SST scripts started to work normally. A good lesson learned: do not overlook security settings!

Share this post

Comment (1)

  • whyqaz

    But now you compromised the security of the server!

    I would have appreciated an alternative approach to fixing the issue.

    Be it, amending and editing the SST script to work with hidepid in mind or maybe better or the only answer would be to create an exclusion group and then you add group mysql to that group and (re)mount proc with gid=$EXCLUSTION_GROUP_ID_NO

    But I get what you saying. I.e. do not overlook security settings!

    P.s. Maybe it might be an idea to have pt-summary to recognize and alert that proc is mounted hidepid

    December 28, 2017 at 2:10 am

Comments are closed.