I recently worked with a customer who had a weird issue: when their MySQL server was started (Percona Server 5.5), if they try to run
service mysql start a second time, the init script was not able to detect that an instance was already running. As a result, it tried to start a second instance with the same settings as the first one. Of course this fails and this creates a mess. What was the issue? A missing rule in SELinux. At least it looks like
If SELinux is set to enforcing and if you are using Percona Server on CentOS/RHEL 6 (other versions could be affected),
service mysql start doesn’t work properly and a fix is simple to run:
# grep mysqld_safe /var/log/audit/audit.log | audit2allow -M mysqld_safe
# semodule -i mysqld_safe.pp
# service mysql restart
Other options are:
- Set SELinux to permissive
- Use the CentOS/RHEL standard MySQL init script (note I didn’t extensively check if that could trigger other errors)
How did we see the issue?
That was pretty easy: if an instance is already running and if you run
service mysql start again, you should see something like this in the MySQL error log:
150717 08:47:58 mysqld_safe A mysqld process already exists
But if you rather see tons of error messages like:
2015-07-17 08:47:05 27065 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2015-07-17 08:47:05 27065 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
it means that the init script is broken somewhere.
When the issue was brought to my attention, I tried to reproduce it on my local box, but with no luck. What was so special in the configuration used by the customer?
The only thing that was slightly out of the ordinary was SELinux which was set to enforcing. Then we set SELinux to permissive, and guess what?
service mysql start was now working properly and it didn’t allow 2 concurrent instances to be run!
Next step was to look at the SELinux logs to find any error related to MySQL, and we discovered messages like:
type=SYSCALL msg=audit(1437121845.464:739): arch=c000003e syscall=62 success=no exit=-13
a0=475 a1=0 a2=0 a3=7fff0e954130 items=0 ppid=1 pid=5732 auid=500 uid=0 gid=0 euid=0 suid=0
fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm="mysqld_safe" exe="/bin/bash"
At this point, we knew that a rule was missing for
mysqld_safe, we needed to add a new one.
Actually what happens is that SELinux prevents this condition to be executed in mysqld_safe:
if kill -0 $PID > /dev/null 2> /dev/null
and then the script assumes that this means the mysqld process is not running. That’s why a second mysqld is started.
However users of Oracle MySQL will probably never experience that issue, simply because the init script is a bit different: before calling mysqld_safe, the init script tries to ping a potential mysqld instance and if it gets a positive reply or an
Access denied error, it concludes that mysqld is already running and it doesn’t invoke mysqld_safe.
Fortunately, this is quite simple. You can generate the corresponding rule with
grep mysqld_safe /var/log/audit/audit.log | audit2allow -M mysqld_safe
And after checking the corresponding .te file, we were able to load that new module:
semodule -i mysqld_safe.pp
After stopping MySQL, you can now use
service mysql start normally.
This issue was quite interesting to work on because finding the culprit was not that easy. Also it only triggers when SELinux is enabled and Percona Server is used. Now should the init script of Percona Server be fixed? I’m not sure of the potential problems that could occur if we did so, but of course feel free to leave your feedback in the comments.