GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

Pidfile is not always created on percona-server start

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Pidfile is not always created on percona-server start

    Hi all

    I'm using percona-server, latests squeeze 5.1 version.

    I'm also using monit to monitor mysqld process. What I've relized is that mysqld does not always create the pidfile under /var/run/mysqld, monit thinks that it's down and tries to restart it or gives up. It also stops all the services that depend on mysql.

    Monit is not the issue. Manually starting mysqld via init.d script confirms that the pidfile is sometimes and sometimes not created.

  • #2
    This sounds like something else on the system is interacting with the server. Usually when the pid-file disappears, it's something like the following scenario:

    1. mysqld starts and writes the pid file
    2. something tries to start mysqld again, and it fails because of the existing instance; but it removes the pid file when it shuts down
    3. you notice the pid file is gone

    There can be many variations on such scenarios, but that's the general kind of thing I usually see.

    Comment


    • #3
      Hi Baron

      It might be a race condition between mysql starting and monit trying to start it at the same time. I'll try to reproduce it without monit.

      thanks.

      Comment


      • #4
        Yes, looks like mysqld_safe removes the pidfile. I took a look to the script and saw this:

        # If there exists an old pid file, check if the daemon is already running
        # Note: The switches to 'ps' may depend on your operating system
        if test -f "$pid_file"
        then
        PID=`cat "$pid_file"`
        if /bin/kill -0 $PID > /dev/null 2> /dev/null
        then
        if /bin/ps wwwp $PID | grep -v " grep" | grep -v mysqld_safe | grep -- "$MYSQLD" > /dev/null
        then # The pid contains a mysqld process
        log_error "A mysqld process already exists"
        exit 1
        fi
        fi
        rm -f "$pid_file"
        if test -f "$pid_file"
        then
        log_error "Fatal error: Can't remove the pid file:
        $pid_file
        Please remove it manually and start $0 again;
        mysqld daemon not started"
        exit 1
        fi
        fi


        During the startup, looks like monit wants to start mysql at the same time mysql is in start process.

        In the race condition looks like mysql has created the pidfile but the process is not yet listed in ps. So the pidfile is detected as old pidfile and removed.

        Can this remove be safely deleted from the script? If mysql is started and an old pidfile exists, wouldn't mysql overwrite it?

        Comment


        • #5
          I have a similar problem with startup from sysv init.d script. When I run /etc/init.d/mysql start, mysql fails to write a pid file. Actually, doing some debugging inside the init.d script shows that mysql starts, but fails to spawn the first child:

          Starting MySQL (Percona Server)/usr/bin/mysqld_safe --datadir=/ssd/mysqldata --pid-file=/ssd/mysqldata/test.com.pid >/dev/null 2>&1 &
          /ssd/mysqldata/test.com.pid

          This is the results of 'ps ax | grep mysql' run inside the "wait_for_pid" function in the init script:

          2100 pts/0 S+ 0:00 /bin/sh /etc/init.d/mysql start
          2109 ? R 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/ssd/mysqldata --pid-file=/ssd/mysqldata/test.com.pid
          2111 pts/0 S+ 0:00 grep mysql

          this is an ls of the data directory where there should be a pid file

          ibdata1 ib_logfile1 performance_schema RPM_UPGRADE_MARKER-LAST sj-slow.log
          ib_logfile0 mysql RPM_UPGRADE_HISTORY test.com.err test

          however, running the exact same command at the command line starts mysql just fine AND creates the pid file:

          /usr/bin/mysqld_safe --datadir=/ssd/mysqldata --pid-file=/ssd/mysqldata/test.com.pid >/dev/null 2>&1 &

          the error log shows nothing, this is a fresh server with nothing else running on it.

          Comment

          Working...
          X