Capturing Per-Process Metrics with Percona Monitoring and Management (PMM)

Capturing Per-Process Metrics with Percona Monitoring and Management (PMM)

PREVIOUS POST
NEXT POST

In this blog post, I will show you how to use Percona Monitoring and Management (PMM) to capture per-process metrics in five minutes or less.

While Percona Monitoring and Management (PMM) captures a lot of host metrics, it currently falls short providing per-process information, such as which particular process uses a lot of CPU resources, causes Disk IO or consumes a lot of memory.

In our database performance optimization and troubleshooting practice, this information has proven quite useful in many cases: batch jobs taking much more resources than developers would estimate and misconfigured Percona XtraBackup or Percona Toolkit are among the most common offenders.

Per-process metrics information can also be very helpful when troubleshooting database software memory leaks or memory fragmentation.

You don’t know which processes cause you problems at the outset, so it is important to capture information about all of the processes (or specifically exclude the processes you do not want to capture information about) rather than capture information about selected few.

While capturing such helpful information is not available in PMM out of the box (yet), you can easily achieve it using PMM’s External Exporter support and the excellent Prometheus Process Exporter by Nick Cabatoff.

These instructions are for Debian/Ubuntu  Linux Distributions but they should work with RedHat/CentOS based versions as well – just use RPM package instead of DEB

1: Download the process exporter packages from GitHub:

2: Install the package

(Note: the file will be different depending on the platform and current release.)

3: Run the Exporter

4: Register Exporter with Percona Monitoring and Management

Assuming the current node is already monitored by PMM you just need one command:

This captures process metrics every 10 seconds (adjust interval if desired).

Important note: due to some internal limitations, you need to use a different service name (“processes-my-host”)  for each host. I suggest just adding the hostname to the descriptive name “processes” for simplicity.

5: Get Matching Dashboard from Grafana.com

While you can browse the data captured by the Advanced Data Exploration Dashboard, it is not any fun. I created a PMM-style dashboard and published it on Grafana.com. I based it on Nick’s original dashboard.

To add this dashboard to your PMM Server, click Dashboard Search on your PMM Server.

PMM Per-Process Metrics

From there, click on “Import Dashboard”. Use 6033 as the Grafana.com Dashboard ID.

PMM Per-Process Metrics

6: You’re done!

You should have data flowing, and you should be able to see the data on the graphs.

PMM Per-Process Metrics

In this example, I have pt-query-digest (shown as Perl) parsing the log file and pushing MySQL Server away from memory.

Note, as you likely have many processes on the system, the graphs are designed to show only the top processes. All running processes, however, are available in the drop-down if you want to access the history for a specific process.

Let us know what you think. We are looking at how to integrate this functionality directly into Percona Monitoring and Management!

PREVIOUS POST
NEXT POST

Share this post

Comments (5)

  • Sergei Reply

    Great!

    May 22, 2018 at 5:43 am
  • Jeffery Smith Reply

    I cannot seem to find the answer to the error I am encountering. This is the command I am using:
    pmm-admin add external:service process-exporter –service-port=9256 –interval=30s –verbose

    These are the errors I am receiving:
    {“error”:”scrape config with job name \”process-exporter\” not found”,”message”:”scrape config with job name \”process-exporter\” not found”,”code”:5}
    {“error”:”{local IP}:9256: Get http://{local IP}:9256/metrics: dial tcp {local IP}:9256: i/o timeout”,”message”:”{local IP}:9256: Get http://{local IP}:9256/metrics: dial tcp {local IP}:9256: i/o timeout”,”code”:9}

    December 20, 2018 at 2:28 pm
    • Roma Novikov Reply

      Jeffery, can you check “http://{local IP}:9256/metrics” from the host? Is this URL accessible and provide some data?

      Also check “pmm-admin list ” command so see what you already has

      December 27, 2018 at 1:05 pm
  • Jeffery Smith Reply

    Hi Roma, thank you very much for your reply and I apologize for the long delay in responding. I can confirm that the “http://{local IP}:9256/metrics” link is working locally and when I call it using “curl” directly on the system I receive a lengthy reply that looks like the data I would expect from the process-exporter.

    When I run pmm-admin list I only see the linux:metrics' service, which is working. I am unable to get the service added using this command:

    sudo pmm-admin add external:service processes-my-host --service-port=9256 --interval=10s --verbose

    With the –verbose` flag, I see this error show up during the process:

    {“error”:”scrape config with job name \”processes-my-host\” not found”,”message”:”scrape config with job name \”processes-my-host\” not found”,”code”:5}

    January 9, 2019 at 1:45 pm
  • Jeffery Smith Reply

    Hello again Roma, I feel a little bit silly. I turns out to have been a firewall restriction. Once I opened port 9256 between the PMM server and the client, it seems to have started working.

    January 9, 2019 at 2:40 pm

Leave a Reply