Extending PMM’s Prometheus Configuration

Extending PMM Prometheus ConfigurationPercona Monitoring and Management 2.4.0 (PMM), released several days ago, added support for custom extensions of the Prometheus configuration file. Before explaining what that is and how to use it, let me tell you a bit of history.

PMM 1.4.0, released 2.5 years ago, added support for hooking external Prometheus exporters into PMM’s Prometheus configuration file. That was quite tricky to implement. Back then, information about monitoring services was stored in Consul, and the Prometheus configuration file prometheus.yml contained consul_sd_config section for using it. We could not use the same mechanism for storing external exporters configuration as it wasn’t flexible enough – for example, there was no place to save custom scrape intervals or HTTP basic authentication credentials. At the same time, we could not move information about monitoring services out of Consul because pmm-admin relied on it being there. Then some customers depended on manual prometheus.yml edits that should be preserved across PMM Server updates, including comments and their placements, a feature that wasn’t supported by our YAML parser library. In the end, we decided to add and remove scrape_configs sections for external exporters via a simple regexp-based text parser and templater. As you can imagine, patching a YAML configuration file with regular expressions is not the most stable approach; in fact, we broke it once just by replacing tabs with spaces. So, while designing PMM 2.0, we decided that the Prometheus configuration will be fully managed by PMM Server’s component called pmm-managed; its database (called Inventory database) is the single source of truth, and manual edits are not supported to avoid the abovementioned problems.

Prometheus Configuration Extensibility

Of course, we understood that Prometheus configuration extensibility is a very desirable feature for many of our users, and we are happy to announce the first iteration of it. While generating prometheus.yml file, pmm-managed now checks if there is a file with a fixed path /srv/prometheus/prometheus.base.yml. If it can be read and parsed, most information from it is preserved and used in the generated configuration: global section is preserved if present, scrape_configs and rule_files arrays are merged together, and alerting, remote_write, and remote_read settings are preserved as-is. For example, it allows one to use long-term storage for Prometheus data like VictoriaMetrics by adding the following to the base file:

Here we add VictoriaMetrics storage running on via remote_write. We also add monitoring for it. Then we should tell pmm-managed to generate a new combined file. The simplest way to do that is to use our change settings API with an empty body:

Normally, that API is used to update PMM Server configuration values like data retention interval, but it also causes pmm-managed to regenerate Prometheus configuration file, and reload it if needed. After running this command, you will notice a change in /etc/prometheus.yml.

Now you may wonder why I said that “most” configuration from the base file is preserved. While implementing this feature, we used Prometheus source code for configuration parser. Unfortunately, it is deeply tied to their service discovery mechanisms that include Kubernetes, and the Kubernetes dependencies list is bigger than the rest of PMM. So we decided to leave some things out of this release. Specifically, all service discovery mechanisms except static_configs are not supported. In the next release, we will use our own code for configuration parser. It is a separate Go library that can be useful for the community.

Oh, and what about re-introducing proper external exporters that can be added with pmm-admin? They are coming; I can’t comment on when exactly they will be shipped, but they are high in our priority list. prometheus.base.yml is only the first step.

Share this post

Comments (2)

  • Sids Reply

    Does PMM 2.x support external Prometheus exporter monitoring ?! As I couldn’t find anything in doc as well version 2 pmm-admin client binary doesn’t list external: service/instance which is there in PMM 1.x..

    March 23, 2020 at 7:27 pm

Leave a Reply