Percona Monitoring and Management 2.4.0 (PMM), released several days ago, added support for custom extensions of the Prometheus configuration file. Before explaining what that is and how to use it, let me tell you a bit of history.
PMM 1.4.0, released 2.5 years ago, added support for hooking external Prometheus exporters into PMM’s Prometheus configuration file. That was quite tricky to implement. Back then, information about monitoring services was stored in Consul, and the Prometheus configuration file
prometheus.yml contained consul_sd_config section for using it. We could not use the same mechanism for storing external exporters configuration as it wasn’t flexible enough – for example, there was no place to save custom scrape intervals or HTTP basic authentication credentials. At the same time, we could not move information about monitoring services out of Consul because pmm-admin relied on it being there. Then some customers depended on manual
prometheus.yml edits that should be preserved across PMM Server updates, including comments and their placements, a feature that wasn’t supported by our YAML parser library. In the end, we decided to add and remove scrape_configs sections for external exporters via a simple regexp-based text parser and templater. As you can imagine, patching a YAML configuration file with regular expressions is not the most stable approach; in fact, we broke it once just by replacing tabs with spaces. So, while designing PMM 2.0, we decided that the Prometheus configuration will be fully managed by PMM Server’s component called pmm-managed; its database (called Inventory database) is the single source of truth, and manual edits are not supported to avoid the abovementioned problems.
Prometheus Configuration Extensibility
Of course, we understood that Prometheus configuration extensibility is a very desirable feature for many of our users, and we are happy to announce the first iteration of it. While generating
prometheus.yml file, pmm-managed now checks if there is a file with a fixed path
/srv/prometheus/prometheus.base.yml. If it can be read and parsed, most information from it is preserved and used in the generated configuration:
global section is preserved if present,
rule_files arrays are merged together, and
remote_read settings are preserved as-is. For example, it allows one to use long-term storage for Prometheus data like VictoriaMetrics by adding the following to the base file:
- url: http://18.104.22.168:8428/api/v1/write
- job_name: victoria-metrics
Here we add VictoriaMetrics storage running on 22.214.171.124:8428 via remote_write. We also add monitoring for it. Then we should tell pmm-managed to generate a new combined file. The simplest way to do that is to use our change settings API with an empty body:
curl -X POST https://username:password@pmm-server/v1/Settings/Change
Normally, that API is used to update PMM Server configuration values like data retention interval, but it also causes pmm-managed to regenerate Prometheus configuration file, and reload it if needed. After running this command, you will notice a change in
Now you may wonder why I said that “most” configuration from the base file is preserved. While implementing this feature, we used Prometheus source code for configuration parser. Unfortunately, it is deeply tied to their service discovery mechanisms that include Kubernetes, and the Kubernetes dependencies list is bigger than the rest of PMM. So we decided to leave some things out of this release. Specifically, all service discovery mechanisms except
static_configs are not supported. In the next release, we will use our own code for configuration parser. It is a separate Go library that can be useful for the community.
Oh, and what about re-introducing proper external exporters that can be added with
pmm-admin? They are coming; I can’t comment on when exactly they will be shipped, but they are high in our priority list.
prometheus.base.yml is only the first step.