If PMM server wasn’t updated properly, or if you have concerns about the release, you can force the update process in 2 ways:
From the UI - Home panel: click with the Alt key on the reload icon in the Update panel (IMG needed) to make the Update Button visible even if you are on the same version as available for update. Pressing this button will force the system to rerun the update so that any broken or not installed components can be installed. In this case, you’ll go through the usual update process with update logs and successful messages at the end.
By API call (if UI not available): You can call the Update API directly with:
curl --user admin:admin --request POST 'http://PMM_SERVER/v1/Updates/Start'
admin:adminwith your username/password, and replace
PMM_SERVERwith your server address.
You will not see the logs using this method.
Refresh The Home page in 2-5 min and you should see that PMM was updated.
PMM Server/PMM Client connection¶
Broken network connectivity may be due to many reasons. Particularly, when using Docker, the container is constrained by the host-level routing and firewall rules. For example, your hosting provider might have default
iptables rules on their hosts that block communication between PMM Server and PMM Client, resulting in DOWN targets in VictoriaMetrics. If this happens, check the firewall and routing settings on the Docker host.
PMM is also able to generate diagnostics data which can be examined and/or shared with our support team to help quickly solve an issue. You can get collected logs from PMM Client using the
pmm-admin summary command.
Logs obtained in this way includes PMM Client logs and logs which were received from the PMM Server, stored separately in the
server folders. The
server folder also contains its own
client subfolder with the self-monitoring client information collected on the PMM Server.
Beginning with PMM version 2.4.0, there is an additional flag that enables the fetching of
pprof debug profiles and adds them to the diagnostics data. To enable, run
pmm-admin summary --pprof.
You can get PMM Server logs in two ways:
- In a browser, visit
- Go to PMM > PMM Settings and click Download server diagnostics. (See Diagnostics in PMM Settings.)
No Integrated Alerting icon¶
You are not logged in as a privileged user. You need either Admin or Editor roles to work with Integrated Alerting.
Integrated Alerting icon but no submenu¶
Integrated Alerting isn’t activated.
- Go to PMM → PMM Settings → Advanced Settings
- Enable Integrated Alerting
Unreachable external IP addresses¶
When I get an email or page from my system the IP is not reachable from outside my organization how do I fix this?
You can configure your PMM Server’s Public Address by navigating to PMM → PMM Settings → Advanced Settings, and supply an address to use in your alert notifications.
What is ‘Alertmanager integration’?¶
There’s already an Alertmanager integration tab without me turning it on, I know because I was using your existing Alertmanager integration.
This will continue to work but will be renamed External Alertmanager.
Notification channels not working¶
I tried to setup a Slack/Email channel but nothing happened
Before you can use a notification channel you must provide your connection details.
- Go to PMM → PMM Settings→ Communication
- Define your SMTP server or Slack incoming webhook URL
For PagerDuty you can configure in the notification channel tab of Integrated Alerting by supplying your server/routing key.
What’s the difference: Username/Password vs Identity/Secret¶
In configuring my email server I’m being asked for a Username and Password as well as Identity and Secret. What is the difference between these and which do I use or do I need both?
It depends on what kind of authentication your system uses:
- LOGIN: Use Username/Password
- PLAIN: Use either Username or Identity and Password
- CRAM-MD5: Use Username and Secret
Alert Rule Templates is disabled¶
Built-In alerts are not editable.
However, you can copy them and edit the copies. (PMM >=2.14.0).
If you create a custom alert rule template you will have access to edit.
I’m ready to create my first rule! I’ve chosen a template and given it a name…what is the format of the fields?
Threshold - float value, it has different meanings depending on what template is used
Duration - The duration the condition must be satisfied in seconds
Filters - A Key, Evaluator, and Value. E.g.
Key must be an exact match. You can find a complete list of keys by using the Explore main menu item in PMM
Evaluator can be any of:
Value is an exact match or when used with a ‘fuzzy’ evaluator (=~) can be a regular expression. E.g.
Variables in Templates¶
The concept of “template” implies things like variable substitutions…where can I use these? Where can I find a complete list of them?
Here is a guide to creating templates for Alertmanager: https://prometheus.io/docs/prometheus/latest/configuration/template_examples/
- Page updated 2021-03-19