Date of release: 19 August 2026
Severity: High
Affected product: PMM
Impacted versions: 3.9.0 and below
Percona has recently been made aware of a security vulnerability affecting PMM. We take the security of our products and the protection of our customers’ data with the utmost seriousness.
This advisory describes the vulnerability, the immediate steps you can take to protect your deployment, and the permanent fix.
PMM’s Grafana instance allows signed-in users, including those with the Viewer role, to call raw data source APIs. If anonymous access has been explicitly enabled (it is off by default), unauthenticated users can also reach these APIs.
Through the Grafana ClickHouse data source, such a user can submit arbitrary SQL. The data source connects to ClickHouse as the default identity, which holds global DDL, DML, and SOURCES privileges and can make outbound HTTP requests.
Chained together, this allows an attacker to reach AWS IMDSv1, obtain a live EC2 role session, read a Terraform remote-state object from S3, and authenticate as the PMM/Grafana administrator.
An unauthenticated attacker can cross the public Grafana boundary into internal databases, AWS instance metadata, S3 remote state, and the PMM administrator account. This yields renewable cloud credentials and a remote-state file that can contain many independently reusable secrets and private infrastructure details.
The severity of impact depends on deployment configuration. The full IMDS-to-credential chain requires anonymous access to be enabled (off by default) and the PMM Server to be running on AWS EC2 with IMDSv1.
All deployments are affected by the arbitrary SQL execution via the ClickHouse data source.
This vulnerability is fixed in PMM 3.9.1, scheduled for release on August 19, 2026. Upgrade as soon as it is available.
If you cannot upgrade immediately, run the script below to close the exploitation path. It creates a least-privilege ClickHouse user for Grafana and points the ClickHouse data source at it, replacing the default superuser and making the exploitation of the vulnerability impossible.
Before running the script, back up your PMM Server and save the pmm-data volume.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
#!/bin/bash # Create a least-privilege ClickHouse identity for Grafana and point the # ClickHouse datasource at it, replacing the default superuser. set -euo pipefail CONTAINER=${CONTAINER:-pmm-server} PMM_HOST=${PMM_HOST:-localhost} PMM_PORT=${PMM_PORT:-443} GRAFANA_URL="https://${PMM_HOST}:${PMM_PORT}" ADMIN_PASS=${ADMIN_PASS:-$(cat /root/pmm-admin-password)} # Drop-ins are loaded from users.d (users_config defaults to users.xml -> # users.d), NOT default-users.d. BOOTSTRAP_XML=/etc/clickhouse-server/users.d/zz-provision-bootstrap.xml CH_PASS=$(openssl rand -hex 24) CH_HASH=$(printf '%s' "$CH_PASS" | sha256sum | awk '{print $1}') BOOT_PASS=$(openssl rand -hex 24) BOOT_HASH=$(printf '%s' "$BOOT_PASS" | sha256sum | awk '{print $1}') ch_wait () { local user=$1 pass=$2 i for i in $(seq 1 45); do if docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \ --user "$user" --password "$pass" -q "SELECT 1" >/dev/null 2>&1; then return 0 fi sleep 2 done echo "ERROR: clickhouse did not accept $user within 90s" >&2 return 1 } # PMM's ClickHouse default superuser has access_management disabled, so it # cannot run CREATE USER / GRANT even with its known password. Install a # short-lived admin to run the DDL instead. Drop-ins must live in users.d; # and a plaintext <password> is rejected outright at startup because PMM # ships allow_plaintext_password=0. docker exec -u root "$CONTAINER" mkdir -p /etc/clickhouse-server/users.d docker exec -u root -i "$CONTAINER" bash -c "cat > $BOOTSTRAP_XML" <<XMLEOF <clickhouse> <users> <provision_admin> <password_sha256_hex>$BOOT_HASH</password_sha256_hex> <networks><ip>127.0.0.1</ip><ip>::1</ip></networks> <profile>default</profile> <quota>default</quota> <access_management>1</access_management> </provision_admin> </users> </clickhouse> XMLEOF docker exec -u root "$CONTAINER" chown pmm:root "$BOOTSTRAP_XML" docker exec -u root "$CONTAINER" supervisorctl restart clickhouse ch_wait provision_admin "$BOOT_PASS" # grafana_ro holds SELECT and nothing else. Without the SOURCES family it # cannot call url(), s3(), mongodb(), remote() or file(); readonly=1 # additionally prevents it overriding server settings such as # max_http_get_redirects. ALTER runs unconditionally so that re-running # this script rotates the password rather than failing. docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \ --user provision_admin --password "$BOOT_PASS" --multiquery <<SQLEOF CREATE SETTINGS PROFILE IF NOT EXISTS grafana_ro_profile SETTINGS readonly = 1, allow_ddl = 0, max_execution_time = 60; CREATE USER IF NOT EXISTS grafana_ro IDENTIFIED WITH sha256_hash BY '$CH_HASH'; ALTER USER grafana_ro IDENTIFIED WITH sha256_hash BY '$CH_HASH' SETTINGS PROFILE grafana_ro_profile; REVOKE ALL ON *.* FROM grafana_ro; GRANT SELECT ON pmm.* TO grafana_ro; GRANT SELECT ON default.* TO grafana_ro; GRANT SELECT ON system.tables TO grafana_ro; GRANT SELECT ON system.columns TO grafana_ro; GRANT SELECT ON system.databases TO grafana_ro; GRANT SELECT ON system.one TO grafana_ro; GRANT SELECT ON system.numbers TO grafana_ro; SQLEOF docker exec -u root "$CONTAINER" rm -f "$BOOTSTRAP_XML" docker exec -u root "$CONTAINER" supervisorctl restart clickhouse ch_wait grafana_ro "$CH_PASS" # Repoint the datasource. The UID is assigned by PMM, so look it up. DS_UID=$(curl -sk -u "admin:$ADMIN_PASS" "$GRAFANA_URL/graph/api/datasources" \ | jq -r '.[] | select(.type == "grafana-clickhouse-datasource") | .uid' | head -1) if [ -z "$DS_UID" ]; then echo "ERROR: no grafana-clickhouse-datasource found" >&2 exit 1 fi # Transient files hold the CH password; keep them in a private dir and # always remove them, even if a curl below fails. umask 077 TMPD=$(mktemp -d) trap 'rm -rf "$TMPD"' EXIT curl -sk -u "admin:$ADMIN_PASS" \ "$GRAFANA_URL/graph/api/datasources/uid/$DS_UID" > "$TMPD/ds-ch.json" jq --arg p "$CH_PASS" \ '.jsonData.username = "grafana_ro" | .secureJsonData.password = $p' \ "$TMPD/ds-ch.json" > "$TMPD/ds-ch.new.json" curl -sk -u "admin:$ADMIN_PASS" -X PUT -H 'Content-Type: application/json' \ -d @"$TMPD/ds-ch.new.json" \ "$GRAFANA_URL/graph/api/datasources/uid/$DS_UID" >/dev/null # Fail the build rather than come up believing this worked. if docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \ --user grafana_ro --password "$CH_PASS" \ -q "SELECT count() FROM url('http://169.254.169.254/latest/user-data','LineAsString','line String')" \ >/dev/null 2>&1; then echo "FATAL: grafana_ro can still reach url()" >&2 exit 1 fi if docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \ --user grafana_ro --password "$CH_PASS" \ -q "CREATE TABLE default.zz_provision_check (x String) ENGINE=Memory" \ >/dev/null 2>&1; then echo "FATAL: grafana_ro can still run DDL" >&2 exit 1 fi unset CH_PASS BOOT_PASS ADMIN_PASS echo "ClickHouse datasource now authenticates as grafana_ro." |
Run the script as follows:
|
1 |
CONTAINER=pmm-server PMM_HOST=localhost PMM_PORT=443 ADMIN_PASS=XXXXX bash ./pmm-ch-user.sh |
If you require further clarification or assistance, we are available 24/7:
For questions about this advisory, upgrade planning, or to discuss options for unsupported major versions, open a case via the Percona Customer Portal or contact your Percona Customer Success Manager.
For other security-related questions, write to [email protected].
Resources
RELATED POSTS
