Disclaimer: This blog post is about migrating Percona Monitoring and Management 2 (PMM) data between PMM2 versions, and not for migrating data from PMM1 to PMM2. Restoring data from PMM1 to PMM2 is NOT supported since there were many architectural changes.
I recently worked on a customer case where he was not using a pmm-data container mounted in /srv as instructed in the official doc. PMM2 data is stored under /srv (changed from PMM1, which stored data in other directories and had different mounts for pmm-data), which means that if the content of /srv is not mounted on a separate container, upgrade or recreation of pmm-server will lose the existing data.
The procedure of backing up data stored in pmm-server and recreating pmm-server + pmm-data containers with the correct mount in /srv and no data loss is described below, but first, you should check if pmm-data is correctly mounted:
Checking if pmm-data is Holding /srv:
If you are using pmm-data container mounted on “Destination: /srv/”, you should see the following outputs while checking pmm-data:
1 2 3 | [root@centos vagrant]# docker inspect pmm-data | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data", "Destination": "/srv", |
And if pmm-server is using pmm-data, then the volume filename “Source” should match the one from pmm-data:
1 2 3 | [root@centos vagrant]# docker inspect pmm-server | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data", "Destination": "/srv", |
If you are not using pmm-data mounted in /srv, then the output will be showing a different mount (or no mount at all) such as:
1 2 3 | docker inspect pmm-server | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/e0b4cd1c5c65737a24ee10d9e84a6d17a0fc5beff028410c80599840d2c89e67/_data", "Destination": "/var/lib/grafana", |
Backup Existing Data
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # create directory for backing up current PMM2 data mkdir -p /root/pmm-data-backup/srv # stop pmm-server docker stop pmm-server # copy PMM2 data outside the container docker cp pmm-server:/srv /root/pmm-data-backup # rename container as a backup docker rename pmm-server pmm-server-backup # if having a pmm-data container without mount in /srv, then you should backup it up docker rename pmm-data pmm-data-backup |
Restore Into a New pmm-data Mounted in /srv:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | # create pmm-data container mounted in /srv docker create \ -v /srv \ --name pmm-data \ percona/pmm-server:2 /bin/true # restore data into pmm-data docker cp /root/pmm-data-backup/srv pmm-data:/ # start new pmm-container using pmm-data container sudo docker run -d \ -p 443:443 \ --volumes-from pmm-data \ --name pmm-server \ --restart always \ percona/pmm-server:2 # restore permissions docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown root:pmm /srv/clickhouse docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/grafana.db docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/png docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown pmm:pmm /srv/logs docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown postgres:postgres /srv/logs/postgresql.log docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus |
Results should be similar to this:
1 2 3 4 5 6 7 8 9 10 11 | docker exec -it pmm-server ls -l /srv total 12 drwxr-xr-x. 3 root root 47 Jun 9 12:49 alertmanager drwxr-xr-x. 9 root pmm 140 Jun 10 22:37 clickhouse drwxrwxr-x. 3 grafana grafana 69 Jun 10 22:27 grafana drwxrwxr-x. 2 pmm pmm 4096 Jun 10 22:26 logs drwxr-xr-x. 2 root root 115 Jun 9 12:48 nginx -rw-r--r--. 1 root root 6 Jun 9 12:50 pmm-distribution drwx------. 19 postgres postgres 4096 Jun 10 22:37 postgres drwxr-xr-x. 4 pmm pmm 31 Jun 9 12:50 prometheus drwxr-xr-x. 2 root root 6 Jun 9 12:49 update |
Finally, pmm-server needs to be restarted so that it reloads files with correct permissions:
1 2 | # restart pmm-server docker restart pmm-server |
Docker should now be working again with no data loss! And the backup of the containers can be safely removed with:
1 2 | docker rm pmm-server-backup docker rm pmm-data-backup |
Conclusion
It’s strongly suggested to use a separate pmm-data container mounted in /srv for PMM2 to avoid loss of historical data when upgrading/recreating pmm-server. Since the data path for PMM changed in PMM2 compared to PMM1, you might be using a pmm-data container with the wrong mounts (or no pmm-data at all!) which can be easily fixed by using the above procedure.
Comment (1)
Hi Carlos,
Can you kindly let me know how to get the O.S metrics like CPU,MEMORY for a cloudsql instance in pmm2