Disclaimer: This blog post is about migrating Percona Monitoring and Management 2 (PMM) data between PMM2 versions, and not for migrating data from PMM1 to PMM2. Restoring data from PMM1 to PMM2 is NOT supported since there were many architectural changes.
I recently worked on a customer case where he was not using a pmm-data container mounted in /srv as instructed in the official doc. PMM2 data is stored under /srv (changed from PMM1, which stored data in other directories and had different mounts for pmm-data), which means that if the content of /srv is not mounted on a separate container, upgrade or recreation of pmm-server will lose the existing data.
The procedure of backing up data stored in pmm-server and recreating pmm-server + pmm-data containers with the correct mount in /srv and no data loss is described below, but first, you should check if pmm-data is correctly mounted:
If you are using pmm-data container mounted on “Destination: /srv/”, you should see the following outputs while checking pmm-data:
|
1 |
[root@centos vagrant]# docker inspect pmm-data | egrep "Source|Destination"<br> "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data",<br> "Destination": "/srv", |
And if pmm-server is using pmm-data, then the volume filename “Source” should match the one from pmm-data:
|
1 |
[root@centos vagrant]# docker inspect pmm-server | egrep "Source|Destination"<br> "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data",<br> "Destination": "/srv", |
If you are not using pmm-data mounted in /srv, then the output will be showing a different mount (or no mount at all) such as:
|
1 |
docker inspect pmm-server | egrep "Source|Destination"<br> "Source": "/var/lib/docker/volumes/e0b4cd1c5c65737a24ee10d9e84a6d17a0fc5beff028410c80599840d2c89e67/_data",<br> "Destination": "/var/lib/grafana", |
|
1 |
# create directory for backing up current PMM2 data<br>mkdir -p /root/pmm-data-backup/srv <br><br># stop pmm-server<br>docker stop pmm-server <br><br># copy PMM2 data outside the container<br>docker cp pmm-server:/srv /root/pmm-data-backup<br><br># rename container as a backup<br>docker rename pmm-server pmm-server-backup<br><br># if having a pmm-data container without mount in /srv, then you should backup it up <br>docker rename pmm-data pmm-data-backup |
|
1 |
# create pmm-data container mounted in /srv<br>docker create <br> -v /srv <br> --name pmm-data <br> percona/pmm-server:2 /bin/true<br><br># restore data into pmm-data<br>docker cp /root/pmm-data-backup/srv pmm-data:/<br><br># start new pmm-container using pmm-data container<br>sudo docker run -d <br> -p 443:443 <br> --volumes-from pmm-data <br> --name pmm-server <br> --restart always <br> percona/pmm-server:2<br><br># restore permissions<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown root:pmm /srv/clickhouse<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/grafana.db<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/png<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown pmm:pmm /srv/logs<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown postgres:postgres /srv/logs/postgresql.log<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres<br>docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus |
Results should be similar to this:
|
1 |
docker exec -it pmm-server ls -l /srv<br>total 12<br>drwxr-xr-x. 3 root root 47 Jun 9 12:49 alertmanager<br>drwxr-xr-x. 9 root pmm 140 Jun 10 22:37 clickhouse<br>drwxrwxr-x. 3 grafana grafana 69 Jun 10 22:27 grafana<br>drwxrwxr-x. 2 pmm pmm 4096 Jun 10 22:26 logs<br>drwxr-xr-x. 2 root root 115 Jun 9 12:48 nginx<br>-rw-r--r--. 1 root root 6 Jun 9 12:50 pmm-distribution<br>drwx------. 19 postgres postgres 4096 Jun 10 22:37 postgres<br>drwxr-xr-x. 4 pmm pmm 31 Jun 9 12:50 prometheus<br>drwxr-xr-x. 2 root root 6 Jun 9 12:49 update |
Finally, pmm-server needs to be restarted so that it reloads files with correct permissions:
|
1 |
# restart pmm-server<br>docker restart pmm-server |
Docker should now be working again with no data loss! And the backup of the containers can be safely removed with:
|
1 |
docker rm pmm-server-backup<br>docker rm pmm-data-backup |
It’s strongly suggested to use a separate pmm-data container mounted in /srv for PMM2 to avoid loss of historical data when upgrading/recreating pmm-server. Since the data path for PMM changed in PMM2 compared to PMM1, you might be using a pmm-data container with the wrong mounts (or no pmm-data at all!) which can be easily fixed by using the above procedure.
Resources
RELATED POSTS
