In this blog post, we’ll see how to use different mount points on PMM Docker deployments (Percona Monitoring and Management). This is useful if you want to use other mount points for the different directories, or even if you want to use a custom path that is not bound to Docker’s volumes directory (which is /var/lib/docker/volumes/ by default) within the same mount point.
There are two ways in which you can achieve this:
docker create command to use different directoriesIn the following examples, /pmm/ is used as the new base directory. One can, of course, choose different directories for each if needed. Also, remember to be aware of any SELinux or AppArmor policies you may have in place.
For this, we need to follow these steps:
Let’s see this with some commands and outputs. In this example, we will use /pmm/ as if it were the new mount point:
|
1 2 3 |
shell> mkdir -p /pmm/opt/ shell> mkdir -p /pmm/opt/prometheus shell> mkdir -p /pmm/var/lib/ |
|
1 2 3 4 5 6 7 8 |
shell> docker create -v /opt/prometheus/data -v /opt/consul-data -v /var/lib/mysql -v /var/lib/grafana --name pmm-data percona/pmm-server:1 /bin/true 4589cd1bf8ce365f8f62eab9f415eb14f1ce3a76b0123b7aad42e93385455303 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
shell> docker inspect pmm-data | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/a191331f6be1a177003ef2fdeee53f92fc190dc67b0c402ee7b47b4461ffa522/_data", "Destination": "/opt/prometheus/data", "Source": "/var/lib/docker/volumes/7208317edff4565f649df294cfb05fc1888e6ab817c18abc5f036c419e364d4b/_data", "Destination": "/var/lib/grafana", "Source": "/var/lib/docker/volumes/547b3f083a0a33b6cd75eb72e2cc25c383f5d4db2d8a493b25eb43499e2f5807/_data", "Destination": "/var/lib/mysql", "Source": "/var/lib/docker/volumes/7473ac5d2dac4440ac94fae2faf4a63af95baaabed4b14d9414f499ae9b5761d/_data", "Destination": "/opt/consul-data", shell> DOCKER_CONSUL_DATA="/var/lib/docker/volumes/7473ac5d2dac4440ac94fae2faf4a63af95baaabed4b14d9414f499ae9b5761d/_data" shell> DOCKER_PROMETHEUS_DATA="/var/lib/docker/volumes/a191331f6be1a177003ef2fdeee53f92fc190dc67b0c402ee7b47b4461ffa522/_data" shell> DOCKER_GRAFANA_DATA="/var/lib/docker/volumes/7208317edff4565f649df294cfb05fc1888e6ab817c18abc5f036c419e364d4b/_data" shell> DOCKER_MYSQL_DATA="/var/lib/docker/volumes/547b3f083a0a33b6cd75eb72e2cc25c383f5d4db2d8a493b25eb43499e2f5807/_data" shell> mv $DOCKER_CONSUL_DATA /pmm/opt/consul-data shell> mv $DOCKER_PROMETHEUS_DATA /pmm/opt/prometheus/data shell> mv $DOCKER_GRAFANA_DATA /pmm/var/lib/grafana shell> mv $DOCKER_MYSQL_DATA /pmm/var/lib/mysql |
|
1 2 3 4 |
shell> ln -s /pmm/opt/consul-data $DOCKER_CONSUL_DATA shell> ln -s /pmm/opt/prometheus/data $DOCKER_PROMETHEUS_DATA shell> ln -s /pmm/var/lib/grafana $DOCKER_GRAFANA_DATA shell> ln -s /pmm/var/lib/mysql $DOCKER_MYSQL_DATA |
After this, we can start the pmm-server container (see below).
Modifying the docker create command
For this, we need to follow these other steps:
-v arguments)Let’s see this in practical terms again, assuming we want to use the /pmm/ mount point.
|
1 2 3 |
shell> mkdir -p /pmm/opt/ shell> mkdir -p /pmm/opt/prometheus shell> mkdir -p /pmm/var/lib/ |
|
1 2 3 4 5 6 7 8 |
shell> docker create -v /opt/prometheus/data -v /opt/consul-data -v /var/lib/mysql -v /var/lib/grafana --name pmm-data-temporary percona/pmm-server:1 /bin/true 76249e1830c2a9c320466e41a454e9e80bf513e9b046e795ec41a33d75df5830 |
|
1 2 3 4 5 6 |
shell> docker cp pmm-data-temporary:/opt/prometheus/data /pmm/opt/prometheus/data shell> docker cp pmm-data-temporary:/opt/consul-data /pmm/opt/consul-data shell> docker cp pmm-data-temporary:/var/lib/mysql /pmm/var/lib/mysql shell> docker cp pmm-data-temporary:/var/lib/grafana /pmm/var/lib/grafana shell> docker rm -v pmm-data-temporary |
|
1 2 3 4 5 6 7 8 |
shell> docker create -v /pmm/opt/prometheus/data:/opt/prometheus/data -v /pmm/opt/consul-data:/opt/consul-data -v /pmm/var/lib/mysql:/var/lib/mysql -v /pmm/var/lib/grafana:/var/lib/grafana --name pmm-data percona/pmm-server:1 /bin/true d4c10ae9fb2e38758df999268573f4a8cddb5b47389b349f55733d2e54815bf0 |
|
1 2 3 |
shell> docker run --rm --volumes-from pmm-data -it percona/pmm-server:1 chown -R pmm:pmm /opt/prometheus/data /opt/consul-data shell> docker run --rm --volumes-from pmm-data -it percona/pmm-server:1 chown -R grafana:grafana /var/lib/grafana shell> docker run --rm --volumes-from pmm-data -it percona/pmm-server:1 chown -R mysql:mysql /var/lib/mysql |
After this, we can start the pmm-server container (see below).
Running pmm-server container
After following either of the steps mentioned above, we can run the pmm-server container with the exact same commands as shown in the online documentation:
|
1 2 3 4 5 6 7 |
shell> docker run -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1 0caa14f6fa22c419876de0dfb635535dbba41a2bd82b51b3d8a5be0b763fa6d2 |
And that’s it! Now you should have custom mount points on PMM docker deployment.
Step #3 is not correct with recent docker versions
The docker paths need to have “/.” at the end as per https://docs.docker.com/engine/reference/commandline/cp/
If this is not done, the cp command will nest the directories in destination path and mysql will fail to start (among to start with).
Change following
shell> docker cp pmm-data-temporary:/opt/prometheus/data /pmm/opt/prometheus/data
shell> docker cp pmm-data-temporary:/opt/consul-data /pmm/opt/consul-data
shell> docker cp pmm-data-temporary:/var/lib/mysql /pmm/var/lib/mysql
shell> docker cp pmm-data-temporary:/var/lib/grafana /pmm/var/lib/grafana
shell> docker rm -v pmm-data-temporary
to
shell> docker cp pmm-data-temporary:/opt/prometheus/data/. /pmm/opt/prometheus/data
shell> docker cp pmm-data-temporary:/opt/consul-data/. /pmm/opt/consul-data
shell> docker cp pmm-data-temporary:/var/lib/mysql/. /pmm/var/lib/mysql
shell> docker cp pmm-data-temporary:/var/lib/grafana/. /pmm/var/lib/grafana
shell> docker rm -v pmm-data-temporary
Hi Janis,
Thanks for your comment. I have double-checked this, and it should work as mentioned in the original steps. The thing is that we never created the innermost directories with mkdir command, so we would be in the following situation (from the same documentation link you sent):
-> SRC_PATH specifies a directory
—-> DEST_PATH does not exist
——–> DEST_PATH is created as a directory and the contents of the source directory are copied into this directory
I have tried this again, just in case, and it is working as expected. Maybe you modified the mkdir commands? Please send the exact commands you sent, so I’m able to reproduce, if not.
We get permission denied error when we run step 5
chown: cannot read directory ‘/opt/prometheus/data’: Permission denied
chown: cannot read directory ‘/opt/consul-data’: Permission denied
chown: cannot read directory ‘/var/lib/grafana’: Permission denied
chown: cannot read directory ‘/var/lib/mysql’: Permission denied
Hi Tanuj,
Sorry for the late reply. Were you able to solve this? Please go to our online forums for these kinds of questions: https://www.percona.com/forums/
Excelent post!
I did this a year ago, and still working
This procedure leads to Prometheus template error on CentOs 7
Hi José. Thanks for your comment.
Can you be more specific as to what is failing, so we can see if there are any issues with the steps? So far, we haven’t seen any errors due to these steps, specifically, although there were errors when some commands were not run as expected. Feel free to use the forums to post full steps used, and their outputs, so we can review them.
Also a workaround. To change the docker default configuration and make it use the Disk/Partition that we want to.
For the above example, If we want to use the /pmm/ .
1. Stop docker service:
systemctl stop docker.service
2. Override defaults(which is /var/lib/docker/volumes/) to use /pmm.
Add in /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=-/usr/bin/dockerd -H fd:// -g /pmm/docker –containerd=/run/containerd/containerd.sock
3. Start docker service:
systemctl status docker.service
systemctl status docker.service
Now follow the same steps to Install and configure: https://www.percona.com/doc/percona-monitoring-and-management/2.x/install/docker-setting-up.html
And then, you will have docker using the /pmm mount.