PMM Server + podman: Running a Container Without root Privileges

October 22, 2019

Author

Ceri Williams

Monitoring

Percona Software

Share this Post:

PMM server and podman In this article, we will take a look at how to run Percona Monitoring and Management (PMM) Server in a container without root privileges.

Some of the concerns companies have about using Docker relate to the security risks that exist due to the requirement for root privileges in order to run the service and therefore the containers. If processes inside the container are running as root then they are also running as such outside of the container, which means that you should take measures to mitigate privilege escalation issues that could occur from container breakout. Recently, Docker added experimental support for running in rootless mode which requires a number of extra changes.

From a personal perspective, it seems that Docker tries to do it “all in one”, which has its issues. While I was looking to overcome some gripes with Docker, reduce requirements, and also run containers as a chosen user regardless of the process users inside the container, podman and buildah caught my attention, especially as podman also supports Kubernetes. To quote the docs:

Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode.

Configuring user namespaces

In order to use containers without the need for root privileges, some initial configuration is likely to be needed to set the mapping of namespaces, as well as a suitable limit. Using the shadow-utils package on CentOS, or uidmap package on Debian, it is possible to assign subordinate user (man subuid) and group IDs (man subgid) to allow the mapping of IDs inside a container to a dedicated set of IDs outside the container. This allows us to resolve the issue where a process is running as root both inside and outside of a container.

The following is an Ansible task snippet that, on a dedicated CentOS machine, makes the necessary changes, although you may need/want to adjust this if already using namespaces:

  - name: install packages<br>    yum:<br>      name:<br>      - runc<br>      - podman<br>      - podman-docker<br>      - buildah<br>      - shadow-utils<br>      - slirp4netns<br>      update_cache: yes<br>      state: latest<br>    become: yes<br>    tags:<br>    - packages<br><br>- name: set user namespace<br>   copy:<br>     content: '{{ ansible_user }}:100000:65536'<br>     dest: '/etc/{{ item }}'<br>   become: yes<br>   loop:<br>     - subuid<br>     - subgid<br>   register: set_user_namespace_map<br>   tags:<br>   - namespace<br><br> - name: set sysctl user.max_user_namespaces<br>   sysctl:<br>     name: user.max_user_namespaces<br>     value: 100000<br>     state: present<br>     sysctl_file: /etc/sysctl.d/01-namespaces.conf<br>     sysctl_set: yes<br>     reload: yes<br>   become: yes<br>   register: set_user_namespace_sysctl<br>   tags:<br>   - namespace<br><br> - name: reboot to apply<br>   reboot:<br>     post_reboot_delay: 60<br>   become: yes<br>   when: set_user_namespace_map.changed or set_user_namespace_sysctl.changed<br>   tags:<br>   - namespace

- name: install packages yum: name: - runc - podman - podman-docker - buildah - shadow-utils - slirp4netns update_cache: yes state: latest become: yes tags: - packages - name: set user namespace copy: content: '{{ ansible_user }}:100000:65536' dest: '/etc/{{ item }}' become: yes loop: - subuid - subgid register: set_user_namespace_map tags: - namespace - name: set sysctl user.max_user_namespaces sysctl: name: user.max_user_namespaces value: 100000 state: present sysctl_file: /etc/sysctl.d/01-namespaces.conf sysctl_set: yes reload: yes become: yes register: set_user_namespace_sysctl tags: - namespace - name: reboot to apply reboot: post_reboot_delay: 60 become: yes when: set_user_namespace_map.changed or set_user_namespace_sysctl.changed tags: - namespace

This set of tasks will:

Install some packages that are either required or useful when using podman.

Configure the 2 files used to map IDs ( /etc/subuid and /etc/subgid ); we allow the user to have 65536 user IDs mapped, starting the mapping from 100000 and likewise for group IDs.

Set user.max_user_namespaces to ensure that you can allocate sufficient IDs, making it persistent after a reboot. If you have needed to configure the namespaces then it is likely that you need to reboot to make sure that the changes are active.

Look Ma, PMM — and no root!

OK, so if all is well, then the system is configured to support user namespaces and we can run a container without needing to be root (or a member of a special group). The commands for podman are identical, or very close to those that you would use for Docker, and you can even set alias docker=podman if you wish! Here is how you could test PMM Server (v2) out, mapping port 8443 to the NGINX port inside the container:

$ whoami<br>percona<br><br>$ podman run -d --name pmm2-test -p 8443:443 docker.io/percona/pmm-server:2

1	$ whoami<br>percona<br><br>$ podman run -d --name pmm2-test -p 8443:443 docker.io/percona/pmm-server:2

In the previous command, the path to the registry is explicitly stated as being a Docker one, but if you were to simply specify percona/pmm-server:2 then by default a number of registries are checked and the first match will win. The list and order of the registries differ per distro, e.g.

Ubuntu: registries = ['quay.io', 'docker.io']

RedHat: registries = ['registry.redhat.io', 'quay.io', 'docker.io']

Checking the processes

The container for PMM server is now running as the user that executed the podman command, so we can take a look and see what the processes look like. First, let’s take a peek inside the container:

$ podman exec pmm-server bash -c "ps -e -o pid,ppid,pgrp,user,comm --forest"<br>  PID  PPID  PGRP USER     COMMAND<br>  862     0   862 root     ps<br>    1     0     1 root     supervisord<br>    9     1     9 postgres postgres<br>   88     9    88 postgres  _ postgres<br>   98     9    98 postgres  _ postgres<br>  100     9   100 postgres  _ postgres<br>  101     9   101 postgres  _ postgres<br>  102     9   102 postgres  _ postgres<br>  103     9   103 postgres  _ postgres<br>  104     9   104 postgres  _ postgres<br>  154     9   154 postgres  _ postgres<br>  173     9   173 postgres  _ postgres<br>  183     9   183 postgres  _ postgres<br>   10     1    10 root     clickhouse-serv<br>   11     1    11 grafana  grafana-server<br>   12     1    12 root     nginx<br>   23    12    12 nginx     _ nginx<br>   13     1    13 root     crond<br>   14     1    14 pmm      prometheus<br>   17     1    17 root     pmm-managed<br>   51    17    17 root      _ supervisorctl<br>   18     1    18 root     pmm-agent<br>  172    18    18 root      _ node_exporter<br>  174    18    18 root      _ postgres_export<br>  153     1   153 pmm      percona-qan-api

$ podman exec pmm-server bash -c "ps -e -o pid,ppid,pgrp,user,comm --forest" PID PPID PGRP USER COMMAND 862 0 862 root ps 1 0 1 root supervisord 9 1 9 postgres postgres 88 9 88 postgres _ postgres 98 9 98 postgres _ postgres 100 9 100 postgres _ postgres 101 9 101 postgres _ postgres 102 9 102 postgres _ postgres 103 9 103 postgres _ postgres 104 9 104 postgres _ postgres 154 9 154 postgres _ postgres 173 9 173 postgres _ postgres 183 9 183 postgres _ postgres 10 1 10 root clickhouse-serv 11 1 11 grafana grafana-server 12 1 12 root nginx 23 12 12 nginx _ nginx 13 1 13 root crond 14 1 14 pmm prometheus 17 1 17 root pmm-managed 51 17 17 root _ supervisorctl 18 1 18 root pmm-agent 172 18 18 root _ node_exporter 174 18 18 root _ postgres_export 153 1 153 pmm percona-qan-api

We can see all of the processes that run as part of PMM, plus the ps command that was executed to look at the processlist.

Now, let’s take a look outside the container to see what these processes show up as:

$ pgrep -f 'postgres|supervisord|clickhouse-serv|pmm-managed' | xargs ps -o pid,ppid,pgrp,user,comm --forest<br>  PID  PPID  PGRP USER     COMMAND<br>32020 32003 32003 percona  node_exporter<br>23532 23358 23358 percona  postgres_export<br>23530 23358 23358 percona  node_exporter<br>23332 23321 23332 percona  supervisord<br>23349 23332 23349 100025    _ postgres<br>23440 23349 23440 100025    |   _ postgres<br>23456 23349 23456 100025    |   _ postgres<br>23458 23349 23458 100025    |   _ postgres<br>23459 23349 23459 100025    |   _ postgres<br>23460 23349 23460 100025    |   _ postgres<br>23461 23349 23461 100025    |   _ postgres<br>23462 23349 23462 100025    |   _ postgres<br>23512 23349 23512 100025    |   _ postgres<br>23531 23349 23531 100025    |   _ postgres<br>23541 23349 23541 100025    |   _ postgres<br>23350 23332 23350 percona   _ clickhouse-serv<br>23357 23332 23357 percona   _ pmm-managed

$ pgrep -f 'postgres|supervisord|clickhouse-serv|pmm-managed' | xargs ps -o pid,ppid,pgrp,user,comm --forest PID PPID PGRP USER COMMAND 32020 32003 32003 percona node_exporter 23532 23358 23358 percona postgres_export 23530 23358 23358 percona node_exporter 23332 23321 23332 percona supervisord 23349 23332 23349 100025 _ postgres 23440 23349 23440 100025 | _ postgres 23456 23349 23456 100025 | _ postgres 23458 23349 23458 100025 | _ postgres 23459 23349 23459 100025 | _ postgres 23460 23349 23460 100025 | _ postgres 23461 23349 23461 100025 | _ postgres 23462 23349 23462 100025 | _ postgres 23512 23349 23512 100025 | _ postgres 23531 23349 23531 100025 | _ postgres 23541 23349 23541 100025 | _ postgres 23350 23332 23350 percona _ clickhouse-serv 23357 23332 23357 percona _ pmm-managed

Great! As we can see from the processlist, none of the processes that we checked are running as root, the ones that are running as root inside the container are running as percona, and the remainder are using unknown user IDs generated by the subordinate mapping.

Using persistent volumes for your data

The documented way to use PMM with persistent volumes does not work in the same manner with podman. However, it is easily fixed as follows (plus we get to try out buildah):

$ cat <<EOS > PMM2.Dockerfile<br>FROM docker.io/percona/pmm-server:2<br><br>VOLUME ['/srv']<br>EOS<br><br>$ buildah bud -f PMM2.Dockerfile --tag localhost/pmm-server:custom .<br>$ podman create --name pmm-data -v /srv localhost/pmm-server:custom <br>$ podman run -d --name pmm-server -v pmm-data:/srv -p 8443:443 localhost/pmm-server:custom

$ cat <<EOS > PMM2.Dockerfile FROM docker.io/percona/pmm-server:2 VOLUME ['/srv'] EOS $ buildah bud -f PMM2.Dockerfile --tag localhost/pmm-server:custom . $ podman create --name pmm-data -v /srv localhost/pmm-server:custom $ podman run -d --name pmm-server -v pmm-data:/srv -p 8443:443 localhost/pmm-server:custom

Summary

It is easy to get PMM server running in a container with podman and buildah while also increasing the security from the outside by restricting the user privileges for the processes.

And while you may not be able to replace Docker usage in certain circumstances, taking a look at podman and buildah is definitely recommended… and you can still use the Docker registry if that is the only location for the image that you want to run.

If you haven’t tried PMM2 yet then you can easily test it out or take a look at the online demo.