PMM Server + podman: Running a Container Without root Privileges

PMM server and podmanIn this article, we will take a look at how to run Percona Monitoring and Management (PMM) Server in a container without root privileges.

Some of the concerns companies have about using Docker relate to the security risks that exist due to the requirement for root privileges in order to run the service and therefore the containers. If processes inside the container are running as root then they are also running as such outside of the container, which means that you should take measures to mitigate privilege escalation issues that could occur from container breakout. Recently, Docker added experimental support for running in rootless mode which requires a number of extra changes.

From a personal perspective, it seems that Docker tries to do it “all in one”, which has its issues. While I was looking to overcome some gripes with Docker, reduce requirements, and also run containers as a chosen user regardless of the process users inside the container, podman and buildah caught my attention, especially as podman also supports Kubernetes. To quote the docs:

Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode.

Configuring user namespaces

In order to use containers without the need for root privileges, some initial configuration is likely to be needed to set the mapping of namespaces, as well as a suitable limit. Using the shadow-utils package on CentOS, or uidmap package on Debian, it is possible to assign subordinate user (man subuid) and group IDs (man subgid) to allow the mapping of IDs inside a container to a dedicated set of IDs outside the container. This allows us to resolve the issue where a process is running as root both inside and outside of a container. 

The following is an Ansible task snippet that, on a dedicated CentOS machine, makes the necessary changes, although you may need/want to adjust this if already using namespaces:

This set of tasks will:

  1. Install some packages that are either required or useful when using podman.
  2. Configure the 2 files used to map IDs ( /etc/subuid and /etc/subgid ); we allow the user to have 65536 user IDs mapped, starting the mapping from 100000 and likewise for group IDs.
  3. Set user.max_user_namespaces  to ensure that you can allocate sufficient IDs, making it persistent after a reboot. If you have needed to configure the namespaces then it is likely that you need to reboot to make sure that the changes are active.

Look Ma, PMM — and no root!

OK, so if all is well, then the system is configured to support user namespaces and we can run a container without needing to be root (or a member of a special group). The commands for podman are identical, or very close to those that you would use for Docker, and you can even set alias docker=podman  if you wish! Here is how you could test PMM Server (v2) out, mapping port 8443 to the NGINX port inside the container:

In the previous command, the path to the registry is explicitly stated as being a Docker one, but if you were to simply specify percona/pmm-server:2 then by default a number of registries are checked and the first match will win. The list and order of the registries differ per distro, e.g.

  • Ubuntu: registries = ['quay.io', 'docker.io']
  • RedHat: registries = ['registry.redhat.io', 'quay.io', 'docker.io']

Checking the processes

The container for PMM server is now running as the user that executed the podman command, so we can take a look and see what the processes look like. First, let’s take a peek inside the container:

We can see all of the processes that run as part of PMM, plus the ps command that was executed to look at the processlist.

Now, let’s take a look outside the container to see what these processes show up as:

Great! As we can see from the processlist, none of the processes that we checked are running as root, the ones that are running as root inside the container are running as percona, and the remainder are using unknown user IDs generated by the subordinate mapping.

Using persistent volumes for your data

The documented way to use PMM with persistent volumes does not work in the same manner with podman. However, it is easily fixed as follows (plus we get to try out buildah):

Summary

It is easy to get PMM server running in a container with podman and buildah while also increasing the security from the outside by restricting the user privileges for the processes.

And while you may not be able to replace Docker usage in certain circumstances, taking a look at podman and buildah is definitely recommended… and you can still use the Docker registry if that is the only location for the image that you want to run.

If you haven’t tried PMM2 yet then you can easily test it out or take a look at the online demo.

Share this post

Comments (8)

  • John Beranek Reply

    Hi, just attempting to build a fresh PMM 2 server on an Oracle Linux 8 VM with Podman/buildah following your instructions above and get:

    # buildah bud -f PMM2.Dockerfile –tag localhost/pmm-server:custom .
    STEP 1: FROM docker.io/percona/pmm-server:2
    STEP 2: VOLUMES [‘/srv’]
    ERRO[0000] Build error: Unknown instruction: “volumes”

    # rpm -q podman buildah
    podman-1.0.5-1.gitf604175.0.1.module+el8.0.0+5336+de35ffea.x86_64
    buildah-1.5-3.0.1.gite94b4f9.module+el8+5201+6423ecab.x86_64

    Any ideas?

    October 30, 2019 at 12:29 pm
    • John Beranek Reply

      Scratch that, if I run it as a non-root user it works !?

      October 30, 2019 at 12:31 pm
  • John Beranek Reply

    OK, simpler than I thought:

    VOLUME [‘srv’]

    October 30, 2019 at 12:37 pm
    • Ceri Williams Reply

      Thanks for spotting the typo, the post has been updated.

      October 31, 2019 at 7:11 am
  • John Beranek Reply

    Unfortunately it looks like non-root podman doesn’t currently work on OL8:

    $ podman info
    could not get runtime: kernel does not support overlay fs: ‘overlay’ is not supported over xfs at “/home/pmm/.local/share/containers/storage/overlay”: backing file system is unsupported for this graph driver

    October 30, 2019 at 12:42 pm
  • John Beranek Reply

    OK, I fixed the rootless podman, but then hit:

    $ podman create –name pmm-data -v pmm-data/srv localhost/pmm-server:custom
    incorrect volume format “pmm-data/srv”, should be host-dir:ctr-dir:[option]

    Tried doing this with a named volume, which worked, but then the PMM server fails to start:


    $ podman create --name pmm-data -v pmm-data:/srv localhost/pmm-server:custom
    c64e65f365d3ba8a08f3077b42666202e87f46b63af054664a87fd9cc4865151

    $ podman run -d --name pmm-server -v pmm-data:/srv localhost/pmm-server:custom
    cd5fd0a3864447c40b02c8b974759caac9568286ca3f7730f059d072c313dd12

    $ podman logs pmm-server
    Error: The directory named as part of the path /srv/logs/supervisord.log does not exist
    For help, use /usr/bin/supervisord -h

    October 30, 2019 at 12:53 pm
  • John Beranek Reply

    Also this, sigh – back to trying to install docker-ce on OL8 I guess…


    $ podman run -d --name pmm-server -p 8443:443 localhost/pmm-server:custom
    port bindings are not yet supported by rootless containers

    October 30, 2019 at 12:56 pm
    • Ceri Williams Reply

      Apologies for the delayed response John. I haven’t tried on an 8-based distro yet, but if you are able to use the host networking then it isn’t too hard to rebuild the image to change the NGINX ports so that you can avoid port binding

      November 13, 2019 at 11:50 am

Leave a Reply