Percona Monitoring Management ContainerMy boss asked me if there is a way to wait for the Percona Monitoring and Management (PMM) Server container to be fully ready so his automation script can start adding monitoring instances. I wanted to give him a general solution, not a PMM-specific one. Turned out, it is not so simple.

The root of the problem is that some big containers, including PMM Server, take some time to boot up. We can’t assume that it is ready once docker run -d exited, or once the network port is ready to accept connections. For example, PMM Server contains nginx, and it is ready to accept connections almost immediately, while other components are still starting. We want to know when it is “fully” ready.

Let’s start with the PMM-specific solution. In both PMM 1.x and 2.x we expose a separate endpoint for that purpose – /ping. You can use it with curl and a simple shell loop:

-f (--fail) flag causes curl to exit with code 22 if the HTTP response code is 400 or above. In the example above, you can see three failed attempts with different exit codes, and then the successful response (empty JSON object).

Is there a more generic way for other Docker containers? It turned out, there is: Docker 1.12, released more than 4 years ago, introduced HEALTHCHECK Dockerfile instruction for that purpose. It can be used like this:

Unlike the previous example, that command runs inside Docker container, not outside. The meaning of those parameters is documented in Docker’s reference. We add || exit 1 because CMD should return either 0 or 1, nothing else.

Surprisingly, there is no built-in docker subcommand to wait for a container to become healthy, so we have to use a shell loop again:

Unfortunately, not many images contain the HEALTHCHECK instruction. In fact, while writing that blog post, I noticed that PMM Server does not have it! So I added it, and it will be released as part of PMM 2.4. But we definitely don’t want to change all images we want to use that don’t contain this instruction. Luckily, HEALTHCHECK functionality is also available via docker run flags:

Then we can use the same “until docker inspect” loop to wait for a container to become healthy.

Hopefully, that information will be useful to you for running third-party containers without health checks. But please, add the HEALTHCHECK instruction to your own Dockerfiles, like I did.