EmergencyEMERGENCY? Get 24/7 Help Now!

Measuring Percona Server Docker CPU/network overhead

 | February 5, 2016 |  Posted In: Docker, MySQL, Percona Server for MySQL

PREVIOUS POST
NEXT POST

Precona Server DockerNow that we have our Percona Server Docker images, I wanted to measure the performance overhead when we run the database in the container. Since Docker promises to use a lightweight container, in theory there should be very light overhead. We need to verify that claim, however. In this post I will show the numbers for CPU and network intensive workloads, and later I will take a look at IO.

For the CPU-bound load, I will use a sysbench OLTP read-only workload with data fitting into memory (so there is no IO performed, and the execution path only goes through the network and CPU).

My server is 24 cores (including hyper-threads), with Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz CPUs, RAM: 256GB, OS: Ubuntu 14.04. The Docker version is the latest on the day of publishing, which is 1.9.1.

First, I measured the throughput on a bare server, without containers – this will be the baseline. For reference, the command I used is the following:

On the bare metal system, the throughput is 7100 transactions per second (tps).

In the next experiment, I started Percona Server in a Docker container and connect to it from the host:

In this case, the container exposed port 3306 to the host, and we used that as an access point in sysbench.

The throughput in this scenario is 2200 tps!!! That is a significant overhead. I suspect it comes from the Docker gateway, which is added to the execution path when we connect through port forwarding.

So to avoid the Docker gateway, in the next run I used the host network by running the container with --net=host:

In this case the container ran directly in the host network stack, so this should exclude any Docker network overhead. In this case, the throughput is basically back to 7100 tps.

From these tests, I can make an important conclusion. There is NO measurable CPU overhead when running Percona Server in a Docker container. But the network path raises some questions.

So in the next experiment I ran both sysbench and MySQL in two different containers, connected over the Docker network bridge.

I created a sysbench container, which you can get from:

To run sysbench:

Just for the reference, I created a Docker network:

and connected both containers to the same network:

In this configuration, the throughput I’ve observed is 6300 tps.

So there is still some network overhead, but not as significant as with the port gateway case.

For the last example, I again excluded the network path and ran the sysbench container inside the MySQL container network stack using the following command:

The throughput in this configuration is back to 7100 tps. 

And the conclusion, again, is that there is no CPU overhead even if we run both client and server inside containers, but there is some network overhead – even when running on the same host. It will be interesting to measure the network overhead when the containers are on different physical hosts.

The following chart summarizes the results:

dockeroverhead

Next time I will try to measure IO overhead in Docker containers.

PREVIOUS POST
NEXT POST
Vadim Tkachenko

Vadim Tkachenko co-founded Percona in 2006 and serves as its Chief Technology Officer. Vadim leads Percona Labs, which focuses on technology research and performance evaluations of Percona’s and third-party products. Percona Labs designs no-gimmick tests of hardware, filesystems, storage engines, and databases that surpass the standard performance and functionality scenario benchmarks.

Vadim’s expertise in LAMP performance and multi-threaded programming help optimize MySQL and InnoDB internals to take full advantage of modern hardware. Oracle Corporation and its predecessors have incorporated Vadim’s source code patches into the mainstream MySQL and InnoDB products.

He also co-authored the book High Performance MySQL: Optimization, Backups, and Replication 3rd Edition.

4 Comments

  • Hi Vadim,

    Thanks for sharing this info!
    Have you tested or considered MySQL in LXC? It would be interesting to see if there is any impact on MySQL performance.
    LXD may also be helpful to manage the containers.

    Thanks!
    -ivan

  • Hi Vadim,

    I know I am very late in getting to this article, but thanks for a great write-up ! I have been trying to use the tpcc-mysql tool from percona-lab to identify the performance overhead while using percona DB container as against running it on baremetal. I found the tpmC metric was considerably lower when I ran the DB on container. In fact 6X lower.

    While I used the same underlying hardware (which is an NVMe device formatted as ext4 fs) as the storage for DB (mounted a directory from this fs as vol for mysql) , the killer seemed to be network. I ran some fio tests on the disk from host & later on the volume mount in the container just to be sure that storage wasn’t causing any issue and found IO perf to be the same across baremetal and docker, something which is validated in your tests as well.

    Bridge network was worst, host network was better by ~ 10K tpmC (and average response time lesser by ~10ms) but I could reach some sort of parity with the baremetal numbers only when I ran the tpcc-benchmark tool from “inside” the percona container itself instead of running it from the host (even this was ~1.4 times lower). Each time the test was run with -h flag to the tpcc scripts set to 127.0.0.1.

    Does this strike something ? Is the TPC-C test more network-intensive ? Also, we are running the mysql with some system variable values overriding the my.cnf like increased io_capacity & buffer_pool_size. It would be great if you could share some insights around this..

    Thanks & Regards,
    Karthik

    • Karthik,

      I believe your results are identical to mine. Running database in container significantly affects the performance.
      You should try some if following: try –net=host or try to run tpcc-mysql also in a container within the same docker network as database.

Leave a Reply