Measuring MySQL Performance in Kubernetes

PREVIOUS POST
NEXT POST

Measuring MySQL Performance in KubernetesIn my previous post Running MySQL/Percona Server in Kubernetes with a Custom Config I’ve looked at how to set up MySQL in Kubernetes to utilize system resources fully. Today I want to measure if there is any performance overhead of running MySQL in Kubernetes, and show what challenges I faced trying to measure it.

I will use a very simple CPU bound benchmark to measure MySQL performance in OLTP read-only workload:

The hardware is as follows:

Supermicro server

  • Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
  • 2 sockets / 28 cores / 56 threads
  • Memory: 256GB of RAM

The most interesting number there is 28 cores / 56 threads.  Please keep this in mind; we will need this later.

So let’s see the MySQL performance in the bare metal setup:

So we can get about 22000 qps on this server.

Now, let’s see what we can get if the same server runs a Kubernetes node and we deploy the Percona server image on this node. I will use a modified image of Percona Server 8, which already includes sysbench inside.

You can find my image here: https://hub.docker.com/r/vadimtk/ps-8-vadim

And I use the following deployment yaml :

The most important part here is that we deploy our image on smblade01 node (the same one I ran the bare metal benchmark).

Let’s see what kind of performance we get using this setup. The number I’ve got:

You can see the numbers vary a lot, from 10550 tps to 20196 tps, with the most time being in the 10000tps range.
That’s quite disappointing. Basically, we lost half of the throughput by moving to the Kubernetes node.

But don’t panic, we can improve this. But first, we need to understand why this happens.

The answer lies in how Kubernetes applies Quality of Service for Pods. By default (if CPU or Memory limits are not defined) the QoS is BestEffort, which leads to the results we see above. To allocate all CPU resources, we need to make sure QoS Guaranteed. For this, we add the following to the image definition:

These are somewhat funny lines to define CPU limits. As you remember we have 56 threads, so initially I tried to set limits: cpu: "56" , but it did not work as Kubernetes was not able to start the pod with the error Insufficient CPU. I guess Kubernetes allocates a few CPU percentages for the internal needs.

So the line cpu: "55500m"  works, which means we allocate 55.5 CPU for Percona Server.

Let’s see what results we can have with Guaranteed QoS:

This is much better (mostly ranging in 20000 tps), but we still do not get to 22000 tps.

I do not have the full explanation of why there is still a 10% performance loss, but it might be related to this issue. And I see there is a work in progress to improve Guaranteed QoS performance but it was not merged into the mainstream releases yet. Hopefully, it will be in one of the next releases.

Conclusions:

  • Out of the box, you may see quite bad performance when deploying in Kubernetes POD
  • To improve your experience you need to make sure you use Guaranteed QoS. Unfortunately, Kubernetes does not make it easy. You need to manually set the number of CPU threads, which is not always obvious if you use dynamic cloud instances.
  • With Guaranteed QoS there is still a performance overhead of 10%, but I guess this is the cost we have to accept at the moment.
PREVIOUS POST
NEXT POST

Share this post

Comments (3)

  • Matt Lord Reply

    Thanks for doing this, Vadim! I’m curious if you have yet done any testing with the beta CPU Manager features available in Kubernetes 1.10+? I wonder if using the –cpu-manager-policy=static option for your kublet(s), while continuing to use the Guaranteed QoS class for your Pod(s) as you showed here, might help to close the noted performance gap completely?
    https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/
    https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/

    I’ll be playing around with similar things myself in the future and will share any interesting findings. Thanks again for sharing this work! Good stuff.

    May 9, 2019 at 8:02 pm
    • Vadim Tkachenko Reply

      Matt,

      I did not test –cpu-manager-policy=static , but this is something I plan to look into.
      This is one area I find problematic with Kubernetes – it is not quite obvious how to changes settings easily cluster wise. I could not figure out it quickly in my current testing.

      May 13, 2019 at 8:17 pm
  • zainal Reply

    Very good inside, but your example is for scale up. Kubernetes design is for scale out. If you add more minion you will have better performance without to increase cpu and memory

    May 12, 2019 at 9:11 pm

Leave a Reply