November 22, 2014

Benchmarks challenges of XtraDB Cluster

We are running internally a lot of benchmarks on our recently announced Percona XtraDB Cluster, and I am going to publish these results soon.
But before that I wanted to mention that proper benchmark of distributed system comes with a lot of challenges.
I am saying that not to complain, but to make sure, if you are going to benchmark XtraDB Cluster yourself, there is a lot of things to take into account.

And it seems that one component, which was not much important before, now appears as critical peace, which easily can became bottleneck in the benchmarks – this is network.

In case of simple client-server setup, the network is not fully utilized.

But as we start testing a cluster setup, the 1Gb network between client and switch is getting fully utilized by sysbench communication with 3 nodes.

In this setup it does not make sense to increase number of nodes, as we will not be able to load them properly.

The solution would be to increase network capacity or add additional client boxes.

Now take into account that there is an internal network communication between nodes also, and that makes a network tuning as the critical part of a cluster setup. This is not something we paid much attention before.

The main conclusion of this post is that if you are going to benchmark a Percona XtraDB Cluster or just use it under intensive communication workload, pay an attention to network component. It is very easy that a client or a client network becomes bottleneck.


About Vadim Tkachenko

Vadim leads Percona's development group, which produces Percona Clould Tools, the Percona Server, Percona XraDB Cluster and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.

Comments

  1. Kevin says:

    Maybe use a dedicated haproxy server with incoming sysbench traffic coming in on eth0 and load balance to the Galera cluster on eth1.

    Load balancer can do weighted balancing to simulate more real world transaction distribution. You can listen on multiple ports/ips to vary distribution of writes/reads, slow queries/fast queries, etc.

  2. Vadim,

    Yeah. You can do standard benchmarks basically having single server. Cluster benchmarks require proper lab with network setup correctly etc which also adds a lot more variance to results. Using multiple clients for benchmark is the most realistic as this is what we frequently have – tens of web servers can be talking to single MySQL server or cluster.

  3. marrtins says:

    How difficult is to bring InfiniBand into play? Not IP-over-InfiniBand, but native? I bet this would be great for cluster nodes located into same rack or same datacenter.

  4. I would be interested to see if IPv4 or IPv6 makes a difference in the benchmarks. And of course MTU size and jumbo frames could be a factor. What about using network bonding or nic teaming (new in linux 3.3?)? This could provide more bandwidth if there are enough network ports available.

  5. masood alam says:

    HI,

    Today I installed percona xtradb cluster on three of my laptops for testing, laptops connect with wireless router. I immediately realised that there are latency issues, then I moved to 100 mbps Ethernet connection. Quality of cables and the router is not giving more then 9-10 Mbps max.

    I was able to connect the nodes together and then played with them they connect fine, but now comes the main part. I have a database script that populates my database of 1GB, once it starts filling the tables with data it crashes, I mean the cluster crashes and then I have to do manual recovery of the cluster.
    Now I am wondering if it is because of my network is not good enough for such testing or is it something to do with cluster limitations that it cannot handle so many writes.

    Would appreciate your answers…
    regards,
    Alam

Speak Your Mind

*