Comparing Percona XtraDB Cluster with Semi-Sync replication Cross-WAN

I have a customer who is considering Percona XtraDB Cluster (PXC) in a two colo WAN environment.  They wanted me to do a test comparing PXC against semi-synchronous replication to see how they stack up against each other.

Test Environment

The test environment included AWS EC2 nodes in US-East and US-West (Oregon).  The ping RTT latency between these nodes was right around 100ms.

All environments used Percona Server or Percona XtraDB Cluster server 5.5.24.  Innodb durability was disabled for all tests (innodb_flush_log_at_trx_commit=2).  All other settings were the same kept the same unless otherwise noted.

I tested against the following setups:

Control

The control environment was simply a Master with log-bin enabled and one slave connected (not semi-sync).

Semi-sync

Same as the control but with semi-sync enabled on the slave (slave was in the other colo).

XtraDB Cluster 1-colo

3  nodes in one datacenter.  Writes were only done on a single node.  The cluster contained the tuning:

based on Codership’s recommended tuning for single-node writing.  I was running wsrep_slave_threads=16 and all other required wsrep settings.

Also note that I disabled log-bin and innodb_support_xa on these nodes.

XtraDB Cluster 2-colo

Just like PXC 1-colo, except with 3 more nodes in the second colo.  I ignored quorum arbitration for the purposes of this test.  Writes were only done on one node in one colo.

Tests

I did two different tests to illustrate the differences between these technologies.

Single-write latency test

For this test, I simply created a table, inserted about 10 rows, and calculated an average time for each INSERT to autocommit.

 

From this I made the following observations:

• The overhead of Galera replication is approximately 10x longer than local binary log writing.  It seems conceivable to me that this may be improvable by some Galera tuning. UPDATE: latency between EC2 nodes, even within the same availability zone seems to average > 1ms, so I don’t think this result is necessarily all that significant.
• Semi-sync, as expected, takes approximately 1 RTT for the slave to verify to the master that it got the transaction.
• PXC 2-colo (with 6 nodes instead of 2) does very well considering how much work it is doing, still approximately 1 RTT for commit.
• Semi-sync does not actually enforce the transaction is committed on the slave, just that the slave received it.
• PXC 2-colo committed the data on all 6 nodes across two datacenters in approximately the same amount of time as one Semi-sync transaction (a whole lot more work).

Sysbench 32-client test

For this test I used the latest sysbench with 20 tables (~5G of data) and 32 test clients.  The results are simply the average transactions per second I got from a 60 second run.  I do not believe these tests were disk-bound in any way, I utilized Percona Server’s Buffer pool dump/restore feature to preserve the caches across server restarts, and a gaussian distribution on sysbench (i.e., non-random).

 

This to me was really interesting:

• PXC 1-colo beat the control.  I attribute this to log-bin and innodb_support_xa being disabled in the PXC environments
• Semi-sync performs dismally under concurrency.  Think about it, our single inserts took ~100ms, so we are effectively serializing our 32 clients with semi-sync replication.  Apparently each client must wait for it’s turn writing to the binlog and waiting for confirmation from the slave, there can be no parallelization here.
• I could not find any way to parallelize semi-synchronous replication (at least in 5.5), I could be missing something.
• PXC’s true parallel replication shines here, applying up to 16 writesets at a time based on the number of threads I configure.   Even though the individual commits were approximately the same speed as semi-sync, parallelism allows PXC to scale much further across high latency environments.
• If each transaction took 100ms to commit, then I’d expect 16 threads to handle about 160TPS at best.  There must be some other optimization here that I can’t explain.
• I did not try optimizing the PXC 2-colo result, but I intend to do so soon.  I would not be surprised to get better results with the right tuning.  Even just setting the wsrep_slave_threads=32 to have 1 applier thread per node per test client.  More on that in a future post.

Conclusion

Percona Xtradb Cluster and Galera on which it is based offers a much more realistic multi-datacenter system for high availability and disaster recovery than semi-synchronous replication in stock MySQL.