TokuDB v6.0: Getting Rid of Slave Lag

Master/slave replication is an important tool that gets used in many ways: distributing read loads among many slaves for performance, using a slave for backups so the master can handle live load, geographically distributed disaster recovery, etc. The Achilles’ Heal of slave performance is that slave workloads are single-threaded. The master can have many clients inserting, updating, querying, whereas the slave has only one insertion client: the master. InnoDB single-client performance is much slower than its multi-client performance, which means that the bottleneck in a master/slave system is often the rate at which a slave can keep up.

If the master has an average transactions per second (tps) that is higher than what the slave can handle, the slave will fall further and further behind. If the slaves are being used to distribute read workload, for example, the results they produce will fall further out of date. If a slave is used to generate a backup (e.g. the slave is taken offline to produce a backup snapshot), then the slave has a harder time catching up with the master once it comes back online, and if it never catches up, the value of a backup is reduced.

So slave lag caused by single-client performance is a big problem. The good news is that TokuDB has enough data ingestion horsepower that it can keep up with some big single-threaded workloads. We’ve been able to show this with our newly released TokuDB v6.0.

Here’s what we did to measure the impact of slave lag. We made a version of TPCC that generates transactions at some user-definable rate. We fed the transactions to a master/slave combo for 60s. At the rates we measured, the master was able to finish all transactions during the 60s window. Then we waited to see how long it would take the slave to finish its work. At 1000tps, both TokuDB and InnoDB slaves were able to complete the work in 60s, which means there was no slave lag. By the time we got to 3000tps, InnoDB was taking over 140s to finish the work, meaning that it was falling more than 80s behind every 60s. TokuDB was still keeping up.

Slave Complete Time

You can find all the details of the experiement here.

In TokuDB v6.0 we are introducing XA (two-phase transactions) which is a common way that binlog replication has been implemented in MySQL. Combined with great slave performance, this make TokuDB a great choice for replication.

To learn more about TokuDB:

  • Download a free trial of TokuDB.
  • Read the press release here.
  • Hear me talk about TokuDB v6.0 on the MySQL Database Community Podcast in Episode 86.
  • Come to our booth #410 at Percona Live.
  • Catch Tokutek Software Engineer Leif Walsh’s presentation at Percona Live on April 11th at 4:30 pm
  • Catch Tokutek VP of Marketing’s Lawrence Schwartz’s Lightning Talk at Percona Live on April 11th at 6:30 pm

Share this post

Leave a Reply