The Future of TokuDB at Percona


TokuDBIn this blog post, I’ll discuss the future of TokuDB at Percona. Spoiler: solid.

As soon as we announced the fact that MyRocks was coming to Percona Server for MySQL at Percona Live Europe, rumors appeared that this means we’re going to phase out TokuDB at the same time.  

I can understand why those rumors would start: just a few months ago we deprecated Fractal Trees technology (called PerconaFT) in favor of MongoRocks and RocksDB for Percona Server for MongoDB.

As much as this might look like the same situation as with PerconaFT, TokuDB is very different. PerconaFT was actually a new port of Fractal Trees technology for MongoDB. It used the MongoDB storage engine API and focused on replacing TokuMX (a full fork of an earlier version of MongoDB). PerconaFT was new, and MongoRocks was already well-established and battle-tested by Parse. The MongoDB internal design was also much more friendly towards the RocksDB storage engine than Fractal Trees technology (as David Murphy explains here).    

Our research revealed that PerconaFT had very few current users, and MongoRocks was a superior choice for new users. This is why we depreciated it.

For MySQL, TokuDB is a rather mature storage engine. It has been in production for more than five years and is trusted by many businesses and large enterprises (in industries ranging from banking and finance to gaming) for use with critical applications.

MyRocks however, with as much promise as it shows, has not been extensively battle-tested outside of Facebook. And as it is hard to beat “Facebook scale,” it hasn’t exposed its performance under a large variety of workloads. There are also some known limitations in MyRocks that need work.

Ultimately though, I believe MyRocks has a great future, and we at Percona are going to take part in making this future reality. But this does not mean we plan to immediately abandon TokuDB, or place it on “life support”.    

You can see new development going on for TokuDB: we have dramatically improved block allocation performance and added a new, more convenient TokuDB file layout option. We are also working on adding support for new compression algorithms in TokuDB, support for Performance Schema and refactoring the checkpoint process to ensure uniform performance on all workloads. It is true TokuDB is not changing as rapidly as RocksDB, but that is to be expected – as the more mature storage engine, it should embrace stability first and foremost.

Our overall principle at Percona is to keep our ego in check. We want to provide the best open source ecosystem solutions available to our customers, whether they were developed in-house or not. This means that within 3-5 years MyRocks could possibly be superior to TokuDB in functionality and performance for all imaginable workloads. Or an even better open source storage engine technology might get developed and come to market. If or when this happens, we will consider depreciating TokuDB in favor of such technology – while providing the customers with enough notice, tools and support to ensure they can successfully migrate.

What this really means is that TokuDB is still being actively supported and developed for at least Percona Server 5.7 and Percona Server 8.0 (and longer). We will continue this as long as a reasonable use case remains for this technology.

Please feel free to contact us with any questions, or add them in the comments below.


Share this post

Comments (6)

  • Mark Callaghan Reply

    The MyRocks team appreciates the challenge. We want it to be used when it is the right choice, not because it is used by a famous web-scale company. We are thrilled that it will be in Percona Server. That will help us make it better faster.

    November 1, 2016 at 1:51 pm
    • Peter Zaitsev Reply

      Thanks Mark. And I believe MyRocks will be right choice in many cases. Yet you should not underestimate the value of Facebook brand in terms of technology usage 🙂

      November 1, 2016 at 6:57 pm
  • Tim Callaghan Reply

    Do you have a TokuDB roadmap you can share? While there have been some features added in the past 1.5 years they are mostly minor.

    November 2, 2016 at 7:51 am
  • Krishna Reply

    Thanks Perter and Mark for the best efforts to integrate MyRocks to Percona.. looking fwd to it..

    November 4, 2016 at 1:18 pm
  • Rick Pizzi Reply

    Peter, glad to read your post above. We are using TokuDB since almost a couple years now and I would be very sad to see it go out of support. At the same time, I’m really excited by MyRocks and happy for your PLAMS announcement that you will include it in PS soon. I believe that although we already put MyRocks to some production use right now, it still needs quite some time to become mature enough to replace TokuDB in highly concurrent workloads . BTW I am really eager to try and compare TokuDB vs MyRocks in our current use case 😉 Keep up the good work guys!!!

    November 4, 2016 at 3:21 pm
  • Philip Orleans Reply

    There is question for which I cannot find a proper answer. TokuDB can take advantage of large_pages=1 and memlock=1 or not? I disabled innodb, and I am only using TokuDB, but I ignore if TokuDB can even touch Large Pages. I am not talking about TransparenHugePages, which is disabled at the kernel level.

    December 4, 2016 at 4:58 pm

Leave a Reply