EmergencyEMERGENCY? Get 24/7 Help Now!

First TokuMXse performance numbers

 | January 12, 2015 |  Posted In: Tokutek, TokuView

PREVIOUS POST
NEXT POST

UPDATE: Since the publication of this blog post, the first release candidate of TokuMXse was made available for testing. Learn more about v1.0.0.RC.0. In addition, MongoDB has decided to call the release formerly known as v2.8, v3.0.

We’ve been working on TokuMXse for quite some time now. TokuMXse is the MongoDB 2.8 storage engine version of TokuMX that we’ll be shipping when MongoDB 2.8 goes GA. We will have an RC version available soon for download and feedback. Before we dig into the numbers I’d like to share my thoughts about them.

First, the MongoDB 2.8 storage engine API is brand new and like all complex software will improve over time. Adding a brand new API that supports the existing MMAPV1 engine (non-document-level-locking) as well as the new WiredTiger engine is a non-trivial task, as is evidenced by the release date of MongoDB 2.8 pushing past their original 2014 delivery goal.

Second, in creating TokuMX we not only eliminated the overhead of a storage engine API, we optimized every aspect of MongoDB with respect to our Fractal Tree indexing technology. This is not only apparent in the benchmark numbers, it is fundamentally responsible for many features of TokuMX that cannot be enabled in TokuMXse. We’ll be highlighting these features in the near future.

Last, there is always more tuning and optimization to be done in performance oriented software. We are currently passing all tests so I thought it was appropriate to publish these results.

Having said that, lets look at the benchmarks.

Benchmark 1 : iiBench single writer
One thread inserting into a single collection with 3 secondary indexes (random values)

iibench-1
Not too bad, only a 9.6% performance drop. This can partly be explained by the fact that TokuMX creates a single PK index for this use-case (it’s a clustering index where the key is _id and the value is the document), whereas TokuMXse (and all storage engines) creates two “indexes” (a record store for the document, and an index for the _id with the value being the key in the record store).


Benchmark 2 : iiBench four writers
Four threads inserting into a single collection with 3 secondary indexes (random values)

iibench-4
Almost no performance loss at all, only a 2.9% performance drop. I’m not sure why the difference was significantly smaller than the single writer version of this test, I have to investigate further.


Benchmark 3 : > RAM Sysbench
Standard Sysbench workload (IO bound), 16 collections, 10 million documents per collection, 8G cache, and directIO. Each Sysbench transaction consists of several queries and aggregations, an indexes update, and unindexed update, a delete, and an insert.

sysbench-10mm-load
In the Sysbench load phase, 16 collections are loaded (insert only) with 10 million documents each, using 8 simultaneous threads each inserting into it’s own collection. TokuMXse was 46% slower than TokuMX in this test.
sysbench-10mm-execute
In each Sysbench execute phase a given number of client threads operate on the 16 collections. The performance drop-off was significant. For example, at 128 client threads the TokuMX throughput was 150.98 transactions per second (TPS) while TokuMXse performed 54.98 TPS, for a 64.9% performance loss.


Benchmark 4 : > In-Memory Sysbench
Standard Sysbench workload (in-memory), 16 collections, 1 million documents per collection, 8G cache, and directIO. Each Sysbench transaction consists of several queries and aggregations, an indexes update, and unindexed update, a delete, and an insert.
sysbench-1mm-execute
In-memory Sysbench the performance drop-off was significant as well, but less than the IO bound test. AT 128 client threads the TokuMX throughput was 2181.20 transactions per second (TPS) while TokuMXse performed 1201.20 TPS, for a 44.5% performance loss.


Benchmark 5 : > In-Memory primary key point queries
Primary key point queries, standard Sysbench schema (in-memory), 16 collections, 1 million documents per collection, 8G cache, and directIO.
point-primary
The in-memory primary key point query performance dropped 20.2%, 128003 queries per second (QPS) versus 102025 QPS. As with standard Sysbench testing, this is partly due to the fact that a primary key point query in TokuMX doesn’t require the additional look-up that the storage engine API version for TokuMXse does.


Benchmark 6 : > In-Memory secondary key point queries
Secondary key point queries, standard Sysbench schema (in-memory), 16 collections, 1 million documents per collection, 8G cache, and directIO.
point-secondary
The in-memory secondary key point query performance dropped 48.6%, 86600 queries per second (QPS) versus 44494 QPS.


Benchmark 7 : > In-Memory primary key range queries
Primary key range queries, limit 1000 documents per range scan, standard Sysbench schema (in-memory), 16 collections, 1 million documents per collection, 8G cache, and directIO.
range-primary
The in-memory primary key range query performance dropped 48.8%, 2593 queries per second (QPS) versus 1327 QPS.


Benchmark 8 : > In-Memory secondary key range queries
Secondary key range queries, limit 1000 documents per range scan, standard Sysbench schema (in-memory), 16 collections, 1 million documents per collection, 8G cache, and directIO.
range-secondary
The in-memory secondary key range query performance dropped 6.9%, 1195 queries per second (QPS) versus 1112 QPS. I’m not sure [yet] why this drop is much less than the primary key range query test, it may partly be the case that the TokuMX performance is gated to begin with.


As I mentioned prior, we plan on offering a release candidate of TokuMXse soon, and it will be announced on this blog when it is available. Any feedback is appreciated.

*** All tests were run on a Dell R710 server : Ubuntu 14.04, 2 x Intel Xeon L5520 CPUs, 48GB RAM, 8 x 10K SAS in RAID10 (~2000 Random IOPs)

PREVIOUS POST
NEXT POST

2 Comments

  • […] The major work in MongoDB 2.8 is the creation of a “storage engine” API within the server.  This means that in the future, it will be much easier to create a product like TokuMX which uses a different storage implementation but keeps the same networking, clustering, and data modeling stack as MongoDB.  TokuMXse is the name of a new product we’ll add to the line, existing alongside TokuMX, which is an implementation of that storage engine interface using the same core TokuFT storage library as TokuDB and TokuMX.  We’ve written about some preliminary performance results comparing TokuMXse to TokuMX. […]

Leave a Reply