Vinyl: why we wrote our own write-optimized storage engine rather than chose RocksDB
Tarantool 1.7 features a new storage engine for solid state and rotating disks: Vinyl. Vinyl implements log structured merge trees, just like Cassandra or RocksDB. In this talk, I will begin with discussing log structured merge trees, their architecture and implementation challenges. I will quote Mark Callaghan and re-assert that read, write and space amplification define the new cost model for write optimized storage. Then I will bring up a less famous quality metric for a good database algorithm: predictable, low latency. And we'll see how this metric alone could challenge the implementer and turn the task of achieving good user experience into a holy grail. I will show how we approached these problems in Vinyl and why RocksDB could not fit. Tarantool actor-based model requires that the database internals do not use locks. And this alone is both a barrier of entrance for RocksDB and a boost for Vinyl. We will discuss how in-memory technology, used for Tarantool in-memory engine, was blended with log-structured tree approach to reduce memory overhead and implement efficient secondary keys. Then we will discuss compaction: what strategies are available for multi-level compaction, and distill those for which efficiency is based on evidence, rather than hand-waving. This is a talk from a database engineer working very close to petabyte-sized clusters, and it's about a year of fun and hard work of a skilled engineering team. This is also a talk about open source since all of the software discussed is available in our recent stable release.
Konstantin is a software engineer with Mail.Ru Group and lead developer of Tarantool - an open source Lua application server and in-memory database. Prior to Tarantool Konstantin was a core member of MySQL development team.