Percona Live 2017 Open Source Database Conference

April 24 - 27, 2017

Santa Clara, California

Vinyl: why we wrote our own write-optimized storage engine rather than chose RocksDB

Sponsor Percona Live Open Source Database Conference

Vinyl: why we wrote our own write-optimized storage engine rather than chose RocksDB

 26 April - 1:00 PM - 1:50 PM @ Room 203
Experience level: 
Intermediate
Duration: 
50 minutes conference
Tracks:
Developer
Topics:
RocksDB
Other OSDB
Fundamentals

Description

Tarantool 1.7 features a new storage engine for solid state and rotating disks: Vinyl. Vinyl implements log structured merge trees, just like Cassandra or RocksDB. In this talk, I will begin with discussing log structured merge trees, their architecture and implementation challenges. I will quote Mark Callaghan and re-assert that read, write and space amplification define the new cost model for write optimized storage. Then I will bring up a less famous quality metric for a good database algorithm: predictable, low latency. And we'll see how this metric alone could challenge the implementer and turn the task of achieving good user experience into a holy grail. I will show how we approached these problems in Vinyl and why RocksDB could not fit. Tarantool actor-based model requires that the database internals do not use locks. And this alone is both a barrier of entrance for RocksDB and a boost for Vinyl. We will discuss how in-memory technology, used for Tarantool in-memory engine, was blended with log-structured tree approach to reduce memory overhead and implement efficient secondary keys. Then we will discuss compaction: what strategies are available for multi-level compaction, and distill those for which efficiency is based on evidence, rather than hand-waving. This is a talk from a database engineer working very close to petabyte-sized clusters, and it's about a year of fun and hard work of a skilled engineering team. This is also a talk about open source since all of the software discussed is available in our recent stable release.

Speakers

Konstantin Osipov's picture

Konstantin Osipov

Developer, Architect, Tarantool.org

Biography:

Konstantin is a software engineer with Mail.Ru Group and lead developer of Tarantool - an open source Lua application server and in-memory database. Prior to Tarantool Konstantin was a core member of MySQL development team.

Share this talk