Tarantool - a Lua based database engine and in-memory data grid
In my talk I will focus on a practical use case: task queue application, using Tarantool as an application server and a database. The idea of the task queue is that producers put tasks (objects) into a queue, and consumers take tasks, perform them, mark as completed. The queue must guarantee certain properties: if a consumer failed, a task should return to the queue automatically, a task can't be taken by more than one consumer, priorities on tasks should be satisfied. With Tarantool, a task queue is a distributed networked application: there are multiple consumer/producer endpoints (hosts) through which a user can interact with the queue. The queue itself is a fault-tolerant distributed database: every task is stored in Tarantool database and replicated in multiple copies. If a machine goes down, the state of a task is tracked on a replica, and the user can continue working with the queue through a replica. Total power failure is also not an issue, since tasks are stored persistently on disk with transactional semantics. Performance of such an application is in hundreds of thousands of transactions per second. At the same time, the queue is highly customizable, since it's written entirely in Lua, is a Lua rock, but the code is running inside the database. This is the strength of Lua: one size doesn't have to fit all, and you don't have to sacrifice performance if you need customization. The second part of the talk will be about implementation details, performance numbers, a performance comparison with other queue products (beanstalkd, rabbitmq) in particular, and an overview of the implementation from language bindings point of view: how we make database API available in Lua, what are the challenges and performance hurdles of such binding.
Konstantin is a software engineer with Mail.Ru Group and lead developer of Tarantool - an open source Lua application server and in-memory database. Prior to Tarantool Konstantin was a core member of MySQL development team.