Linux Kernel Extension for Databases
Several years ago I had a task to develop custom storage for IM history. Started from simple index tree over mmap(2), I quickly found that mmap(2), madvise(2) & Co. do poor job to be used as the foundation for database engine. So I switched to direct IO and started to develop my own memory management, advanced eviction, IO scheduling with adaptive read ahead and surely transaction processing. All these things are what database developers do for decades. Not surprisingly, modern operating systems do this stuff on their own, not suitable for databases, way. There are virtual memory management which evicts pages, IO schedulers and read ahead. File systems handle their operations in transactional manner as well. However, database engines still must implement the same mechanisms on their own. In this presentation we'll look at Tempesta DB, a part of Tempesta FW project (http://github.com/natsys/tempesta). Tempesta DB extends Linux kernel such that it implements memory management, eviction and IO mechanisms suitable for engines of user-space databases. It's designed for near real-time workloads, so currently it requires that all data must fit in RAM. We will discuss these topics: - Memory management and IO in OS and databases, using Linux and InnoDB as the examples; - Efforts of Linux kernel community to develop necessary kernel interfaces for databases; - Linux kernel adjustments introduced by Tempesta DB; - Tempesta DB interfaces for custom databases; - Tempesta DB applications servicing Web content and firewall rules in Tempesta FW; - Opportunities in development of generic clustered databases on top of Tempesta DB.
CEO, Tempesta Technologies
Alexander is CEO at Tempesta Technologies Inc. and lead developer of Tempesta FW. He's also founder of NatSys Lab., a company providing consultancy in high performance computing in Linux/x86-64 environment. Alexander has more than 10 years of experience in Linux kernel and 5 years in MySQL, InnoDB and Galera Cluster.