XDB: Shared MySQL hosting at Facebook scale

DevOps
16 April 11:00AM - 11:50AM @ Ballroom C

Experience level: 
Intermediate
Duration: 
50 minutes conference

Rate This Session

As companies grow, they tend to accumulate many special-purpose databases, with diverse workloads and requirements. Once a company has hundreds of engineering teams and thousands of engineers, provisioning and managing these "special snowflake" databases can consume an increasingly large percentage of a DBA team's time. This is a scalability problem, but of a different sort than we are typically accustomed to. As with all scalability problems, we address this situation at Facebook by building automation software. This session describes XDB, our in-house system for self-service database allocation and management. XDB empowers any engineer at Facebook to create a fully-replicated production-ready MySQL database in mere seconds through a web interface, or even create a large collection of sharded databases through an API. This talk will cover the motivations and history of XDB, including the tedious manual DBA processes that it replaced. We will then discuss the design and implementation of the system, with a focus on what we have learned and how attendees can apply this knowledge to their own environments. Finally, we will talk about future possibilities of open-sourcing XDB, and challenges we still need to overcome to make the system fully generic and reusable by the community.


Speakers

Director of Engineering, Tumblr
Evan is a software engineer and database automation expert based in New York City. He was originally engineer #4 at Tumblr, where he recently returned to direct several backend engineering teams. Previously he was a member of Facebook's MySQL Infrastructure team, where he deployed GTID to the world's largest MySQL fleet, and led development of the company's internal DBaaS platform. He has been using MySQL since 2003, and is the creator of two open source tools: Jetpants, a shard orchestration toolkit; and Skeema, a schema management system.

Slides