"Just" shard it

MySQL Case Studies
15 April 2:00PM - 2:50PM @ Ballroom A

Experience level: 
Intermediate
Duration: 
50 minutes conference

Rate This Session

Etsy started with a monolithic postgres database and moved to a horizontally sharded mysql store with lookup table and master-master replication. Scaling your data store is never as easy as just adding more hardware though, even with sharding. In this talk, we'll go over the pains and perils of sharding databases and how we solved some of the problems in Etsy's second take on sharding, what we call logical sharding. Logical sharding is where we put several databases on one mysqld, as opposed to one database per server. Upon hitting a resource limit on a box (CPU, Memory, Connections, etc) that would normally require individual row migration, we can instead do file level migrations of a database, replicate, then drop previous databases, which is super fast and a cleaner way to delete old unnecessary shard data. We'll also go over how logical sharding benefits other daily operational challenges like schema changes and backups, as well as makes life easier for our downstream analytics consumers. Etsy is a fast growing site with over 40 million members and 26 million items listed. Learn about Etsy's painpoints in growing a 120TB+ sharded MySQL datastore and how the architecture and tooling evolved over time.


Speakers

Software Engineer, Etsy
Maggie Zhou works on the Core Platform team at Etsy where she builds infrastructure to manage scaling challenges in the database and API. Before Etsy, Maggie spent some time at Hacker School and wrote software at Google. 
Principal Software Engineer, Etsy Inc.
Keyur is a Systems Engineer at Etsy. He's part of the team that keeps MySQL, GearmanD, Memcached, PHP, and the Linux kernel humming smoothly. Prior to Etsy, he was a software engineer at PayPal. 

Slides