The Shard Revisited: Tools and Techniques Used at Etsy

Trends in Architecture and Design
12 November 11:00am - 11:50am @ Orchard 1
Experience level: 
50 minutes conference
Running a data store at the scale of a site like Etsy took several phases of iteration to get to where it is today. This included the data migration from a monolithic postgres database to a sharded mysql store, in addition to creating in depth tooling to adapt with our growing needs along the way. At Etsy, we made a decision to horizontally scale our data by sharding it across MySQL multi-master replicated shard pairs. This gives us the benefit of infinite horizontal scalability, redundancy within each shard, excellent fault tolerance, and improved performance. Etsy uses a lookup based approach where index servers are queried initially for the shard the data resides on. This gives us the flexibility to easily move data around based on capacity requirements. One tool covered with regards to this is the shard migrator. Learn how this approach is much more granular than other shard paradigms that are built to move or split entire databases when better performance is needed. This talk will cover other tooling required to run a massively sharded data store, such as schemanator, our tool for making schema changes on the database without downtime. In addition, it will cover problems such as how we deal with development data at scale, including modeling hard to reproduce use cases using a proxy to the production database. Etsy has a rigorous sharded datastore that is horizontally scalable and along with that comes necessary tooling to run daily operations. Learn about the architecture and discover the tooling created over the years to solve some of the challenges of running a large cluster.


Dev Infrastructure Tools Lead Engineer, Etsy
John Goulah has been working in New York City over the last several years for a number of web sites in both technical and management roles, as well as the co-founder of several startups. Having spent much of his youth touring in rock bands and hacking from the road, he is no stranger to crowds, be it a smoke filled room or presenting to the company board. He strives for non mundane tasks and has automated himself out of his last few endeavors, which has landed him in his current role as an Engineer at Etsy, the leading marketplace for handmade goods.