Managing large sharded topologies with Jetpants

Huge, rapidly-growing MySQL architectures can be quite challenging to maintain. Sharding helps tremendously with horizontal scalability, but introduces a great deal of operational complexity. In order to stay sane while juggling hundreds of database servers and tens of billions of rows, sophisticated automation becomes a necessity.

To solve this problem at Tumblr, we created Jetpants, a multi-purpose toolchain for managing giant MySQL topologies. In this session, we will outline how we implemented fast, resilient automation for many complex operational tasks.

  • Rebalancing range-based shards without locking or downtime
  • Cloning slaves to multiple destinations simultaneously
  • Defragmenting tables efficiently
  • Handling replication promotions when the master has failed

We’ll also discuss why we opted for a range-based sharding scheme, without a central lookup service or pre-allocation of tiny shards -- and how InnoDB's clustered indexes allow Jetpants to split up a billion-row range-based shard in just a few hours.

Finally, we'll cover the design of the Jetpants plugin system, and how you can use it to seamlessly integrate your site's hardware asset management system, monitoring tools, query killers, and more.

This session is geared towards operations engineers, DBAs, and automation software developers. Some previous familiarity with MySQL replication and sharding fundamentals is assumed.

Track: 
Tools
Experience level: 
Intermediate

Schedule info

Room: 
Nolita

Schedule Info

2 October 13:30 - 14:20 @
Nolita

Speakers

Evan Elias's picture
Core Application Engineering Lead, Tumblr

Evan is a team lead at Tumblr, where he develops database solutions that scale to tens of billions of rows spanning over 200 MySQL installations. He has previously worked at Six Apart and Harvard University, and his software is used by major publications such as The New Yorker and Consumer Reports. He is intimately familiar with the internals of four major blogging platforms, and has been developing online software since the days of dial-up bulletin board systems.

Slides


Sponsored By