Managing large sharded topologies with Jetpants

Huge, rapidly-growing MySQL architectures can be quite challenging to maintain. Sharding helps tremendously with horizontal scalability, but introduces a great deal of operational complexity. In order to stay sane while juggling hundreds of database servers and tens of billions of rows, sophisticated automation becomes a necessity.

To solve this problem at Tumblr, we created Jetpants, a multi-purpose toolchain for managing giant MySQL topologies. In this session, we will outline how we implemented fast, resilient automation for many complex operational tasks.

• Rebalancing range-based shards without locking or downtime
• Cloning slaves to multiple destinations simultaneously
• Defragmenting tables efficiently
• Handling replication promotions when the master has failed

We’ll also discuss why we opted for a range-based sharding scheme, without a central lookup service or pre-allocation of tiny shards -- and how InnoDB's clustered indexes allow Jetpants to split up a billion-row range-based shard in just a few hours.

Finally, we'll cover the design of the Jetpants plugin system, and how you can use it to seamlessly integrate your site's hardware asset management system, monitoring tools, query killers, and more.

This session is geared towards operations engineers, DBAs, and automation software developers. Some previous familiarity with MySQL replication and sharding fundamentals is assumed.