Everyday We’re Shuffling — Online Shard Migration at Facebook

Description

Any engineer interested at how we run MySQL at Facebook can tell you that we shard our data, and run multiple shards per MySQL instance. This has been the widely accepted approach to “shared nothing” scaling of MySQL in large installations. What do you do when classic sharding is no longer sufficient?

Older shards can grow beyond their expected size, and cause imbalance. Some shards can become too hot, and require more resources to be allocated. In some cases, they can even be underused. These imbalances cause very large inefficiencies in absolute numbers at large scale.

Learn how we built our online shard migration solution. We’ll start from the building blocks that explain how we move databases between replica sets, which could be applicable at any scale. Then we will dive into  the details of how we constructed an intelligent system which balances capacity and allocates resources in an optimal way, without human intervention.

Speakers

Daren Seagrave

MySQL Infrastructure Engineer, Facebook

Biography:

Daren is a MySQL Infrastructure Engineer at Facebook. Previously worked as Architecture and Data Services Manager at Web Reservations International.

Share this talk