Backing Up Facebook

9.16667
Average: 9.2 (6 votes)

Logical and binary backups of thousands of database servers with multiple geographic locations, with complete coverage in 24 hours, requires a massively parallel backup architecture. Decentralized storage, multi-staged repositories, and real-time binary log collection enable rapid disaster recovery, but also provide surprising day-to-day benefits.

Tools Facebook has developed to support this infrastructure include: Real-time binlog collection for both masters and slaves, to maximize data integrity and allow point-in-time recovery; a web server that automatically streams archived binlogs allows fast slave catchup; a new wrapper for xtrabackup that replaces innobackupex and implements full and incremental binary backups, with automatic retrieval of credentials, streaming of the xtrabackup logfile to the remote target (never consuming local disk), and operation under a non-privileged user to minimize risk; and continuous automated restores ensure both mysqldump and xtrabackup archives remain intact and usable, across a wide variety of software and tool versions.

Track: 
Database Administration
Experience level: 
Intermediate

Schedule info

Time slot: 
11 April 15:30 - 16:20
Room: 
Ballroom F

Schedule Info

11 April 15:30 - 16:20 @
Ballroom F

Speakers

Kevin Knapp's picture
DBA, Facebook

I’ve been steadily scaling MySQL up for the last 5 years at Godaddy.com and then Facebook.com, using a combination of bash, php, and python in a vain attempt to replace myself with a collection of scripts.

Eric Barrett's picture
Storage Engineer, Facebook

Eric Barrett has 13 years of experience the storage and database industries, having worked at NetApp, Facebook, and a few startups in between. He currently leads the operations development group that handles MySQL backups at Facebook.




Sponsored By