MySQL, the ebay Classifieds Way

Best Practices for Businesses
12 November 4:00pm - 4:50pm @ Orchard 2
Experience level: 
Intermediate
Duration: 
50 minutes conference
Talk Submission for the Percona Live Conference 2013. The ebay Classifieds Group (eCG) is the part of the ebay Corporation dedicated to the Classified Ads market. We help people find whatever they're looking for in their local communities, from apartments to jobs, from couches to cars. In the process, we churn through amazing amounts of data, most of which is, at least at some point, stored in MySQL Databases. I'm the most experienced Database Administrator servicing the eCG. I am responsible for more than 400 MySQL database servers across 15 business units in 10 different time zones. I also support the 19 Development and 5 Site Operations teams responsible for the development and operations of the systems we build and run in the house. In this session I will share with the audience the issues we face and the solutions we adopted in the subjects of cluster architecture, high-availability, stability, disaster prevention and recovery, and performance monitoring and tuning. I will address the following topics: * What kind of database-related issues and concerns we face at ebay Classifieds Group business units, what motivates us; * How this motivation drives our Master-Slave Cluster Architecture; * How we manage our configuration using Puppet, and a three-tier configuration file; * What kind of hardware we use and how we organise our database files and distribute them across multiple disks for better performance; * How we achieved the reliability and stability we require using Percona Server with XtraDB; * How we deploy and regenerate the databases in production using Percona XtraBackup; * How we manage our master database with Master HA and Netscaler Load Balancers, and how we are able to fail-over to a new master when disaster strikes, even between data-centres. * How we monitor our databases for Performance issues, and what role does pt-query-digest plays on this; * How we are integrating Box/Anemometer with pt-query-digest to help exposing to the developers as much information as possible about problematic queries in production; * How we use pt-table-checksum to locate and prevent replication divergence; * How we keep our data backed up constantly and allow point-in-time recovery using pt-slave-delay, and how we made this work with Master HA using a black-hole slave as binary log proxy; * How we use CollectD, Graphite, and GDash to monitor our databases and produce beautiful and very useful graphs with a lot less effort than other popular alternatives.


Speakers

Database Administrator, ebay Classifieds Group
Biography: 
Luis has been working with Information Systems since 1999. He worked as a developer, system administrator, and has been dealing with Relational Databases since 2001, when he started as a certified Oracle DBA.<p>Luis is a proud member of the <a href="http://london.pm.org/">London Perl Mongers</a> since 1999. He is also the author and maintainer of a handful of <a href="http://search.cpan.org/~lmc">irrelevant Perl Modules</a>. He <a href="http://nl.linkedin.com/in/luismottacampos/">worked</a> for <a href="http://www.ripe.net/">RIPE NCC</a>, where he had the opportunity to manage the <a href="http://whois.ripe.net/">RIPE Whois Database</a> and maintain the software behind it.<p>Since he joined <a href="http://www.ebayclassifiedsgroup.com/">ebay Classifieds Group</a> in 2010 he has been working exclusively with MySQL databases, servicing ebay Classifieds Group business units in more than 13 countries.<p>When time allows Luis blogs about practical Database Administration in his personal blog, <a href="http://deafbysalami.blogspot.com">Deaf by Salami</a>, watches "House, MD" and Science Fiction movies, and travels around the world making <a href="http://www.flickr.com/photos/luismottacampos/sets/">photos of landscapes and people</a>.