Infrastructure at Scale

Trends in Architecture and Design
24 April 11:10am - 12:00pm @ Ballroom A

Many operations teams are tasked with competing goals of quickly delivering reliable infrastructure and also ensuring high uptime for all infrastructure components. By the time an organization has more than 50,000 hosts, this process is very well defined, usually the result of some very expensive mistakes made along the way. The purpose of this talk is to share best practices and lessons learned as we have scaled many organizations from tens to thousands of servers. We'll ask, and answer, questions like:

- How & what should we automate?
- What metrics matter?
- When does change control process become important?

We'll also talk about the rule of threes, why user ids are important, why time matters, and many other details that are often overlooked at the early phases of scaling.


Ryan Lowe
Production Engineer, Square, Inc.
Ryan is a Platform Engineer at Square, where he manages thousands of instances of MySQL and PostgreSQL. Before joining Square, Ryan helped build and scale some of the largest web, social, gaming, and mobile properties as a Consultant at Percona. As an author, speaker, and active contributor to F/OSS projects, his expertise extends to a variety of data stores, cloud and big data, and beyond.
Fernando Ipar
Senior Consultant, Percona
Fernando works as Senior Consultant at Percona. Prior to joining the company, he worked as a contractor for several financial services institutions, telcos, and smaller technology providers. During that period, he acquired experience managing different UNIX type systems, developing or improving fault tolerant software, and managing small teams of technically skilled people. He has been involved in the Open Source Community as a user since 1997, and as a contributor since 2001, when he deployed his first production MySQL server and right away started working on a set of scripts to manage master/slave pairs via the mysql-ha (later renamed highbase) project. Since then, he has continued to be actively involved in the community, more recently with contributions to Percona Toolkit and MySQL Fabric. He lives with his family in Montevideo, Uruguay. When not working, he enjoys family life, music and coffee.
Marcos Albe
Senior Support Engineer, Percona
After 12 years working as a PHP/JS developer for local and remote firms, Marcos decided to pursuit true love and become full time DBA, so he has been doing MySQL Support at Percona for the past 5 years providing lead web properties with advise on anything-MySQL and in-depth system performance diagnostic help.
Dave Apgar
Production Engineer, Square
Dave Apgar works for Square as a Production Engineer, focusing on system automation, monitoring, and obviously databases. Prior to Square he worked at Yahoo! as High Availability Architect and lead developer on Yahoo!'s global server load balancing platform.