Building a High-Volume Reporting System on Amazon using MySQL, Tungsten, and Vertica

BigDoor is a gamified loyalty platform that helps websites keep and retain their users. We store each publisher's loyalty economy configuration data in MySQL, plus all of their users' transactional data - points they've received for taking actions that stimulate loyalty, engagement, virality and revenue on their site(s). The transactional data flows into backend analytics systems where it's aggregated for reporting purposes and analyzed to produce optimizations that ultimately improve end-user loyalty. In this talk we'll show how we built this system using MySQL and Vertica running on Amazon and the lessons we learned. We will cover the methods we developed to move sharded data from MySQL to Vertica, starting with variations on MySQL Replication and home-grown ETL, then extending to increasing capable solutions based on Tungsten Replicator that scale to many millions of transactions per day. Finally, we'll talk about what we could do better and our plans for growing our analytics.

Track: 
Best Practices for Businesses
Experience level: 
Intermediate

Schedule info

Time slot: 
11 April 15:30 - 16:20
Room: 
Ballroom A

Schedule Info

11 April 15:30 - 16:20 @
Ballroom A

Speakers

Jeff Malek's picture
Co-founder and CTO, BigDoor

Jeff Malek, Co-Founder & CTO – Prior to starting BigDoor, Malek co-founded the company that became Zango in 1999 and became the company’s Vice President of Engineering. Malek held various roles within Zango including serving as Chief Client Architect and Vice President of Product Development. While the company was still in its infancy, Malek authored and tested the first versions of the company’s desktop software, and aided in the development of the middle tier and backend systems that supported it. Since then, his responsibilities have included the direction of the Project Management, Web Development, Internal Systems Development, Quality Assurance, Desktop Applications Development, and Product Management teams. Malek holds a Bachelor of Arts degree from the University of Colorado and suspended work towards a Masters in Computer Science at the University of Southern Oregon in order to devote himself full-time to his career.

Slides




Sponsored By