Percona Live: Data Performance Conference 2016 Logo

April 18-21, 2016

Santa Clara, California

Running Hadoop on Mesos at Uber

Running Hadoop on Mesos at Uber

 19 April 01:20 PM - 2:20 PM @ Ballroom E
Experience level: 
Beginner
Duration: 
50 minutes conference
Tracks:
New and Trending Topics
Operations and Management
Topics:
Hadoop
Spark
Devops

Description

Companies often have to build their infrastructure to handle the peak load of their systems while being over-provisioned during off-peak times. At Uber we are leveraging Apache Mesos, Apache Aurora, and other frameworks to make our systems more efficient, reliability and to take advantage of excess off-peak capacity for use in resource-intensive jobs like Hadoop. We will go over why we are using Apache Mesos, the pros and cons of the frameworks we are using, and the benefits and challenges of running a multi-framework environment to handle the various services and jobs we have. Additionally we will discuss the design and implementation of our solution and show how we use puppet to tie it together. The presentation will also cover operation and management of our Apache Mesos cluster to make the shifting of resources easy and the loss of hardware seamless. The target of this talk is DevOps but it should be accessible to a general audience. The goal of this presentation is to provide a general understanding of how to set up and use Apache Mesos and which frameworks are suited for which jobs.

Speakers

Brandon Bercovich's picture

Brandon Bercovich

Senior Software Engineer, Uber

Biography:

Brandon has been working in the tech industry for over 15 years. Recently he has been working at Uber to leverage Apache Mesos to help improve datacenter utilization.

Share this talk


Slides