Percona Live Online Agenda Slot: Tue 20 Oct • New York 12:30 p.m. • London 5:30 p.m. • New Delhi 10:00 p.m. • Singapore 12:30 a.m. (next day)
Not so long ago, operations specialists worked much like today’s data engineers do: with specialized skills, they were the people who kept sites running, who responded to emergencies, and who—unfortunately—spent much of their time dealing with incidents and other “fires.” When the DevOps revolution came, this began to change. Better tools, better practices, and better culture shaped how Ops folks worked. A subset of that DevOps culture soon emerged: Site Reliability Engineers. These were people whose focus was not just on the day-to-day deployment of applications, but running platforms, products, and services with very high performance, very large scale, and with very high demand for reliability. Data Engineering was left out of this revolution.
But it is not too late! By taking concepts from SRE culture, in particular, the theory of Service Level Objectives, we look at how teams operating and developing data platforms and data products can be built more reliably through the use of quantitative measures and product thinking. This talk will discuss concrete examples of the benefits of this approach for data teams and how organizations can benefit from this mindset.
Why is your talk exciting?
I see a lot of the same pains over and over in the data engineering world: data engineers spending too much time firefighting or dealing with ad hoc requests to innovate, data scientists pained by long lead times for pipeline engineering, and poor data quality eroding trust and leading organizations to make “gut” decisions instead of data-driven ones. This doesn’t have to be our world. The reality that many data engineers face today is similar to the one ops folks faced years ago, before Site Reliability Engineering (SRE) practices began to solidify. However, most of the learnings in the SRE space, particularly Service Level Objective (SLO) theory, don’t translate directly to the data space unless we adapt them to our unique reality. But if we can build solid, data-driven best practices, we can achieve so much—less firefighting, more creation; less guesswork, more trust.
Who would benefit the most from your talk?
Certainly, data engineers will benefit. But also managers, executives, and product owners will all benefit from learning how we can deliberately craft data engineering practices to optimize for reliability. Data should be a business driver, but too often I see it as a cost center. We need to change that calculus.
What other talks are you most looking forward to?
I’m excited to see Karen Ambrose’s talk. I think that building technology to address rapidly-evolving crises is an enormous challenge, and frankly, I think that maybe a lot of organizations have been too complacent and risk averse to manage rapid pivots. I’m really curious to hear the story about how people came together to change the status quo in an effort to literally save the world.
Is there any other question you would like to answer?
There’s a Millennial Prize Problem or two still unsolved, and I’d love to answer one of those.