Talks

Cultivating Production Excellence

Taming the complex distributed systems we're responsible for requires changing not just the tools and technical approaches we use; it also requires changing who is involved in production, how they collaborate, and how we measure success.

In this talk, you'll learn about several practices core to production excellence: giving everyone a stake in production, collaborating to ensure observability, measuring with Service Level Objectives, and prioritizing improvements using risk analysis.

Presented at QCon London 2019, Velocity San Jose (keynote), DevOpsDays Atlanta, and more. Example slides (from DevOpsDays Minneapolis)

Tradeoffs on the Road to Observability

SRE and infrastructure engineering are about allocating adequate time to do project work that improves the long-term sustainability of our services. But what do we reward SREs for doing? Does your company have a culture of "not invented here" or the converse of "ask the consultants to design it for us"?

We need to step back and ensure that we are doing the most efficient thing with our time as SREs. The main way that we can improve sustainability is empowering other engineers, especially non-SREs, to understand our systems, rather than for the sake of our resumes and/or glory.

Presented at Monitorama PDX 2019 (slides).

Refining Systems Data Without Losing Fidelity

It is not feasible to run an observability infrastructure that is the same size as your production infrastructure. Past a certain scale, the cost to collect, process, and save every log entry, every event, and every trace that your systems generate dramatically outweighs the benefits. If your SLO is 99.95%, then you'll be naively collecting 2,000 times as much data about requests that satisfied your SLI than those that burnt error budget. The question is, how to scale back the flood of data without losing the crucial information your engineering team needs to troubleshoot and understand your system's production behaviors?

Statistics can come to our rescue, enabling us to gather accurate, specific, and error-bounded data on our services' top-level performance and inner workings. This talk advocates a three-R approach to data retention: Reducing junk data, statistically Reusing data points as samples, and Recycling data into counters. We can keep the context of the anomalous data flows and cases in our supported services while not allowing the volume of ordinary data to drown it out.

Presented at SREcon Europe 2019 (slides); upcoming at Monitorama Baltimore 2019.

Publications & Videos

Observability Engineering w/ Charity Majors and George Miranda, in fine bookstores near you as of May 2022!

SRE and DevOps video series w/ Seth Vargo on the GCP YouTube channel

"SRE vs. DevOps: competing standards or close friends?" w/ Seth Vargo on the GCP Blog

"How SRE relates to DevOps" w/ Betsy Beyer and Niall Murphy in the Site Reliability Workbook

Sustainable Operations in Complex Systems With Production Excellence in InfoQ

Framework for an Observability Maturity Model (vendor-neutral whitepaper sponsored by Honeycomb)

"Intersections between Operations and Social Activism" w/ Emily Gorcenski in Seeking SRE

“Jeff Bezos is wrong, tech workers are not bullies” w/ Laura Nolan et al. in the Financial Times

“Our Executives Engaged in Abuse. Don’t Let Kink and Polyamory Be Their Scapegoats.” in Medium Featured Stories

“Google Workers Lost a Leader, But the Fight Will Continue” in Medium Featured Stories

"Interrupt Reduction Projects" w/ John Tobin and Betsy Beyer in USENIX ;login:

"A Hierarchy of SRE Needs" (blog)

"The Myth of Psychological Safety" (blog)

Interviews & Podcasts

GCP Podcast Episode 127 with Seth Vargo, Melanie Warrick, and Mark Mandel

GCP Podcast Episode 139 with Melanie Warrick and Mark Mandel

Screaming in the Cloud Episode 19 with Corey Quinn

Fireside Chat at FutureStack NYC with Matthew Flaming

DevOps/SRE AMA with Charity Majors and Adam Jacob, hosted by Andrew Smirnov of Catchpoint

o11ycast Episode 6 with Charity Majors and Rachel Chalmers

I also frequently sit on panels about management, SRE, and ethics.

Three Years of Misery Inside Google, the Happiest Company in Tech

Nitasha Tiku, Wired, August 13, 2019

To Liz Fong-Jones, a site reliability engineer at Google, the memo's arguments were especially familiar. Google's engineers are not unionized, but inside Google, Fong-Jones essentially performed the function of a union rep, translating employee concerns to managers on everything from product decisions to inclusion practices.

As part of this internal advocacy work, Fong-Jones had become attuned to the way discussions about diversity on internal forums were beset by men like Cernekee, Damore, and other coworkers who were “just asking questions.” To her mind, Google's management had allowed these dynamics to fester for too long, and now it was time for executives to take a stand.

Inside Google’s Civil War

Beth Kowitt, Fortune, May 17, 2019

Much of the organizing efforts have been led by site reliability engineers (SREs). Their remit is to operate the most critical services Google runs. When something breaks, they’re the ones who get paged to fix it. They troubleshoot and diagnose problems, and they are expected to have opinions and questions. “You have to go probe for weaknesses,” says Fong-Jones, who was an SRE, “and also challenge people when you think something that they’re trying to railroad through is not okay.” Within the SRE world, there’s a concept called blameless postmortem—it’s a way of looking back at mistakes made without throwing anyone under the bus. “It’s a fundamental part of the culture at Google,” says Tariq Yusuf, a privacy engineer who’s been with the company almost five years. “It’s an ability to say this is a thing that’s wrong.” Retaliation, he says, removes the core barrier of being able to safely raise issues. “The whole process breaks down.”

Other stories

Community

2017-present: Global Steering Committee Member, SREcon (USENIX)

SREcon Asia/Australia 2022: Program Co-Chair

SREcon Americas {2016,2017,2019}: Program Co-Chair

SREcon Europe {2016,2017}: Program Committee Member

SREcon {Americas,EMEA,Asia/Australia} 2018: Program Committee Member

OpenTelemetry Governance Committee Emeritus