You are viewing content from a past/completed QCon - March 2021. Check out our upcoming events.

User Simulation for Rapid Outage Mitigation

Due to unforeseen circumstances, our Speaker today was not able to join us. Instead of "Incident Analysis: Your Organization's Secret Weapon by Nora Jones we are pleased to bring you: “User Simulation for Rapid Outage Mitigation” by Carissa Blossom from Uber’s Production Engineering team. Carissa’s presentation is pre-recorded from a former QCon Plus and features a Q&A with Haley Tucker, Senior Software Engineer from Netflix’s Resilience Team. We apologize for the inconvenience and hope you enjoy the talk.

Uber operates in over 10,000 cities across the world, with different offerings and features in each market. People all over the world rely on Uber for critical aspects of their daily lives. Uber can be their source of income, their commuting strategy, or their ride to the hospital. In COVID times, our role is all the more critical with Uber Eats acting as the central source of income for an increasing number of restaurants and couriers. With so much at stake, we don’t have the luxury of waiting for aggregate production tracking metrics to notice that some discernible percentage of users are experiencing an issue -- the changes made to our 4000+ microservice architecture are rolled out by both engineering and operations teams leveraging multiple different tools at the city, zone, region and global levels at a frequency that is impossible to coordinate. In this environment, how does Uber maintain a high level of reliability and prevent outages before they are felt by users?    

In this talk, I walk through the independent, external monitoring service that Uber developed to identify issues in production at the individual city level all across the globe, and how we leveraged composable integration tests simulating thousands of diverse, test users to cut our time to mitigation in half. Attendees will also learn how Uber surfaces and predicts its most dire outages and how the combination of machine learning and tracing enables us to reliably narrow down the root cause of an outage. 


Carissa Blossom

Member of Production Engineering team for Eats & Delivery @Uber

Carissa Blossom is a member of the Production Engineering team for Eats & Delivery at Uber. She is also an Incident Commander for Ring0, Uber Engineering’s primary task force for critical outage mitigation. In her four years at Uber, she has served as Production Engineer for Eats, Software Networking Edge and Marketplace.


Learn more about the organizations that joined us on this journey