{"id":2060,"date":"2020-03-12T21:49:45","date_gmt":"2020-03-12T21:49:45","guid":{"rendered":"https:\/\/uptime.com\/blog\/?p=2060"},"modified":"2025-01-08T15:21:29","modified_gmt":"2025-01-08T15:21:29","slug":"how-to-improve-downtime-response","status":"publish","type":"post","link":"https:\/\/uptime.com\/blog\/how-to-improve-downtime-response","title":{"rendered":"How to Improve Downtime Response: Error Budgeting and Unplanned Downtime"},"content":{"rendered":"

Every one of us reading this blog has seen a fire spring up and quietly walked away from the impending chaos. And everyone one of us has managed to live this long because we understand when to react to a fire.<\/p>\n

\"\"<\/p>\n

A real fire affects our Service Level Objectives (SLO), and affects the user base. You need to figure out where it is, what started it, and what your team will do about it, and you need to do that now. Then you need to examine how you responded to see whether any holes or roadblocks impede your progress.<\/p>\n

What are the key factors that have to align to get you out of bed with alarms and whistles? What wakes you up?<\/p>\n

We know incidents don\u2019t conform to our schedule, and that not every fire is worth bringing in the cavalry. So today, we\u2019re going to ask ourselves if this is really worth getting out of bed for. How do we not only prioritize alerts, but improve our response time?<\/p>\n

Error Budgeting Your Bedtime<\/h2>\n

Vince Lombardi said \u201cPerfection is not attainable, but if we chase perfection we can catch excellence.\u201d SLOs built on unrealistic fulfillment are the first recipe we have for disaster. You are guaranteed to overpromise and under-deliver to a dwindling audience over time.<\/p>\n

How to Error Budget<\/h4>\n

Good SLOs use something called an \u201cerror budget\u201d to ensure a manageable user experience. Secondarily, good SLOs make sure that the devops team isn\u2019t quietly plotting to burn the place down when no one\u2019s looking.<\/p>\n

\"\"<\/p>\n

Error budgeting is another way of referring to the amount of unplanned downtime your organization deems acceptable. Here\u2019s a table from Google that provides figures for the math behind the nines<\/a>, but you want as many nines as you can conceivably handle. Budgets will differ based on team and infrastructure size. One person managing an app with hundreds of users has very different concerns than a team managing a service with hundreds of thousands logged in concurrently.<\/p>\n

But error budgets serve another important function: they define when to stop what you\u2019re doing and fix something. An SLO may or may not be legally binding, but it does provide a framework for how to treat systems under stress.<\/p>\n

Simply put: you can continue deployments under ABC conditions, but stop everything for analysis and reliability work under XYZ conditions.<\/strong><\/p>\n

The best SLOs are extremely specific about when and how the team needs to react to unplanned downtime. If the error budget is consumed too quickly, the team must be able to explain what happened and create an incident report that details what went wrong. That report will help further define your escalation policies.<\/p>\n

Defining an Escalation Policy<\/h4>\n

So, then, the question becomes: what is unplanned downtime? Simple answer: anything not within your control that triggers downtime. But when we think about SLOs, the answer gets a bit muddled. Let\u2019s say we have a website that is OK 200, with a support chat function that is closer to :sadface: 400. Does that down chat factor into our SLO?<\/p>\n

Is it worth getting out of bed for?<\/p>\n

Yes, we should know it\u2019s down and we should have a plan to fix it. Do we need to act on that at 3 AM when we first receive the alert it\u2019s dead? Probably not.<\/p>\n

So, part of error budgeting involves wasting less time figuring out whether the fire is a big one. When we know more about which systems affect the customer experience, we can waste less time deciding whether an alert is worth acting on.<\/p>\n

You can take this a step further. Finding yourself getting out of bed just to reboot a service? Automate it! Recurring errors giving you the blues? Tell tier 1 how to fix them while you sleep so you can feel refreshed enough to do the root cause analysis tomorrow and figure out how to stop them.<\/p>\n

All of these methods help improve downtime response without excessive costs to your error budget.<\/p>\n

Two Steps to Beating Your Error Budget<\/h2>\n

There are two steps you can take to improve your downtime response and ultimately beat the numbers you’ve set as your error budget.<\/p>\n

Collect Data on Uptime and Performance<\/h4>\n

The first is to start collecting performance data now. Here\u2019s a handy link to create a RUM check<\/a> if you\u2019ve not already done so. Use this performance indicator as your first line of defense, because the user experience is why we all show up in the morning.<\/p>\n

\"rum-screenshot\"<\/p>\n

Real User Monitoring<\/a> tells us how our website is performing, and it\u2019s powered by real user data. When you review a RUM report you step into the user\u2019s shoes for a clear picture of how they perceive the website. It\u2019s also got a built-in threshold for performance, alerting you when the load time exceeds the value you define.<\/p>\n

Performance checks like RUM are the sensors on our star cruiser, diligently informing us of the dangers ahead.<\/p>\n

The more performance thresholds you can identify as meaningful, the more data you have for first response. You know when it\u2019s slow, when slowness becomes downtime, and the severity and length of that downtime.<\/p>\n

Escalate Unplanned Downtime<\/h4>\n

The second step we can take to improve downtime response is to practice meaningful escalations.<\/p>\n

The kind of downtime you\u2019ve experienced is crucial in determining what you do with your alerting. If we circle back to our RUM check, our defined threshold helps determine when it\u2019s time to examine the problem. Until that threshold is met, it\u2019s nice to know performance is fast or slower but service levels must have an acceptable operating range. No need to blow the horn just yet.<\/p>\n

If our RUM check triggers an alert, and some of our HTTP(S) checks<\/a> go down, we might need to get out of bed.<\/p>\n

We can think of escalation<\/a> in terms of when and where. When should we escalate to the next tier, and where will our message have the greatest impact. First, let\u2019s tackle the \u201cwhere\u201d portion of escalations.<\/p>\n

Where is easy: wherever your team lives. Do you live on Slack? Are you all married to some other provider? Do you exist on mobile, or have internal dashboards and ticketing? Where is easy to define when we think about how your team works.<\/p>\n

Now let\u2019s think about \u201cwhy\u201d, which is the really interesting part.<\/p>\n

Modern applications are full of complex and interlocking systems, with redundancy built into everything we do. If one server from a farm of 10,000 goes down, is it really worth getting out of bed for?<\/p>\n

You would know if you had more clarity on which one went down.<\/p>\n

An alert that has no actionable takeaway isn\u2019t very useful. You\u2019ve just been roused out of sleep to learn that you shouldn\u2019t or can\u2019t actually do anything. Alerts like these are over before you see them, or they have little to no impact on the end-user.<\/p>\n

On the other hand, too many alerts will drive anyone to levels of insanity that create DevOps horror stories that make the front page of Reddit.<\/p>\n

\"\"

Photo by Lewis Parsons on Unsplash<\/p><\/div>\n

Why we escalate becomes a question of the cost of downtime to your organization. We looked at some hypothetical downtime costs during Black Friday 2019<\/a>, if you\u2019re interested to see what these calculations might take into account.<\/p>\n

Simply put: what\u2019s the end-user cost to something going down? Is there potential for a bounce or a lost sale? If so, how apocalyptic is that threat?<\/strong><\/p>\n

Here are some quick guidelines you can use to assist in escalating:<\/p>\n