3 Ways that Continuous Delivery and Incident Response Enable Fast Feedback

One of the most impressive books on DevOps, “The DevOps Handbook”, emphasis three fundamental principles underpinning DevOps: systems thinking, amplify feedback loops, and continual experimentation & learning.

Amplifying feedback loops is described as creating the right to left feedback loops, which helps corrections to be made continually, by Gene Kim in his blog post. But, let’s start with why we should do this in the first place.

Why should we amplify feedback loops?

The first version of a new feature or product often doesn’t fully cover the needs of our customers. Even when we spend weeks or months into building something, the final product is often doomed to miss important pieces. Most of the time, even customers don’t know what they need!

In the end, there is no way we can avoid shipping useless or broken software if we are going fast enough. Instead of striving to avoid this, we should embrace the idea of shipping small pieces of value.

If we ship faster, we can break and fix things when they are small. When things get bigger, they become harder to manage and refactor. Early feedback allows us to interfere and fix along the way when we still can. It enables us to learn from our customer, and from our mistakes at the right time.

All this sounds awesome but requires a lot effort. Two important enablers that help us get the invaluable feedback we need is continuous delivery and incident response. We divided the benefits into three pieces: ship often & small, recover fast, and learn from failures.

1. Ship often and small

Companies that do continuous delivery can deploy their software with ease and in small pieces. Deployments don’t require weeks or even months. Manual processes and approvals are ideally eliminated or put only before going live.

Whenever we have something new to show to our customers, we push the code to release branch, run tests, and deploy our code. This ensures faster time to market, hence early feedback, with higher quality & low risk releases.

By leveraging techniques like blue-green or canary deployments in continuous delivery, teams release in confidence. The beauty in techniques like canary deployments is that they help teams test their fixes or new features by experimenting on a small number of users in the customer base. This becomes very important if we are facing a lot of users.

2. Recover Fast

MTTR (Mean Time to Resolve) is our metric to identify how fast we recover. Although this metric can simply be calculated by subtracting alert’s creation time from resolve time, reducing this number is no easy job!

Recovering quickly requires a lot of practice and preparation before the incident. Teams should have on-call schedules with multiple rotations showing who is on-call, and when. Ideally, these teams manage their own on-call schedules to avoid dependencies that can slow down operations. Escalations are another key ingredient to be able to call for backup when the on-call responder is not available or doesn’t know how to fix the problem.

Accuracy and contextual information on alerts is a key point which we can’t deep dive in this post. To put it simply, actionable alerts are needed to triage and remediate issues faster. However, it requires a lot of effort and the right tooling. In the cloud era, automatic repair is possible and actionable alerts, which support actions that can trigger code pieces, make auto remediation easier than ever.

3. Learn from failures

Post-incident reviews or “post-mortems” are the main enablers for learning from incidents. These papers often include the timeline of events, affected services, and action items that will be taken into consideration to avoid or reduce the impact of similar incidents in the future.

A proper incident response plan must include this step. Often this step is underestimated and sometimes skipped. Yet, it is a very important step because good plans are made based on previous experiences and learnings. Complex incidents require experience to solve and post-mortems are the concrete evidence of that crucial experience.

How does the integration between XebiaLabs and OpsGenie help?

Recently, we released a deep integration with Xebialabs XL Release. Our integration combines the three items mentioned in this post and enables faster feedback in a fast-paced development environment.

OpsGenie and XebiaLabs integration allows teams to create release pipelines and incorporate incident response into each step natively.

If you’d like learn more about 5 ways that our integration with XebiaLabs help your Dev-Ops-Dev feedback loop, read more in this blog post.

Originally published at www.opsgenie.com.