Leading Incident Management Best Practices

Introduction

Being prepared for unexpected events is crucial. Incidents can significantly disrupt businesses and cause harm to stakeholders and customers alike, so preventing them, whether they are technical or caused by human error, is essential for maintaining business continuity and retaining customer trust. And when an incident does occur—such as a failed backup job, a power outage, or even a full-blown ransomware attack—handling and managing it properly is pivotal to the survival and prosperity of your organization.

A site reliability engineer (SRE) working for any corporate entity, be it a managed services provider (MSP), a small or medium-sized business, or a large enterprise, should define what constitutes an incident and proactively put forth efforts toward prevention and handling. Then, as engineers, they must prepare for the inevitable and have the right tools and information at hand to resolve whatever comes up.

This article dives into the top ten best Incident Management Practices, showcasing the lifecycle of an incident and a roadmap for successful incident resolution.

Key Incident Management best practices

Best practice	Description
Have the right people assigned to the incident	An established incident management team of top talent from various departments is essential.
Establish proper communication channels	Have a communication plan for internal and external communications and decrease noise by only including the necessary people.
Disseminate information on incident status to a predefined list of people	Use tools to detect and report incidents.
Define what is considered an incident for your organization	Know what an incident is for your organization. For example, an incident is typically something like a server outage, not high CPU usage; data loss, not delayed backups. It may also be any security breach.
Identify an incident manager	Assigning someone to coordinate and facilitate communications during an incident is vital for the task force’s effectiveness.
Build a solid knowledge base and extend it as required with each incident	Easily navigable, up-to-date knowledge bases are important.
Know your SLOs and keep an eye on your SLAs	Ensure alignment between the expectations set by SLOs and the commitments made in the SLAs.
Automate everything possible, and have runbooks where human intervention is needed	Automate time-consuming tasks to save valuable time for your incident team. As you gain experience, promptly identify and implement more automation opportunities.
Note actions taken and conclusions made as they happen	Collecting information when the incident occurs and creating documentation as the lifecycle progresses simplifies the post-mortem.
Keep actions blameless	A blameless culture reduces anxiety in teams and individuals, improves collaboration, and retains talent. In a transparent environment, people tend to be more accountable.

The lifecycle of an incident

As an SRE, it’s critical to have a complete understanding of the lifecycle of an incident, which encompasses 10 steps:

Detection: An issue is detected and logged using a tool of your choice designed for this task.
Reporting: The incident is reported and communicated to the designated personnel.
Response: Engineers work to resolve the incident.
Communication: The team communicates regular updates to stakeholders via agreed communication channels.
Resolution: The issue is resolved by configuration changes or other remediation.
Post-incident review: Action items are documented, and a root cause analysis (RCA) is conducted.
Documentation and knowledge sharing: Lessons learned are recorded and shared with the team.
Monitoring and followup: The team ensures the system’s stability and implements prevention measures.
Closure: The incident is considered closed.
Post-mortem: A written record of the incident, its impact, actions taken by the people involved, and follow-up actions are collectively documented and discussed among team members.

Top ten incident management best practices in detail

There are many different opinions on the best practices for handling incidents. Organizations may follow different procedures, and different products dictate specific approaches.

That said, there are common themes that apply widely. Let’s dive into the top ten best practices in more detail to understand them clearly.

Have the right people assigned to the incident

A decision-maker must make an effort to form an incident response task force consisting of the right set of knowledgeable, experienced, and responsible people. This group should be multidisciplinary, experienced, and responsible and should have a proven track record.

Access and privilege rights must already be in place to avoid delays and increased response times. The member roles of such a team should include an infrastructure specialist, the application owner, a subject matter expert (SME) relevant to the technologies used, and an SRE with access to monitoring, alerting, and communication tools.

Ideally, the members of the team should know each other and be able to communicate directly, precisely, and in a timely manner. There must be some overlap in their skills to augment each other, but they must have distinct areas of responsibility for resolving the issue.

Establish proper communication channels

Coordination channels must be in place to allow individuals within their respective teams to coordinate their efforts and exchange information.

During the lifecycle of an incident, everyone needs to be on the same page. Confusion can be detrimental, resulting in data losses or adding costs for the organization.

One or more predefined lists of people should be compiled, and the correct information must be disseminated to the right list, increasing incident status visibility and reducing noise during incident handling.

Use tools to detect and report incidents

It is necessary to use specialized tools that allow you to set and aggregate alerts, define thresholds, and integrate with other tools in your arsenal.

Ensure that your tools provide multiple notification methods, including SMS, push notifications, emails, or phone calls. Create dashboards and status pages to give overviews of incident status and ensure that you are aware of scheduled events that may have a maintenance window or other planned downtime.

In addition, the tools of your choice should provide an easy way for you to know what jobs are scheduled during the timeframe of your incident in order to reschedule or mitigate resources accordingly.

Define what is considered an incident for your organization

When you have an event or occurrence that impacts the reliability of an application, service, or system by affecting its availability or performance, you might have an incident. Labeling a non-incident as an incident during reduced reliability is easy, but not everything falls under the incident category. Using your monitoring tools, you will notice events that warn you of imminent incidents, downtime, and loss of connectivity.

In any organization, team leads should periodically run a capacity planning exercise to ensure that the system can handle unexpectedly high loads. SREs must pay attention to the monitoring tools and the alerts and warnings they generate while being aware of scheduled maintenance windows and thoroughly understanding what constitutes a problem as opposed to an incident.

While the exact choice will depend on the organization, the table below shows some typical differentiations.

Incident	Problem
Server outage	Unusually high CPU usage
Data loss and restoration failure	Delayed backups
Security breach	Expiration of credentials in a non-production environment

Identify an incident manager

An incident manager acts as a centralized communication, coordination, and decision-making hub. This person will prioritize tasks, approve or deny actions, and be accountable for the outcome.

The incident manager is often the one keeping detailed records for post-incident analysis. An incident manager also reviews the documentation produced during and after the incident to ensure accuracy.

Build a solid knowledge base and extend it as required with each incident

Having, updating, and maintaining an easy-to-navigate knowledge base is most important because having one that can be read and reviewed by task force members has been proven to reduce incident resolution times. It also aids in knowledge sharing, thus improving the team’s efficiency.

Remember that the knowledge base must be structured to be easy to search and navigate, no matter which application you use to create it.

Know your SLOs and keep an eye on your SLAs

Knowing and tracking your service-level objectives (SLOs) keeps you focused on what you need to deliver regarding performance, reliability, and other business-set criteria. Incidents happen, but as long as SLOs are met and service-level agreements (SLAs) are kept, it’s all business as usual.

Automate everything possible and have runbooks where human intervention is needed

When it comes to managing an incident, automation is critical. SREs and incident managers should always be working on identifying and implementing automation opportunities. These can cover a broad spectrum of services and functions, some technical and others procedural.

Automation opportunities commonly arise in these areas:

Alerting
Prioritization
Notification
Escalation
Mitigation
Scaling and allocation of resources
Integrations with security information and event management (SIEM) platforms

Note actions taken and conclusions made as they happen

Collecting information while the incident is happening and creating documentation as the lifecycle progresses make life easier when the time comes for the post-mortem.

A post-mortem is one of the most important ceremonies of an incident. Still, with the incident behind you, it’s easy to forget details of what happened during the incident resolution. It’s always beneficial to document your actions and note conclusions, potential automation opportunities, and optimizations in current runbooks while working to make the subsequent incident resolution faster and painless.

Keep actions blameless

A blameless culture reduces anxiety in teams and individuals, improves collaboration, and retains talent. In a transparent environment, people tend to be more accountable.

Trial and error is encouraged, leading to increased innovation. In an environment where people are not worrying about being blamed, trust is built along with positive relationships and work enjoyment, contributing to successful and quick incident resolution.

Conclusion

Following the correct practices and utilizing the right tools is the recipe for successful incident resolution. The above-mentioned techniques, procedures, and methodologies will reduce incidents and streamline incident management as time passes.

Having a centralized platform where task force management, incident response, coordination, alerting, and monitoring work together harmoniously is bound to reduce toil and keep error budgets in check. Effective communication channels and native integrations with the most popular tools in the market will guarantee a positive, enhanced experience during an incident.

SolarWinds Incident Response integrates with almost 100 monitoring tools, aggregates information for reliable alerting, and provides notifications for scheduled maintenance windows. In addition, escalation policies can be set using a combination of rules to alert the right people at the right time via email, push notifications, SMS, or phone calls. In addition, you can have a status page where all interested parties can get an overview of where the task force currently is within the incident lifecycle.

Ready to Transform Your Organization’s Incident Response?

Learn More