What to Do When "IT" Hits the Fan

Author: Matt Wilson

TL;DR – When a cyber incident strikes… PANIC, wait that's not right… Actually, you need a plan, a calm demeanor, and a capable team to survive. Let’s walk through the basics of Incident Response and why every organization, no matter the size or budget, should have at least a 101-level understanding. 

Why Incident Response Matters 

Information Security events happen, truly every day in some organizations. Regardless of how you've prepared, it's inevitable. Whether it’s a single employee clicking a rogue link (“I swear it looked legit!”) or an attacker finding the one vulnerability in your Internet footprint, how you respond in those critical first hours can make the difference between a minor hiccup and a major catastrophe. That’s where Incident Response (IR) comes in. 

Think of IR as your cookbook of effective plays to run down a score with a minute left to play. You'd rather not be losing in the first place, but you're confident because you've prepared for this exact moment. 

Key Phases of Incident Response

While accepted frameworks and standards (e.g. NIST SP 800-61, ISO 27035) may define or organize concepts a bit differently, IR processes boil down to a few core phases:

1. Preparation

What It Is: Drafting and distributing your IR plan, establishing and communicating roles/responsibilities, training your people, implementing strong InfoSec processes, and deploying the right tools before a breach happens.

Why It Matters: The best time to think through how you're going to react to a major event is when everything is running smoothly and cooler heads can prevail. Otherwise, you’re just making it up as you go—and that rarely ends well. Remember, we're not planning for EVERY potential scenario, just a collection of the most common ones, supported by some guidelines on how to handle the special unicorn situation no one thought of.

2. Detection & Analysis

What It Is: Figuring out something’s amiss and what options your organization has. This might come via an alert from a fancy SIEM tool , an employee’s gut feeling, or a dreaded call from a partner saying, “Hey, your data is on the dark web…”. Here, we weigh what we know versus what we don't, and start decisioning the next steps… Does the escalation occur to both internal and external stakeholders ? If so, how? What message do we share, if at all? Has legal approved said message?

Why It Matters: Early detection and coordinated decisions can significantly reduce the damage. Spotting the flames when they’re a spark is much easier than waiting until you’re dealing with a five-alarm blaze.

3. Containment

What It Is: Limiting the attacker’s mobility and preventing further damage. Attackers often want to "land and expand." Think of this phase as isolating infected systems, blocking malicious IP addresses, or resetting compromised credentials. This may also be where you make contact with partners such as Cyber Insurance, IR Retainer firms (not me reminding you SEVN-X offers these), Legal Counsel, Law Enforcement, and/or Regulators. All relationships you already established back in the Preparation Phase.

Why It Matters: If you find water leaking in your house, you stop the water flow before you start mopping the puddle.

4. Eradication

What It Is: Removing the attacker’s access and any malicious artifacts (malware, backdoors, etc.) from your environment.

Why It Matters: Nobody wants to play an endless game of cyber whack-a-mole. Wipe out the threat completely, or risk having the attacker pop back up like Kimmy Gibbler (IYKYK).

5. Recovery

What It Is: Restoring systems, verifying that everything is back to normal, and ensuring an effective control-set is in place to prevent a repeat performance.

Why It Matters: Getting back to business is the ultimate goal, but don't rush—nobody wants to reintroduce a compromised system into the environment.

6. Lessons Learned (the MOST overlooked Phase)

What It Is: A comprehensive post-incident review. Document what happened, what went well, what went poorly, and how to improve next time. There are likely Lessons Learned at each phase, so spend time breaking the event down to reduce the risk of recurrence and impact going forward.

Why It Matters: Incidents happen. If you don’t learn from the mistakes (and the successes), you’re doomed to repeat them. Plus, your insurance carrier, auditors, or regulators may want proof you’re adapting your security posture. 

Roles & Responsibilities

You don’t want to be figuring out who’s in charge mid-event, that's when poor decisions become devastating decisions. Even a small business should assign clear IR roles:

  • Incident Commander / IR Lead: The quarterback of your response. Orchestrates the overall response, coordinates communication, and ensures tasks are assigned and completed.
  • Technical Team: The sword wielded by the incident responder used to investigate the breach, contain the threat, and execute eradication plans. This team often includes system admins, network engineers, and security analysts.  
  • Executive Team: Supports the entire response effort through the timely review of information and analysis provided by support personnel, then decisions at inflection points to keep the response activities moving forward.  
  • Communications Lead: Handles all internal and external communications, including updates to executive management, employees, customers, regulators, and potentially the media.
  • Legal & Compliance: Assesses the regulatory implications, handles breach notification requirements, and coordinates with law enforcement, if needed.
  • Human Resources (if applicable): Steps in when insider threats or employee-related issues are part of the incident. 

When roles and responsibilities are defined upfront, the chaos of an active incident becomes more manageable.

Common Pitfalls to Avoid

  1. No Plan = Plan to Fail: Without a formalized Plan, you're trying to build the plane in-flight. Possible, but not recommended.
  2. Slow or Poor Detection: How will you know you have an event? Many breaches go unnoticed for weeks or months. The faster you realize something’s off, the better your odds of containing it.
  3. Unclear Communication: When confusion reigns, mistakes get made. Who informs the CEO? Who calls the CIO at 3AM? Define communication lines and escalations to smooth out the inevitable bumpy ride.
  4. Incomplete Eradication: When is the event… over? Know what assurances you need to make sure you've fully addressed the threat.
  5. Skipping the Lessons Learned: After an incident, the adrenaline fades, and everyone wants to forget it happened. Resist that urge. Document everything, share learnings, and update your plan.
  6. Slowing the Momentum: A major event grabs the attention of leadership. Capitalize on the opportunity to make impactful changes that stick, long-term.

Practical Tips & Quick Wins

  • Workshop your IR: Gather your team once a quarter to role-play a hypothetical breach scenario. Consider a larger-scale exercise annually where you pressure-test the lessons learned across multiple business units (not just IT and InfoSec).
  • Update Your Contact List: Have an up-to-date list of whom to call if something goes down. Include after-hours phone numbers, incident response vendors, and relevant law enforcement, business partner, customer, and regulator contacts.
  • Mind Your Offsite & Cloud Assets: Today’s networks are often hybrid environments. Make sure your IR plan covers laptops, home networks, SaaS services, and any third-party cloud providers.
  • Document Everything: In the heat of a breach, it’s easy to lose track of what was done and when. Good documentation helps you retrace your steps, prove compliance, and learn from mistakes.
  • Stay Compliant, Stay Secure: Recognize the regulatory environment your business operates in (think HIPAA, PCI-DSS, GDPR). Incorporate those requirements into your IR plan. It’s not just about avoiding fines; it’s about building trust with customers and partners. 

Looking Ahead

Incident Response isn’t just a technical exercise; it’s a learned business capability. From the boardroom to the mailroom, everyone plays a role in ensuring that when a breach (inevitably) happens, your organization navigates the response confidently.

Remember, IR is never a set-it-and-forget-it strategy. As attackers remain persistent, your IR plan must mature to address their activities and changes in your business. Keep it updated, keep testing it, and keep your team fresh on their roles.

Prepared, not perfect. The organizations that weather breaches best are the ones that take Incident Response seriously, turning a potential meltdown into a contained event with minimal disruption. And if you’re thinking, “I’ll get around to that plan next quarter,” here’s your sign: start now, before you’re forced to start in the middle of a crisis.

Have questions? Connect with SEVN-X to see how we help organizations stand tall before, during, and after a cybersecurity event. 

Submit a comment

You may also like

NIST Cybersecurity Framework 2.0: Respond
NIST Cybersecurity Framework 2.0: Respond
21 January, 2025

Authors: Mark Keppler | Steve Foret Cybersecurity Frameworks Series, part 8 Today, we will take a deep dive into the RES...

NIST Cybersecurity Framework 2.0: Recover
NIST Cybersecurity Framework 2.0: Recover
21 January, 2025

Authors: Mark Keppler | Steve Foret Cybersecurity Frameworks Series, part 9 Today, we will take a deep dive into the REC...

NIST Cybersecurity Framework 2.0: What’s New?
NIST Cybersecurity Framework 2.0: What’s New?
21 January, 2025

Authors: Steve Foret | Mark Keppler Cybersecurity Frameworks Series, part 3 With the NIST releasing CSF 2.0, the first u...