The CrowdStrike Incident - Resilience Matters

Now that most MM-ISAC member organizations impacted by CrowdStrike's issue on Friday are through the worst of the recovery, I want to share some thoughts on lessons we can learn from this incident to help make us all more resilient this week than last.

I will avoid semantic discussions about whether this was or was not a security incident, whether EDR solutions need to run in the Kernel, or whether other EDR/XDR solutions are more or less susceptible to this type of failure. Why? As fun as those circular arguments are, they don't solve anything because they don't matter. Your business users and their leaders don't care if the reason they can't use their systems is a blue screen of death from a bad driver, a mistake made by an IT or Security team member, a backhoe taking out the fibre line, or Thunderstruck playing on a loop. At the same time, data is encrypted by the next Ransomware-as-a-Service offering. They care that business is stopped, they can't deliver value to their customers, and the organization can't execute its mission. From that perspective, this is an incident. Full stop.

The organizations that handled this incident best were those who pulled out their incident response plan, turned to the ransomware section (the closest playbook to the situation they likely had), and executed. Those organizations had attributes in common. They followed ICS. That is why the MM-ISAC incident response plan template is based on the principles of ICS4ICS—it provides the necessary structure to execute a large-scale response, maintain control of the incident, enable effective communication with stakeholders, and protect your responders from burnout.

If this incident impacted you, once your environment is stable and your teams have had time to rest and recover, please conduct thorough lessons learned. The lessons-learned activity should be led by someone not involved in the incident. For MM-ISAC members, we can help with that activity; for the rest of you, please reach out to your ISAC, an independent consultant, or an internal facilitator who was not involved... do the activity soon while memories are fresh. The goal of the process is not to assign blame; it must be blame-free. However, you must walk through your response, what went well, where there are opportunities to improve resilience and bake those lessons back into your incident response plan. Please also share those learnings with your ISAC and other peers, and speak at events. We all need to learn and get better.

This was a warning for those who watched this from the sidelines. You got lucky. Until we find a way to fix the foundational software quality issue across IT/OT/Cloud... You will get your turn for something similar. Tabletop, your EDR is failing, and our core network is going down... Identify your "Crowdstrike" and test your Incident Response Plan. Don't wait; schedule that today and do the same lessons-learned activity.