Evolving Insights: Minimize Disruptions From the CrowdStrike Outage

Businesses, governments and end users across the globe on July 19, 2024, faced an unprecedented IT outage triggered by a software update from endpoint protection provider CrowdStrike. The outage, which resulted in Microsoft Windows systems crashing and being unable to properly reboot, struck in the early morning hours and quickly impacted operations worldwide for a wide range of industries, including financial services and airlines.

On July 24, CrowdStrike provided a preliminary postincident review explaining that the issue was related to a Falcon content update for Windows hosts.

• Favor an “immediate” and “midterm” approach to help with planning, workload and stress management.

• Alert and engage your incident response and crisis management teams.

• Establish a triage process to categorize assets and processes based on the disruptions and remediation complexity.

• Empower your skilled IT professionals to support affected users without granting them direct access to recovery tools or elevated privileges.

• Avoid overreactions. Don’t immediately decommission, disable or replace CrowdStrike. Defer to your existing postincident review and vendor risk management process.

• Identify “straggler” machines, systems that may have been impacted but have not yet been identified in the first wave of remediations.

• Review anomalies and alerts to minimize the risks of an undetected attack.

• Participate in a business impact analysis to ensure balanced discussions about next steps that could impact your security posture.

• Communicate about your current state, planned future-state and long-term plans to avoid similar issues in the future.

• Monitor team burnout and fatigue. Consider rotating operational staff to relieve stress and collaborate with HR.

• Verify software development and deployment best practices to reduce future risk. Ensure a seamless transition of your product from creation to consumption by clearly defining the purpose, capability and ownership of test environments. Use mature test-in-production principles and practices to:
  • Gain confidence and insights about technology performance, resilience and user experience in production
  • Deliver on time with controlled pilots using feature toggles, beta groups or a canary release
  • Mitigate risk and customer impact by allowing quick rollbacks
  • Accelerate fault detection and resolution
  • Gather user-driven data to focus on metrics and features that matter most

The primary long-term focus is to mitigate or reduce the risk of the same level of business impact or exposure caused by the CrowdStrike outage:

• Focus on resilience. Such conversations should be holistic in nature and connect to overall strategic objectives, via a top-down approach.

• Review large-scale outage operational processes, such as large unexpected volumes of support requests.

• Check and update downtime procedures for key operations.

• Revise communication plans and business continuity plans. Ensure all key employees with recovery responsibilities have the desired training and are involved in testing efforts.