BLOG

by Morgan Hague

A global CrowdStrike CSAgent outage began on July 18, 2024, at approximately 03:00 UTC. Following widespread reports of windows system crashes (and some uncertainty about the cause), initial investigations by industry personnel and subsequently CrowdStrike's engineering team identified a critical issue in the latest software update which inadvertently introduced a bug causing widespread service disruption. The bug impacts multiple versions of the ‘Falcon Sensor’ monitoring agent (also referred to as CSAgent) and triggers a looping Windows system crash, colloquially referred to as the ‘blue screen of death’.

The event was initially attributed to Microsoft in-part due to the nature of the symptoms (a Windows crash); however Microsoft and authorities have since confirmed the outage is attributable to the known update package. Authorities have also reiterated that the cause of the event is not malicious, and simply a configuration failure.

The widespread outage impacted a staggering variety of critical industries, transportation, banking, media, and healthcare alike, with the impacts to those services driving significant operational downtime and causing enaction of emergency protocols.

Trouble in Healthcare

Healthcare organizations, including the NHS, have been particularly affected by the CSAgent outage.

The following issues have been observed at large:

1. EHR Outage: Electronic Health Records (EHR) systems, including Epic, faced significant delays and outages, affecting the timely access to patient information. This disruption was reported by multiple healthcare facilities experiencing difficulties in accessing and updating patient records.
2. Critical Care Delays: Interruptions in systems that rely on real-time data (e.g., radiology reporting, transport booking) impacted patient treatment and care, leading to potential delays in critical care services. The outage highlights the vulnerability of critical care operations to cybersecurity issues.
3. Compliance Risks: Healthcare providers must adhere to strict regulatory and performance standards. The outage has raised concerns about compliance, as interrupted security monitoring and failure to appropriately respond and recover from similar outages can leave organizations liable.
4. Resource Constraints: IT departments within healthcare facilities have had to divert resources to manage the outage, which as of now requires a tedious and manual fix.

Timeline of the Outage

• 03:00 UTC, July 19, 2024: Initial reports of CSAgent failures begin to surface from users across different time zones. IT administrators start noticing that their endpoints may become stuck in a loop, causing the maligned ‘blue screen of death (BSOD)’.
• 05:30 UTC: CrowdStrike acknowledges the issue via social media and their status page, stating that they are investigating the cause of the disruption.
• 06:27 UTC: CrowdStrike issues a formal statement detailing the problem and providing instructions for IT administrators to manually restore services where necessary.
• 09:45 UTC: A confirmation that fixes have been deployed was made by CrowdStrike CEO George Kurtz, also stating that Mac and Linux devices are not impacted by the faulty update.
• Ongoing Remediation: As the resolution progresses, organizations with devices that have already been impacted will need to deploy a manual fix to remediate the issue.

Immediate Resolution

For those organizations with impacted devices, the following workaround was provided by CrowdStrike:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate the file matching “C-00000291*.sys” and delete it.
4. Boot the host normally.

For additional remediation steps relevant to cloud systems and more, check out recent guidance from CrowdStrike.

Moving Forward

The CrowdStrike CSAgent outage serves as a reminder of the inherent vulnerabilities in even the most advanced cybersecurity systems. As the digital landscape continues to evolve, so too must our approaches to risk management and system resilience. It is crucial for organizations, particularly those in sensitive sectors like healthcare, to maintain robust incident response plans and continuously evaluate their cybersecurity posture.

CrowdStrike’s prompt response and ongoing efforts to mitigate the effects of this outage are commendable, but the incident underscores the need for continuous vigilance and adaptability in cybersecurity practices, including management and update hygiene for third party applications.

In what is likely to be one of the largest cyber-adjacent incidents in the past decade, entire industries will be reeling over the weekend as IT departments work overtime to remediate the errant update. As arguably the leader in endpoint detection and response, it’s clear that nothing is certain when it comes to provider assurance and highlights the need for constant vigilance.

Meditology Services provides tailored incident response planning and exercises for many of the country’s top healthcare organizations. Contact us today for more information on how we can help ensure your teams are prepared in the event of a critical outage or event.

Furthermore, with our RITHM™ subscription-based IT risk management program, we provide core risk and compliance services with a predictable spend. This allows healthcare organizations to maintain a consistent cybersecurity cadence, continually assessing and addressing vulnerabilities like MOVEit and major incidents akin to CrowdStrike’s outage.

As always, Meditology Services remains committed to providing robust cybersecurity solutions tailored to the unique needs of healthcare organizations. With offerings that include risk assessments, cybersecurity testing, and incident response, we are here to provide support and guidance as you navigate this urgent situation.

Contact our team to see how we can help! Your cybersecurity is our priority.

Together, let's move healthcare cybersecurity forward.

About the Author 

MORGAN HAGUE | MANAGER, IT RISK MANAGEMENT

Morgan is an experienced security and emerging technologies consultant, with varied expertise across information security, organizational governance, and IT audit practices. As the leader of the Privacy, Cloud Advisory, and Strategic Risk Transformation service lines at Meditology, he has led and contributed to hundreds of consulting engagements across public and private entities. Since 2019, he has served as lead architect and product owner of an innovative risk quantification, analysis, and reporting solution utilizing MITRE ATT&CK and similar authoritative sources to establish a data-driven and dynamic mechanism to assess, report on, and manage organizational risk – supporting a variety of premier healthcare organizations, including the nation’s largest hospital system. Morgan is currently an executive board member with InfraGard Atlanta, and a contributor to OWASP’s AI Security Guide