Systemic IT Failure Precedents: CrowdStrike, NotPetya, Colonial, Log4j

Large scale IT disruption is not hypothetical. It has happened several times, and each case has public primary sources. Looking at them is more useful than guessing about the future, because the failure patterns repeat. This post uses only facts from sources we fetched.

What happened

NotPetya, 27 June 2017. CISA's alert TA17-181A says delivery was through a backdoor in the M.E.Doc tax accounting software, after which the malware spread with stolen credentials (via WMI and PsExec) and the EternalBlue and EternalRomance exploits against SMBv1. Microsoft had released the MS17-010 patch on 14 March 2017. In February 2018 the alert was revised to add the US government attribution of NotPetya to the Russian military.

Colonial Pipeline, May 2021. CISA and the FBI issued advisory AA21-131A on 11 May 2021 about DarkSide ransomware against a US pipeline company. The attackers hit the IT network, and the company disconnected some operational technology as a precaution. The advisory said there was no indication the attackers had moved into OT.

Log4j, December 2021. The Cyber Safety Review Board's report of 11 July 2022 calls Log4j an "endemic vulnerability" likely to linger for "a decade or longer." It also highlights how thinly resourced volunteer maintainers are for widely used open source.

CrowdStrike, July 2024. A content update for the Windows sensor caused crashes. Microsoft estimated about 8.5 million Windows devices were affected, "less than one percent of all Windows machines." CrowdStrike's root cause analysis of 6 August 2024 says the sensor expected 20 input fields and the update supplied 21, causing an out of bounds memory read. The company and a third party concluded it was not exploitable by an attacker. This was a failure with no attacker.

What is affected

Anyone sharing the dependency: a tax package used by many Ukrainian firms, an IT network connected to operations, a logging library embedded in countless products, or an endpoint agent with kernel access.

How to check if you're exposed

• List vendors that run code with privileged access on your machines.
• List components that sit inside many of your products (libraries, base images).
• Identify which systems have a manual fallback and which do not.

What to do now

1. Keep an inventory of third party software and libraries, with owners.
2. Stage updates where you control the rollout. Ask vendors how they stage theirs.
3. Segment IT from operational systems, as CISA advised after Colonial.
4. Keep tested offline backups and a rebuild path.
5. Write and rehearse a manual operating mode.

How this could have been avoided

• NotPetya: applying MS17-010, disabling SMBv1 and limiting admin credentials, all in CISA's mitigations, would have limited lateral movement.
• Colonial: MFA on remote access and IT and OT segmentation are in the advisory's mitigation list.
• Log4j: a current software inventory shortens the time to find affected products.
• CrowdStrike: customer controlled staging of content updates, where a vendor offers it, limits blast radius.

Why it matters

None of these needed a rare event. They needed one trusted channel, an update, a shared library or a flat network, to fail or be abused. The lesson is to know your shared dependencies and practise running without them.

Try it yourself

Walk the incident and postmortem demo, then write a one page postmortem for the dependency you listed as most critical.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• CISA, Petya Ransomware alert TA17-181A
• CISA, DarkSide Ransomware advisory AA21-131A
• CISA CSRB, Review of the December 2021 Log4j Event
• Microsoft, Helping our customers through the CrowdStrike outage
• CrowdStrike, Channel File 291 Root Cause Analysis