Nine lessons from recent software supply chain incidents

Five incidents between 2021 and 2026 teach nine repeatable lessons: know what you run, treat build systems as production, distrust single points of publishing, verify artifacts against source, limit privilege, rotate fast, read advisories as data, plan for vendors failing, and rehearse. Each lesson below comes with one action.

Lesson 1: Know what you run

CISA's Log4j guidance of December 2021 stressed enumerating every instance, including third-party products, SaaS and internal software. Teams with an inventory scoped it quickly. Action: keep an SBOM per release.

Lesson 2: Treat build systems as production

Mandiant reported that attackers in the 2023 3CX case compromised the Windows and macOS build environments before shipping a trojanized desktop app. Action: isolate build hosts and review changes to build scripts.

Lesson 3: A trusted vendor can be the first victim

In 3CX, the first break was at a different company: a tampered installer for a discontinued trading app. Action: retire old installers and signing certificates.

Lesson 4: Verify the artifact, not just the repository

In xz, the malicious build line was in release tarballs for 5.6.0 and 5.6.1 and not in the upstream git source, according to the disclosure. Action: compare release artifacts to tagged source for your most critical components.

Lesson 5: Limit privilege

The xz payload reached sshd because some distributions link a service manager library that depends on lzma. A library loaded into a privileged daemon extends the blast radius. Action: rank dependencies by privilege and exposure.

Lesson 6: One publishing account is a single point of failure

In March 2026, axios 1.14.1 and 0.30.4 were published through a compromised maintainer account, per GitHub and Singapore CSA advisories. Action: use phishing-resistant authentication and short-lived tokens for publishing, and publish provenance where supported.

Lesson 7: Rotate secrets before you investigate

GitHub's advice on axios was to treat affected machines as fully compromised and rotate secrets from a clean computer. PyPI said the same on LiteLLM. Action: write a rotation runbook that lists every secret by system and owner.

Lesson 8: Tools inside the pipeline are part of the supply chain

Reporting on LiteLLM says the compromise appeared to come from a security scanner used in CI. Anything with access to publish credentials is in scope. Action: run third-party tools in jobs without publish secrets.

Lesson 9: Rehearse

None of the above helps if the first time you search for a package version is during an incident. Action: pick a past advisory and time how long it takes you to list affected products.

A constructed example

A team runs the Lesson 9 drill using the axios advisory. Searching 60 repositories for the two bad versions takes three hours by hand, because half have no lock file. The team writes down the gaps: no SBOMs, mixed lock files, no list of secrets. Two weeks later the same drill takes ten minutes.

How to check or apply this

1. Pick the one lesson where you are weakest today.
2. Do its action this week.
3. Re-run the drill next month.
4. Add the next lesson to the following sprint.

Safeguard can generate SBOMs, scan dependencies and apply policies and gates in pull requests, which covers lessons 1, 4 and 6 in part.

Why these five incidents?

They are well documented in primary or advisory sources, they span 2021 to 2026, and they show different entry points: a build environment, a library, a tarball, a maintainer account and a pipeline tool. We left out incidents we could not confirm from at least one primary source. If your own experience suggests a tenth lesson, add it to the drill.

Quick check

Sources

• CISA, Apache Log4j Vulnerability Guidance
• Mandiant (Google Cloud), 3CX software supply chain compromise
• Andres Freund, xz disclosure on oss-security
• GitHub Advisory GHSA-fw8c-xr5c-95f9
• Cyber Security Agency of Singapore, Advisory AD-2026-002
• PureAI, LiteLLM compromise report

Last reviewed: 18 October 2026.