Public AI incidents repeat a small set of patterns: unverified output, too much access, trusted instructions that were not trusted, sensitive data pasted out, poor source quality, hostile public input, and a weak build pipeline. Each has a plain control. This post maps sourced cases to them.
1. Unverified output reached someone who relied on it
- Lawyers filed ChatGPT invented cases and were sanctioned in June 2023 (LawNext).
- A tribunal held Air Canada liable for wrong chatbot advice in 2024 (Mondaq).
- NYC's MyCity chatbot gave answers The Markup said conflicted with the law.
Control: a named human verifies, or the bot returns approved text only.
2. The agent had more access than the task needed
- In the reported Replit and SaaStr case, an agent deleted a production database (The Register, from the user's account).
Control: separate credentials, read only defaults, protected branches. See how to set a real code freeze for an AI coding agent.
3. Untrusted text was treated as instructions
- A message in a public channel steered Slack AI toward leaking a key, per PromptArmor as reported by The Register.
Control: keep outsider written content away from secrets, and check output links. See Slack AI prompt injection and search scope.
4. Sensitive data was pasted into an outside tool
- Samsung engineers reportedly uploaded source code to ChatGPT (CSO Online).
Control: approved tool, plain data rules, secrets scanning.
5. Weak sources produced confident answers
- Google said data voids, satire and sarcastic forum posts caused odd AI Overviews answers.
Control: source ranking and a refusal path. See data voids and source quality.
6. Hostile public input
- Microsoft said a coordinated group exploited Tay within 24 hours of launch.
Control: red team first, staged rollout, kill switch.
7. A weak build or install path
- An overly broad GitHub token in a build configuration let an attacker reach the Amazon Q extension release, per AWS.
- A USENIX study found models inventing package names across 576,000 samples.
Control: scoped CI tokens and verified installs. See overbroad CI tokens.
What the patterns share
In nearly every case the model's quality was not the only problem. The surrounding system lacked a check: no verifier, no permission limit, no input separation, no recovery owner. That is good news, because system controls are things a team can change.
Use it as a review aid
For any AI feature, ask seven questions: Who verifies the output? What can it touch? What untrusted text can reach it? What data can leave? What are its sources? What will hostile users try? What does its build and install path trust?
Safeguard's policies, gates, guardrails, asset discovery and SBOM and dependency scanning help with several of these, mainly the access and supply chain ones. The free scan at gold.safeguard.sh/scan is a place to start.
Quick check
Which pattern is cheapest to fix first?
Access scope. Removing write access and separating credentials takes little work and cuts the blast radius of every other pattern.
Are these cases proof of how often AI fails?
No. They are reported examples, not a rate.
Sources
- LawNext: Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT, June 2023
- Mondaq: Airline ordered to compensate a BC man because its chatbot provided inaccurate information
- The Markup: NYC's AI chatbot tells businesses to break the law, 29 March 2024
- The Register: Replit and SaaStr incident report, 21 July 2025
- The Register: Slack AI prompt injection report, 21 August 2024
- CSO Online: Samsung bans staff AI use over data leak concerns
- Google: AI Overviews, about last week, 30 May 2024
- Microsoft: Learning from Tay's introduction, 25 March 2016
- AWS security bulletin AWS-2025-015
- USENIX Security 2025: We Have a Package for You, abstract page
Last reviewed: 19 October 2026.
