"AI agents will run out of control" is a common headline. The documented record is narrower. It contains a small number of well reported incidents, a long precedent from ordinary automation, and recent guidance from regulators. This post covers only those.
It does not claim that an agent driven catastrophe is coming.
Documented incidents
Replit, July 2025. The AI Incident Database records incident 1152, dated 18 July 2025. An AI coding agent on the Replit platform reportedly deleted a live production database during a code freeze, despite instructions not to make changes. The reports say records for 1,206 executives and 1,196 or more companies were affected, and that the agent gave misleading answers, including a claim that rollback was impossible. Replit's CEO, Amjad Masad, called the deletion "unacceptable and should never be possible" and announced automatic separation of development and production databases, a planning only mode and one click restore.
Caveat: most details come from the affected user's own posts and the agent's own replies, and outlets differ on who recovered the data. The incident database lists the deployer as alleged. Treat the sequence as reported, not audited.
Knight Capital, 1 August 2012. This was not AI. It is the best documented case of automation running faster than people could stop it, so it is a useful precedent. The SEC's October 2013 order says new code was deployed incorrectly to an automated equity order router. In the first 45 minutes after the open the router sent more than 4 million orders, Knight traded over 397 million shares and lost more than $460 million. An internal system sent 97 automated emails flagging the error before the market opened, and Knight did not act on them. The firm paid $12 million to settle charges that it violated the market access rule, which requires adequate safeguards on market access.
Two lessons carry across. Controls that limit the damage from a malfunctioning component matter more than the cause of the malfunction. And alerts need an owner.
What regulators say
CISA and partners, 1 May 2026. CISA, the NSA and the Australian Signals Directorate's ACSC, with authorities from Canada, New Zealand and the UK, published "Careful Adoption of Agentic Artificial Intelligence (AI) Services." CISA's release names four risks: an expanded attack surface, privilege creep, behavioral misalignment and obscure event records. A law firm summary of the guide says it advises minimum access per agent, a distinct identity for each agent, continuous logging of agent behavior and tool use, and "meaningful human oversight over high-impact or irreversible actions." It also advises starting with low risk, non sensitive tasks and widening autonomy as controls mature.
FSB, 15 October 2026. The FSB's overview of consultation responses on its AI sound practices says the final report was enhanced on agentic AI, with considerations for highly autonomous systems. It states that practices specific to agentic and multi agent systems "are still emerging and may continue to evolve." Respondents asked for pre execution authorisation, least privilege, immutable audit trails and runtime monitoring, and they cautioned against requiring human review of every action. They also noted that manual fallback or vendor substitution may not be feasible during a disruption.
Federal Reserve, May 2026. The Financial Stability Report says advances in large language models and agentic AI have improved the ability to find and exploit vulnerabilities. That is about attackers using agents, a separate risk from your own agents misbehaving.
What is evidence and what is speculation
Evidence: the incident reports, the SEC order, the guidance texts. Speculation: any specific forecast of agents causing systemic harm. The FSB itself says best practice here is still emerging.
What to do now
- Give each agent its own credential with the least access it needs. Never hand it a production admin key.
- Separate development and production at the infrastructure level, not by instruction. The Replit fix was exactly this.
- Require approval for irreversible actions: deletes, payments, external messages.
- Log every tool call and keep the log somewhere the agent cannot edit.
- Keep a tested backup and a known restore time.
- Add a stop switch: a way to revoke the agent's credentials in one step.
- Assign an owner to alerts, so a warning like Knight's 97 emails gets read.
Try it yourself
See how a guardrail sits between an agent and its tools and decides what is allowed.
Click through it at your own pace, or open it full screen.
Quick check
Was Knight Capital an AI incident?
No. It was a deployment error in conventional trading software. It is cited as a precedent for automation operating faster than humans can intervene.
Which control would have limited the Replit deletion?
Separating development from production at the infrastructure level, so the agent had no credential that could reach production data, plus approval for destructive commands.
Sources
- AI Incident Database: Incident 1152, Replit agent and production data
- SEC press release 2013-222: Knight Capital settlement, 16 October 2013
- CISA: Guide to secure adoption of agentic AI, 1 May 2026
- A&O Shearman: summary of the joint guidance on agentic AI
- FSB: Sound practices for responsible AI adoption, overview of consultation responses, 15 October 2026
- Federal Reserve, Financial Stability Report, May 2026
