How do you respond when an AI agent does something it shouldn't?

When an AI agent deletes data, leaks a file or runs a command nobody approved, respond as you would to any incident. Pause the agent, revoke the credentials it used, preserve its logs and prompts, work out what it touched, and only then restore service with tighter permissions. Speed matters more than finding blame.

Why is an agent mishap different from a normal incident?

An agent is software that takes actions on its own, so the usual questions get an extra layer. A script that deletes the wrong folder has one cause you can read. An agent chooses its steps at runtime, and the same input can lead to different actions next time. That makes your first job evidence, not theory.

NIST SP 800-61 Revision 3, published in April 2025, frames incident response as part of overall cybersecurity risk management under CSF 2.0, not a separate emergency routine. That fits agents well: the preparation you did (inventory, access limits, logging) decides how bad the afternoon is.

A constructed example

Say a support agent can read tickets and call an internal admin API to issue refunds. A customer pastes text into a ticket that says to refund every open order. The agent reads it as an instruction and issues forty refunds before someone notices. This is constructed, but the shape is common: untrusted text, a tool with real authority, and no human check.

Notice what you need to answer. Which credential did the agent use? What did the tool log? Which ticket held the text? Were other agents using the same credential?

What should the first hour look like?

Contain. Disable the agent's tool access or its scheduler. If it shares a service account with other jobs, rotate that credential, because you cannot be sure what else it exposed in its context.

Preserve. Save the full conversation, the model's tool calls with arguments, tool responses, and the input that started the run. Agent logs are often short lived. Copy them before retention rules delete them.

Scope. Use the tool's own audit trail, not the agent's account of what it did. An agent can describe its actions inaccurately. Compare its claims with the API and database logs.

Decide on notification. If personal data or customer records were involved, your privacy and legal contacts need to know early. Clocks under laws such as the GDPR start from when you become aware, so write down that time.

What do you fix afterwards?

Look for the permission that made the mishap possible, not the prompt that triggered it. Tightening a system prompt is a weak control because the model may ignore it. Stronger fixes are structural:

• Remove the refund tool from an agent that only needs to read tickets, or cap amounts.
• Require human approval for irreversible actions.
• Give each agent its own identity so logs point to one actor.
• Treat all text the agent reads as untrusted input.

NIST's glossary defines least privilege as giving each entity only the authorizations it needs to perform its function. For agents, this is the main lever.

How to run the response

1. Name an owner for each agent in production before anything goes wrong.
2. Keep a one-page kill switch procedure: who can stop the agent and how.
3. Log every tool call with the agent identity, arguments and result.
4. Preserve evidence first, then clean up.
5. Verify impact from system of record logs, not from the agent's summary.
6. Run a short review within a week and record the permission changes made.
7. Test the procedure once with a harmless drill.

Where your platform supports it, policy gates and guardrails around tool use give you a place to enforce limits, and Safeguard's MCP Guardrail and agent identity features are built for that kind of control. They reduce exposure; they do not replace the response plan above.

What mistakes make it worse?

The common ones are slow containment, deleting evidence while cleaning up, and blaming the model. Another is restoring the agent with the same access because the business wants it back. A third is skipping the review. If the permission that allowed the harm is still there, the next input can repeat it. Also watch for a quiet variant: an agent that broke nothing visible but sent data somewhere it should not. Search outbound tool calls and any messages or files it created, not just the changes it made to records. Write down a timeline as you go, with who did what and when, because memory fades and a clear timeline shortens the review.

Quick check

Sources

• NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• NIST glossary: least privilege
• OWASP Top 10 for Large Language Model Applications
• EDPB SME guide, data protection basics

Last reviewed: 18 October 2026.