When an LLM application leaks data, takes an unintended action or is manipulated, follow the incident steps you already use, adapted for prompts, retrieved content and tools. Contain by cutting the model's access first, preserve prompts and tool logs, record NIST's minimum incident fields, then fix root cause and review.
This runbook is defensive and generic. Adapt it to your systems and your legal obligations.
How does this fit your existing process?
NIST SP 800-61 Revision 3 (April 2025) replaced Revision 2 and is written as a profile of the Cybersecurity Framework 2.0. It says Govern, Identify and Protect help prevent some incidents and prepare for others, and that lessons learned feed back into risk management. Keep one incident process, not a separate AI one, and add AI specific steps to it.
NIST's generative AI profile (AI 600-1) asks organisations to define who does incident response for AI systems, with diverse composition suited to the incident type, and to verify that responders have the right skills. It also suggests rehearsing incident plans for third party AI technologies on a regular cadence and defining who owns each response function.
What goes in the first hour?
1. Declare and assign. Name an incident lead and a scribe. Start a timeline.
2. Contain by removing capability. An LLM application causes harm through what it can reach. Disable the affected tool or connector, switch the feature off, revoke or rotate the application's credentials and pause the retrieval source if it is suspected. Isolating systems and using out of band communication are standard steps in CISA's ransomware response checklist and apply here too.
3. Preserve evidence. Keep the conversation transcripts, system prompt version, retrieved documents, tool calls and outputs, model and version identifiers and access logs for the affected period. Snapshot before you change anything.
4. Triage impact. OWASP lists what prompt injection can lead to: disclosure of sensitive information, exposure of system prompts or infrastructure details, biased or incorrect output, unauthorised access to functions and commands run in connected systems. Ask which of these happened, and for whom.
What do you record?
NIST AI 600-1 suggests a minimum set of criteria for incident reporting: system ID, title, reporter, system or source, data reported, date of incident, description, impacts and stakeholders impacted. Use those as the header of your incident record. Add the model version and prompt version.
How do you notify?
Follow your normal rules for security incidents: privacy regulators, customers, contractual duties and insurers. If the problem is in a third party model or service, notify that provider and check the incident duties in your contract. NIST suggests seeking contract terms on notification and disclosure for serious incidents arising from third party AI data and systems, along with incident response terms in vendor agreements.
NIST also points to public resources for tracking and sharing AI incidents, such as the AI Incident Database, AVID, CVE, NVD and the OECD AI incident monitor.
How do you recover and learn?
Fix the cause, not the symptom. If the cause was excess permission, reduce it. If untrusted content reached the model with authority, separate it and require approval for risky actions, as OWASP recommends. Add the scenario to your regression tests. Hold an after action review and update the incident plan; NIST's profile calls for policies for after action reviews of AI incident response and disclosures.
What to do
- Add AI specific scenarios to the incident plan: data exposure, tool misuse, poisoned source.
- Make a kill switch for each tool and connector, and name who can use it.
- Turn on logging of prompts, retrievals and tool calls before you need it.
- Keep the NIST minimum fields as your incident template.
- Check vendor contracts for incident notification terms.
- Run a tabletop on one AI scenario each quarter and update the plan.
Quick check
Do we need a separate AI incident process?
Usually not. Extend your existing process with AI steps such as disabling tools, preserving prompt and retrieval logs and tracking model and prompt versions.
What is the first containment step?
Remove the model's access: disable the affected tool or connector, revoke or rotate its credentials and pause the suspect data source, while preserving logs first.
Sources
- NIST: SP 800-61 Rev. 3
- NIST: AI 600-1, Generative AI Profile
- OWASP: LLM01 Prompt Injection
- CISA: #StopRansomware Guide
Last reviewed: 18 October 2026.
