To handle a security incident in an LLM feature, contain it first by switching off the affected feature, tool or data source. Then preserve the prompts, retrieved documents and tool calls that show what happened, fix the cause and re-enable gradually. AI incident response follows the same process as any security incident. What changes is where the evidence lives and which switches you need to have built in advance.
What counts as an AI security incident
An AI security incident is any event where an LLM feature exposes data, takes an action or produces output it shouldn't, whether an attacker caused it or not. Typical examples:
- The assistant shows one customer another customer's data or documents.
- The system prompt, and any secret placed in it, is extracted and posted online.
- Instructions hidden in an email or web page make an agent send messages or change records.
- A response embeds an image URL that carries conversation data to an outside server.
- A leaked model provider API key, or scripted abuse of your chat endpoint, drives usage far above normal.
- A support bot makes commitments your policy doesn't allow, and customers act on them.
A wrong answer on its own is a quality problem. It becomes an incident when it causes harm or reaches many users before anyone notices.
How AI incident response differs from a normal incident
NIST SP 800-61 Revision 3, published in April 2025, places incident response across all six functions of the NIST Cybersecurity Framework 2.0, from preparation through detection, containment and recovery. That structure fits AI features. LLM features add two complications.
The trigger is often text. An injected instruction in a document, ticket or web page keeps firing until someone removes it from wherever the model reads it. Patching code doesn't remove a poisoned document from a search index.
The evidence is spread out, and the system isn't deterministic. Rerunning a prompt may not reproduce the failure, so you need the logged trace: user input, prompt version, retrieved document IDs, tool calls, model identifier and raw output. Without it, the investigation stops at guesswork.
Prepare before anything goes wrong
Most of the work happens before the incident. NIST AI RMF Playbook item MANAGE 2.4 asks for mechanisms and assigned responsibilities to supersede, disengage or deactivate AI systems that perform inconsistently with intended use. In product terms: kill switches someone can use at 2 a.m. without a deploy.
Build these as runtime flags:
- Turn off the whole AI feature and show a fallback message.
- Disable a single tool, or move it to "requires human approval".
- Exclude one data source or document from retrieval.
- Turn off markdown image and link rendering.
- Lower per-user and global rate limits and token budgets.
- Pin or roll back the model version and the prompt version.
Log each request with a trace ID, user and tenant, the fields listed above, and tool results. That log holds personal data, so restrict access and set a retention period you can defend. Our guide to securing AI agents that can use tools covers logging for agent actions.
Finally, name an incident owner per AI feature and write a one-page playbook per incident type. NIST SP 800-61r3 notes that playbooks give people actionable steps for specific scenarios.
How to detect an incident in an LLM feature
Signals worth wiring up before launch:
- A "report this response" control in the UI, routed to a monitored queue rather than a general inbox.
- Token usage per user, per tenant and per API key, with alerts on sudden changes.
- Tool call volume per user, and calls that fail authorization checks.
- Content Security Policy (CSP) violation reports for image or connection requests to hosts outside your allowlist.
- A canary string: a unique random value placed in the system prompt, with an alert whenever it appears in an output.
- A check after retrieval that flags any document whose tenant or permissions don't match the requester. Our post on RAG access control explains where that check belongs.
AI incident response, step by step
- Contain. Use the narrowest switch that stops the harm. If you can't yet tell which part failed, turn off the whole feature. Rotate any credential that may be exposed, including the model provider key.
- Preserve evidence. Export traces for affected sessions before retention jobs delete them. Copy any poisoned document, with its hash and source, before removing it. Record the prompt version, model setting and index state.
- Scope. Search the logs for the same pattern across all users and the full retention window. Identify which data, records or accounts were affected, and since when.
- Eradicate. Remove the injected content from every store the model reads, fix the permission or rendering bug, and revert anything the agent changed.
- Recover. Re-enable for internal users, then a small share of traffic, while watching the signals that caught the incident.
- Learn. Add the reproduction to your test suite and update the playbook. MANAGE 4.3 asks that incidents are communicated to relevant AI actors and that tracking, response and recovery are documented.
Containment options for common AI incidents
| Incident | First containment step | Evidence to keep |
|---|---|---|
| Another user's data in responses | Disable retrieval or the feature | Retrieved document IDs, requester identity, index permissions |
| System prompt or secret disclosed | Rotate the secret and move it out of the prompt | Prompt versions, outputs containing it |
| Agent took unintended actions | Disable the tool or require approval | Tool calls, arguments, the content that triggered them |
| Data sent out through image URLs | Turn off image rendering | Outputs, CSP reports, outbound request logs |
| Usage spike from abuse or a leaked key | Lower limits, block accounts, rotate the key | Usage by key, user and IP address |
| Harmful commitments to customers | Turn off the feature or restrict its topics | Conversations and affected customers |
Notification and reporting duties after an AI incident
This section is general information, not legal advice. If personal data was exposed, deadlines may start when you become aware of the breach. The EDPB's guide for small businesses states that under the GDPR a controller must notify the data protection authority of a personal data breach within 72 hours, unless it's unlikely to result in a risk to individuals. Individuals must be told without undue delay when the risk is high, and a processor must notify the controller without undue delay.
For B2B AI products, your customers are often the controllers, and your contract says how fast you must tell them. In the US, breach notification rules vary by state and sector. Bring in counsel early and keep the scoping notes from step 3, because the notification decision depends on them. Also find your model provider's status page and security contact now, since some incidents start on their side.
Common AI incident response mistakes
- Treating a data leak as a model quality bug and tuning the prompt while the feature stays live.
- Rebuilding the search index before exporting the poisoned or wrongly permissioned documents.
- Logging only final outputs, so nobody can see which document or tool call caused the problem.
- Kill switches that need a code deploy, or that only one engineer knows how to use.
- Fixing the reported case without searching the logs for the same pattern from other users.
- Leaving the model provider API key unrotated after it appeared in client-side code.
- Re-enabling for all users at once, with no regression test for the failure.
If you want an outside view, our AI security and reliability services test AI features against real attack patterns, and an ongoing retainer can add monitoring, alerting and incident response. The AI app security checklist covers the wider basics.
Frequently asked questions
Who should own AI incident response?
The team that runs the feature owns containment and the fix, and your security lead coordinates. Name the owner in advance. Whoever is on call must be allowed to use the kill switch without asking for approval.
Should we turn off the AI feature completely during an incident?
Only if you can't contain it more narrowly. Disabling one tool or data source often stops the harm while the rest keeps working. When you don't yet know the cause, turn it off fully and narrow it down later.
How long should we keep LLM logs for investigations?
Long enough to cover how long it usually takes you to discover an incident, and no longer than your privacy commitments allow. Document the period and the reason. Logs deleted before anyone notices can't support scoping or notification decisions.
How do we practice for an AI incident?
Run a tabletop exercise on one scenario from the table, using the real kill switches in staging. Our guide on how to test an LLM feature before launch has attack scenarios you can reuse. Fix every switch or log field that turned out to be missing.