Observability in AI models means collecting enough data about each request, model version, retrieval step, tool call and output that you can explain what the system did and why. For security teams, it is the difference between guessing after an incident and reading the trace.
Why is it harder than ordinary monitoring?
Traditional software is deterministic. The same input gives the same output, so a crash or a wrong value can be reproduced. A language model sampling from probabilities can answer the same question differently twice. Add retrieval from documents and calls to tools, and a single user request fans out into many steps. If you log only the final answer, you cannot tell whether a bad result came from the prompt, the retrieved text, the model, or a tool.
NIST's AI Risk Management Framework is a voluntary framework for incorporating trustworthiness into the design, development, use and evaluation of AI systems. Its companion generative AI profile (NIST AI 600-1) addresses risks specific to generative systems. Both assume you can measure and track what a system does. Observability is how you get that data.
What should you record?
Think in layers.
Request level. A trace identifier, time, calling user or service identity, application and model version, and any configuration that changes behavior, such as temperature or system prompt version.
Retrieval level. Which documents or records were fetched, from which source and with what identifier. Without this, you cannot investigate indirect prompt injection or a data leak through retrieved content.
Tool level. Every tool call: name, arguments, the identity it ran under, the result status and any approval given by a human. This is the audit trail for an agent that acts.
Output level. The response, plus the results of validation and filtering steps, and whether a guardrail blocked or altered anything.
Operational level. Latency, token counts, cost, error rates and rate limit hits. A sudden jump in token use can signal abuse or a runaway loop.
Quality signals. User feedback, evaluation scores on sampled traffic, and drift in these scores over time.
OpenTelemetry maintains semantic conventions for generative AI so traces from different tools use the same field names. At the time of writing, the project says these conventions have moved to a dedicated repository, so check there for the current definitions.
What are the privacy catches?
Prompts and outputs often contain personal or confidential data. Logging everything verbatim can turn your observability store into your largest data risk. Decide what to redact, who can read traces, and how long you keep them. Apply the same minimisation and retention thinking you would to any personal data.
A constructed example
A support assistant starts quoting internal pricing notes to customers. With only final answers logged, the team sees the leak but not the cause. With traces, they find that retrieval began returning a shared folder after a permissions change, so the model had those notes in context. The fix is in access control and indexing, not in the prompt. The tool-call log also shows no actions were taken, which limits the incident scope.
What should trigger an alert?
Start with a short list. A spike in tokens per request, which may mean a loop or abuse. A tool call to a function that is rarely used. A blocked output rate that doubles overnight. Retrieval from a source the application has never used. A new model or prompt version in production without a change record. Each of these has an ordinary explanation most days, and a security explanation on the day it matters. Write down who looks at the alert and what they check first.
How to check or apply this
- Give every request a trace identifier and carry it through retrieval and tool calls.
- Log model and prompt versions so you can tell which configuration produced an output.
- Record tool calls with identity, arguments and result, and keep them tamper resistant.
- Redact or hash sensitive fields before storage, and restrict who can read raw traces.
- Set alerts for unusual token use, blocked outputs and failed tool calls.
- Sample traffic for evaluation and track results over time.
- Define retention for traces and delete on schedule.
Quick check
Is logging prompts and responses enough?
Usually not. You also need retrieval sources, tool calls, versions and validation results to explain why an output happened.
Does AI observability create privacy risk?
It can, because traces may hold personal or confidential data. Redact, restrict access and set retention.
Sources
Last reviewed: 18 October 2026.
