Which AI observability metrics matter for security?

For security, the AI metrics that matter beyond latency and cost are tool calls per session, guardrail blocks, permission denials, sensitive data detections, retrieval source mix, and model and prompt version per response. Together they show what an AI feature did, what stopped it, and when its behavior changed.

What does observability cover for an AI feature?

Ordinary application monitoring watches requests, errors and latency. An AI feature adds new things that can go wrong without any error code: a model gives a confident wrong answer, an agent calls a tool it should not, a retrieval step pulls in a poisoned document. None of these shows up as a failed request.

OpenTelemetry, the open standard for telemetry, publishes semantic conventions for generative AI so that traces and metrics are named consistently across tools. Check the current status of that specification before you build on it, as it is evolving. The NIST AI Risk Management Framework, in its measure function, asks organizations to track and assess AI risks with appropriate metrics, which is the formal reason to do this work.

Which metrics tell you something about security?

Group them by the question they answer.

What did it do? Tool calls per session, by tool name; number of distinct tools used; actions that change state, such as writes and deletes, versus reads; external hosts contacted.

What stopped it? Guardrail or filter blocks, by category; permission denials from downstream systems; human approvals requested and denied.

What went in and out? Counts of sensitive data detections in prompts and in responses; size of prompts and outputs, since sudden growth can signal stuffing or exfiltration; share of retrieved content by source, which shows when an unusual document starts to dominate.

What changed? Model version, prompt template version and tool configuration recorded on every response. Without these, you cannot explain why behavior shifted last Tuesday.

How is it going overall? Error and timeout rates, latency percentiles, token usage and cost per session. Cost spikes are also a security signal, because abuse often shows up as unexpected usage.

What does this look like in an incident?

A constructed example. A support assistant can look up orders and issue refunds up to a limit. One afternoon the dashboard shows refund tool calls per session tripling, with most coming from a few sessions. The permission denial rate for refunds also rises, since the downstream system enforces the limit. A trace shows each of those sessions began with a pasted message containing hidden instructions. Because the team logged the prompt template version and the retrieval source, they could see the injection came through a customer uploaded document and block that path.

Without those metrics, the team would only have noticed an increase in refund costs a week later.

What should you avoid?

Do not log everything blindly. Prompts and responses can contain personal data and secrets. Decide what to keep in full, what to hash or redact, and how long to retain it, with legal input. Do not rely on a single quality score from a model grading another model as your only signal. It can drift, and it can be fooled. Finally, avoid alerts on averages. Attacks show up in outliers, so alert on per-session and per-user extremes.

How to check or apply this

1. List every tool and data source your AI feature can reach.
2. Emit a structured event for each tool call with session, user, tool, arguments summary and result status.
3. Record model version and prompt version on every response.
4. Count guardrail blocks and downstream permission denials by category.
5. Set alerts for per-session outliers in tool calls, token use and refund or write actions.
6. Review a sample of full traces weekly and write down what you learn.
7. Set retention and redaction rules for prompts and outputs.

Quick check

Sources

1. OpenTelemetry, Semantic conventions for generative AI
2. NIST AI Risk Management Framework
3. OWASP Top 10 for Large Language Model Applications

Last reviewed: 18 October 2026.