Good AIDR metrics measure coverage, response and reversibility: how many AI systems are monitored, how fast a bad behaviour is contained, and how often detections are right. Misleading ones count activity, such as blocked prompts or alerts raised, which rise when attackers try or when rules are noisy.
Why do AI security metrics go wrong so easily?
Because the field is new and the easy numbers are volume numbers. A dashboard that says "12,000 threats blocked" feels like proof. It may only show that you have a noisy rule, or that people are experimenting with a chat window.
A better test comes from NIST's continuous monitoring guidance, SP 800-137, which frames monitoring as giving visibility into assets, awareness of threats and vulnerabilities, and visibility into whether controls are effective. Each AIDR metric should serve one of those three purposes. If it does none, drop it.
Metrics that matter
Coverage of the inventory. Share of AI systems and agents that are listed, owned and sending logs. Everything else depends on it. Report the denominator too: how sure are you that the list is complete?
Time to contain. From the moment a harmful behaviour begins (or an alert fires) to the moment the agent is paused, the credential revoked or the output blocked. Measure it in real incidents and in constructed exercises. Report the median and the worst case.
Detection precision, as reviewed. Of the alerts a person reviewed, how many were real? Pick a sample each month and label it. Precision tells you whether the team can trust the feed.
Detection coverage against named risks. Map detections to risks from catalogues such as MITRE ATLAS and the OWASP lists. The metric is the share of risks relevant to your systems with at least one tested detection. Add "tested", because an untested rule may never fire.
Permission scope. For agents: the share holding write or delete access, and the share whose access was reviewed within the policy window. OWASP names excessive functionality, excessive permissions and excessive autonomy as root causes of agent harm, so these numbers track the causes, not the symptoms.
Rollback success. In exercises, did the pause or revoke work, and how long did it take to undo?
Metrics that mislead
Blocked prompt count. Rises with curiosity, testing by your own staff and noisy rules. It says nothing about whether harmful prompts got through. Use blocked rate only next to a measured miss rate from tests.
Total alerts. More alerts can mean more risk or worse tuning. Pair with precision and with time to review.
"Zero incidents". May mean you are safe or that you cannot see. Report detection tests passed next to it.
Model accuracy scores from vendors. A score on the vendor's test set is not a score on your traffic. Ask for results on your data in a trial.
Mean time to detect, alone. Hard to measure when you do not know when the behaviour began. Use time from alert to containment, which you can actually timestamp.
Percent of traffic inspected. Inspection that cannot judge content adds latency and little else. Pair coverage with the depth of what is checked.
How should you present them?
Keep the board or leadership view to five numbers with a trend and a note: inventory coverage, time to contain, precision, tested coverage of relevant risks, and the share of agents with reviewed access. Put the rest in an operational view for the team.
How do you stop gaming?
Any target will be gamed. If you set a goal for fewer alerts, people mute rules. If you set a goal for precision alone, they delete the noisy rules that also catch real events. Use pairs: precision with tested coverage, speed with rollback success.
What about software supply chain data?
AI behaviour depends on components: models, plugins, tool servers and libraries. A useful addition is the share of AI systems whose component list is known and checked for flaws. It connects AI monitoring to the rest of your vulnerability work.
What to do
- Define the five headline metrics and how each is measured.
- Add a monthly sample review to label alert precision.
- Run a quarterly constructed exercise to time containment and rollback.
- Retire any metric that does not serve visibility, awareness or effectiveness.
- Report each metric with its denominator and its pair.
- Link AI systems to their component lists.
Safeguard focuses on the application and software supply chain side, including MCP Guardrail for MCP servers, and does not replace a dedicated AI detection tool.
Quick check
Is the number of blocked prompts a good AIDR metric?
Not alone. It rises with testing and noisy rules. Use it only alongside a measured miss rate from controlled tests.
What is the best single AIDR metric?
There is none, but time from alert to containment, measured in exercises, comes close. It shows both detection and response working together.
Sources
Last reviewed: 18 October 2026.
