Track six things after rolling out an AI coding agent: how many actions policy blocked, how often people approve prompts, how many dependencies agents add and how many fail checks, how deeply agent pull requests are reviewed, how old agent credentials are, and how many tools or servers are not on the approved list.
Why measure at all
Controls drift. A setting changes, a developer flips an auto approve flag, a new server gets connected. Metrics tell you when the world differs from your policy. They also give leadership something firmer than a feeling.
Pick few metrics and define each one precisely. Dashboards with thirty tiles get ignored.
The six
| Metric | Definition | What it tells you |
|---|---|---|
| Blocked action count | Agent actions refused by policy per week | Controls are active, and what people try to do |
| Approval rate | Share of risky prompts that are approved | Very high means rubber stamping, very low means friction |
| Agent dependency intake | New packages added in agent pull requests, and the share failing a check | Quality of what agents pick |
| Review depth | Time to first review, comments per pull request, share merged without comment | Whether humans are really reading |
| Credential age | Median age of agent tokens, and count past expiry | Hygiene of the identity layer |
| Unapproved tools | Connected servers or extensions not on the allowlist | Shadow use and inventory gaps |
How to read them
Blocked actions. A count of zero is suspicious, not reassuring. It may mean nothing is enforced or nothing is logged. Look at what is blocked: repeated attempts to reach a new host deserve a look.
Approval rate. If nearly every prompt gets approved within a second, the prompts are noise. Shrink the prompt set to the risky tier, as described in our post on whether agents should run commands without approval.
Dependency intake. Compare agent pull requests with human ones. A higher failure rate for agent added packages points to guardrails or prompts that need work.
Review depth. An agent pull request merged with no comments and a ten second review is a warning. Combine it with diff size.
Credential age. Old tokens are the stale keys of the agent world.
Unapproved tools. Needs an inventory. Developers add servers through local config, so scan for it.
Avoid vanity numbers
Lines of code produced, suggestions accepted, and hours saved are productivity measures. They have a place, but they say nothing about risk. Keep them on a separate page so nobody confuses the two.
Set thresholds and owners
For each metric, write a trigger and an owner. Example: "If median agent token age passes thirty days, the platform team rotates and reports." A metric without an owner is a chart.
Review cadence
Look weekly during the first month, then monthly. Compare against the baseline you set during the pilot. The OWASP Top 10 for LLM Applications is a useful lens for choosing what to watch: Excessive Agency (LLM06) suggests tracking permissions, and Supply Chain (LLM03) suggests tracking intake of components.
Close the loop
Every review should end with one decision: tighten a control, loosen a friction point, or leave things alone. A metric that never changes a decision can be dropped.
Safeguard can help at the dependency and pipeline end of this: SBOM and dependency scanning, reachability analysis, policies and gates on pull requests, and agent identity. You can try a free scan at gold.safeguard.sh/scan.
A worked example: a first month dashboard
Imagine a team with fifteen developers and one sanctioned agent. After thirty days they have these numbers.
| Metric | Week 1 | Week 2 | Week 3 | Week 4 |
|---|---|---|---|---|
| Blocked actions | 12 | 9 | 4 | 3 |
| Approval rate on risky prompts | 97 percent | 96 percent | 85 percent | 78 percent |
| Agent added dependencies | 6 | 11 | 8 | 9 |
| Share failing a dependency check | 33 percent | 27 percent | 12 percent | 11 percent |
| Agent pull requests merged with no comments | 70 percent | 62 percent | 40 percent | 33 percent |
| Median agent token age (days) | 3 | 9 | 16 | 22 |
| Unapproved tool servers found | 4 | 3 | 1 | 1 |
How to read it. Blocked actions fell as developers learned the boundaries, which is healthy. The approval rate near 97 percent in week one signaled rubber stamping, and it fell after the team trimmed the prompt set and added short training. The failure share on dependencies fell after a policy gate was added. The merged without comment share is still high at 33 percent, so the team schedules a reviewer session. Median token age is climbing, which means rotation is not automated yet, and that becomes the next fix.
Percentages here are an illustration of how to read a dashboard, not benchmarks for your team.
Defining each metric precisely
| Metric | Numerator | Denominator | Source |
|---|---|---|---|
| Approval rate | Prompts approved | Prompts shown | Tool or wrapper log |
| Dependency failure share | Agent added packages failing a check | Agent added packages | Pipeline results |
| No comment merges | Agent pull requests merged without comment | Agent pull requests merged | Source control |
| Token age | Median days since issue | Active agent tokens | Identity system |
Writing definitions down stops arguments about whether a number moved or the definition did.
Thresholds and actions
- If approval rate stays above 95 percent for two weeks, shrink the prompt set.
- If the dependency failure share rises, review the guardrails and the agent's instructions.
- If median token age passes your limit, rotate and automate expiry.
- If unapproved servers appear, run the vetting steps or remove them.
- If blocked actions spike for one user, talk with them before assuming bad intent, since they may be doing legitimate work that the policy mishandles.
Reporting upward
Leadership wants three lines: are controls working, what changed this month, and what we are fixing next. Use a single page, with trend arrows and one sentence each. Keep the detail available but not on the first page.
Pair with a risk register
Record any metric that stays red in your risk register with an owner and a date. That connects the numbers to decisions, which is the whole point of measuring.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through and notice how findings are tracked and prioritized over time.
Quick check
Is zero blocked actions good?
Not by itself. Confirm that enforcement and logging work by running a test that should be blocked.
How many metrics should we start with?
Six or fewer, each with a definition, a threshold, and an owner.
Should we compare agent pull requests with human ones?
Yes, on review depth and dependency failures. Differences show where agent specific guardrails or reviewer habits need work.
Sources
Last reviewed: 19 October 2026.
