Five measures answer the question: the share of AI comments developers act on, security defects that escape to production, time from finding to fix, review time per pull request, and developer opt-out. Comment volume alone is a vanity number, because a noisy tool produces plenty.
Why comment count misleads
A tool that posts 40 comments per pull request looks busy. If developers resolve three, it is mostly noise. A tool that posts two comments, and both lead to changes, is doing more. Pick measures that connect the tool to outcomes, and set a baseline before you switch it on.
The five measures
| Measure | How to compute | What it tells you |
|---|---|---|
| Acceptance rate | Comments that led to a code change divided by comments posted | Signal versus noise |
| Escaped security defects | Security bugs found after merge, per quarter, with a note on whether the tool could have caught them | Whether it prevents risk |
| Time to fix | Days from finding raised to fix merged | Whether findings are actionable |
| Review time | Median hours from pull request opened to approved | Whether it adds friction |
| Opt-out and mute rate | Repositories or people who disabled or ignored it | Trust |
Acceptance rate
Count a comment as accepted when the author changes code in response, or replies agreeing and links a follow-up. Dismissals with a reason are neutral, and silent ignores count as not accepted. Track by rule or comment type, so you can switch off the worst ones.
Escaped defects
This is the measure that matters most and is hardest to read. Fewer escapes can come from many causes. Look at each escape individually: was the vulnerable change reviewed by the tool, what did it say, and why was it missed? Over a few quarters, patterns appear.
Time to fix
If findings sit open for weeks, they are either wrong, unclear or low priority. Compare AI-raised findings with findings from other sources. Our piece on reopen rate and false positive rate shows how to check data quality of these numbers.
Review time
Some tools save human reviewer time. Others add reading. Measure both the pull request cycle time and the hours reviewers report spending. If cycle time rises with no drop in escapes, the tool is costing more than it returns.
Opt-out rate
Developers vote with their settings. A growing mute rate means trust is falling, whatever the dashboard says.
Set up the baseline
Before enabling the tool, collect eight to twelve weeks of the same measures. After enabling, compare like periods. Other changes in the same window, such as a new framework or team growth, will muddy results, so note them.
What to avoid
- Reporting a count of "issues found" as success.
- Counting duplicates of findings other tools already raise as new value.
- Treating a drop in findings as proof of safer code. It may mean a rule stopped firing.
- Setting targets on acceptance rate that push developers to click accept.
How often to review
Look monthly during the first quarter, then quarterly. Share a one-page summary with engineering leadership, with the raw counts attached.
Quick check
What is a healthy acceptance rate?
There is no standard figure. Compare against your own baseline and against other review sources, and watch the trend by rule.
Can I measure escaped defects without a long wait?
Use seeded test cases for a fast read, and use real escapes for the long view.
Safeguard includes reachability analysis and autonomous remediation with fix PRs (off by default, with every category on Manual). Try the free scan at gold.safeguard.sh/scan.
Last reviewed: 19 October 2026.
