How to measure false positives and misses in a security tool trial

To measure accuracy in a trial, build a labeled answer key for a small set of repositories, run each tool, and count three things: findings that are real, findings that are not, and real issues the tool missed. From these you get precision and recall, two numbers you can compare across vendors.

Build the answer key

You need ground truth. Choose 3 to 5 repositories and label every issue you can confirm. Sources for labels:

• Findings your team already triaged and fixed in the past year
• Planted flaws you add to a test repository, such as a known vulnerable dependency version
• Public test suites designed for tool comparison. The OWASP Benchmark project, for example, is a runnable Java and Python web application that SAST, DAST, and IAST tools can scan, and its stated aim is that every planted flaw is genuinely exploitable. It has its own limits as a stand in for your code, so use it as one input.

Write the key in a spreadsheet: location, type, severity, and whether it is real. Lock it before any tool runs.

Count the outcomes

For each tool, classify each reported finding:

1. True positive: matches a real item in the key
2. False positive: reported, but not a real issue in your context
3. Duplicate: same issue reported more than once (track it separately)

Then count false negatives: items in the key that the tool never reported.

Compute two numbers

• Precision is true positives divided by all reported findings. It tells you how much of what you read is worth reading.
• Recall is true positives divided by all real items in the key. It tells you how much of the real problem the tool saw.

Worked example: the key has 40 real issues. Tool A reports 60 findings, 30 are real. Precision is 30 divided by 60, or 50 percent. Recall is 30 divided by 40, or 75 percent. Tool B reports 35 findings, 28 are real. Precision is 80 percent, recall is 70 percent. Tool B is cleaner, Tool A sees slightly more. Which matters more depends on your pain, which is why the weights in your rubric come first.

Handle the gray areas

Some findings are real but unreachable, or real but low risk. Decide in advance how to classify them. A common rule: a finding is a true positive if the issue exists, and record separately whether it would change your priorities. Count "real but not worth fixing" as its own column so you can see the effect of prioritization.

Have two reviewers label disputed items and agree. A third person breaks ties.

Beware these traps

• Small samples. With 10 items, one miss moves recall by ten points. Use at least 30 labeled items if you can.
• Tuning for the trial. If a vendor adjusts rules after seeing your key, rerun on a fresh repository.
• Counting severity labels as accuracy. A tool that marks everything critical is not accurate, it is loud.
• Ignoring the time to triage. Record how long each false positive took to dismiss.

Report it plainly

Put the results in one table with precision, recall, duplicates, and triage minutes per vendor. Add two sentences on what surprised you. Reviewers read the table and the notes, so keep both short.

For a quick look at how dependency findings and reachability appear on real code, try the free scan at https://gold.safeguard.sh/scan, and see how reachability analysis changes the count of items worth reading.

Sources

• OWASP Benchmark project

Quick check

Last reviewed: 19 October 2026.