To calculate scanner precision, pull a random sample of findings, have someone who knows the code label each as real or not, and divide real by sampled. Then scale that rate to the full list. Precision tells you about noise. It says nothing about what the scanner missed.
Step 1: define what counts as real
Precision depends entirely on the label. Decide before sampling:
- Real means the issue exists and would matter if left alone.
- Not real covers wrong matches, test code, unreachable paths, and findings on code you do not ship.
- Real but accepted is a third bucket. Track it separately, because it is a policy decision, not a scanner error.
Write the definition down. Two reviewers who disagree on it will give you two precision numbers from the same data.
Step 2: draw a random sample
Do not take the first 40 rows or the top severities. Number the findings and pick at random, or sort by a random key and take the top. If you want precision per severity or per rule, sample each group on its own.
A sample of 40 gives a rough answer. Our guide on sample size for false positive rates covers how many you need for a given margin.
Step 3: worked example
The numbers are illustrative. A scan returns 200 findings. You sample 40 and a developer labels 14 as real.
- Precision in the sample: 14 divided by 40, which is 35 percent.
- Estimate for all 200: about 70 real findings and 130 that are not.
With only 40 labels the true figure could plausibly sit anywhere from the high 20s to the high 40s in percent. Report it as roughly one third, not as 35.0.
Step 4: pair it with recall
Precision answers "of what you showed me, how much is real?" Recall answers "of what is really there, how much did you show me?" A scanner can have high precision by being very quiet, and it will miss a lot. You measure recall differently, with seeded issues or an independent second source. See testing a SAST tool with seeded bugs for one approach.
Reporting only one of the two is the most common way scanner comparisons mislead.
Step 5: track it over time
Precision drifts as rules change and as your codebase changes. Re-sample a smaller batch each quarter and plot the number. A sudden drop after a rule update is a sign to look at that rule.
Common mistakes
- Letting the tool vendor label the sample. Use your own engineers.
- Counting duplicates as separate findings. Deduplicate first, or you will inflate both real and not real counts.
- Mixing finding types. A scanner may be precise on secrets and noisy on style checks. One blended number hides that.
- Treating a reviewer's "I do not have time to check" as not real. Label it unknown and exclude it, and say how many were unknown.
Quick check
Is precision the same as accuracy?
No. Precision is the share of reported findings that are real. Accuracy mixes in the things correctly left unreported, which makes it look better when most items are benign.
Can I get precision without labelling anything?
No. Someone has to judge a sample. Triage outcomes you already recorded, such as dismissed versus fixed, can seed the estimate if the dismissal reasons are consistent.
Last reviewed: 19 October 2026.
