To evaluate a scanner, run it on your own repositories for about two weeks and measure five things: coverage of your stack, noise, how many findings get fixed, scan time, and who owns the results. A demo on sample code tells you little. Your own code and build setup tell you what you will live with.
The five measures
1. Coverage of your stack
List your languages, package managers, container bases and infrastructure formats. Confirm each one is supported by reading the tool's documentation, then test it. Seed known problems: an old library, a fake credential, a risky infrastructure setting. Record which ones the tool reports.
2. Noise
Sample fifty findings and label each as real, not applicable, or wrong. Calculate the share that is real. OWASP lists high false positive counts as a common weakness of static analysis, so check this for SAST in particular.
3. Fix rate
Give the findings to the owning team and count how many change code or configuration within the pilot. A tool with fewer, clearer findings often beats one with more.
4. Speed and fit
- Time added to a pull request.
- Whether it runs with the CI you already use.
- Whether results appear where developers work.
- Whether it needs build access, and how it handles private packages.
5. Ownership and workflow
Who triages? Who may accept a risk? Can findings be suppressed with a recorded reason? Can you export results?
Questions about the data
- Where is your code sent, and is that acceptable under your policies?
- How often is vulnerability data updated?
- Are components it could not identify shown, or silently dropped?
Scoring
Give each of the five areas a score from one to five, weighted by what matters to you. Write down the weights before you start, so the result does not follow the demo that impressed you most.
Exit criteria
Decide in advance what would make you stop: for example, a real share under half, a scan time over your budget, or a missed seeded issue. Pilots drift when exit criteria are vague.
Quick check
How long should a pilot run?
Two weeks is enough to see several pull request cycles and a few scheduled scans on real repositories.
Why seed known problems?
They give you a measure of misses, which counting reported findings alone cannot.
Sources
Last reviewed: 19 October 2026.
