To test a vulnerability scanner in a proof of concept, use a project you understand and check each pipeline stage on purpose: seed known components, confirm matching, inspect enrichment, test prioritization, read the reports, and run a fix through to a clean rescan. Judge the tool on what it explains, not on how many findings it prints.
Prepare a test project
Pick one repository you know well, and prepare a short answer key before scanning:
- Its real dependency list, direct and transitive, from the lockfile.
- Two or three components you know have published advisories.
- One vendored or copied library with no manifest entry.
- One component you know is development-only.
- One advisory that has since been withdrawn, if you can find one.
The answer key is the point. Without it you are comparing counts, which tells you little.
Test each stage
| Stage | Test | Pass looks like |
|---|---|---|
| Inventory | Compare the tool's component list with your answer key | Every real component found; vendored code flagged or listed as unidentified |
| Matching | Check the known advisories appear, and one clean package does not | Correct ecosystem, name and version range handling |
| Matching | Check the withdrawn advisory | Not reported, or clearly marked |
| Enrichment | Inspect severity, exploit and fix fields | Source and version of each score shown |
| Prioritization | Ask why finding A ranks above B | A reason you can read, with reachability and exploit data |
| Reporting | Export in a machine format and a readable one | Both are complete and agree |
| Handoff | Open a fix for one finding | Specific version, correct repository, reviewable change |
For exports, check that the formats you need are supported. SARIF is the OASIS standard for static analysis results, and the OSV schema is a common format for advisory data.
Then test the awkward cases
- Run it twice. Results on unchanged code should match, with no finding flipping to new.
- Change nothing but wait. Rescan after a few days and see whether new advisories arrive without a rebuild.
- Break a path. Move a manifest and see whether the tool warns about the coverage change.
- Ask what it skipped. A good tool tells you what it could not read.
- Run the fix. Apply a suggested change, rebuild, rescan and check the finding closes.
Questions for the vendor
- How often is advisory data refreshed, and can you see the data version on each result?
- Which sources does it read, and how are aliases merged?
- How does it treat an unidentifiable component?
- Can reachability verdicts be inspected?
- What can you export if you leave?
Score it honestly
Use a simple sheet: for each stage, record pass, partial or fail, with evidence. Weight stages by what you care about. A tool that is excellent at matching but cannot explain ranking may suit a team with strong triage staff, and poorly suit one without. Avoid a single combined score that hides the shape.
Keep the answer key, the scan outputs and your notes together, so the decision can be reviewed later.
Where Safeguard fits
Safeguard's free scan is an easy way to run a first test on a project you know, covering SBOM and dependency scanning, with reachability analysis, policies and gates available beyond that.
Quick check
How many findings should a good scanner report?
There is no right number. Judge accuracy against your answer key and look at how well each finding is explained.
Why include a vendored library in the test?
It shows how the tool handles components with no manifest entry, which is where many scans go quiet.
Sources
Last reviewed: 19 October 2026.
