Ask a software composition analysis vendor five things: how it resolves your real dependency tree, which vulnerability data it uses, how it separates reachable risk from noise, what it exports, and where it is known to be wrong. Run those questions against your own repositories, not a demo project.
Does it see the dependencies you actually ship?
Most disappointing SCA trials fail at step one. The tool reads a manifest, guesses at versions, and misses what the build resolves. Ask whether it reads lockfiles, resolves transitive packages, and handles monorepos with several ecosystems. Then check the output against a build you know well.
A fair test: pick a service with a messy tree, count the packages your build tool reports, and compare. Gaps of a few percent are normal. Large gaps mean the scanner is guessing.
Also ask about container images and vendored code. Libraries copied into a source folder have no manifest entry, and many tools skip them silently.
Which vulnerability data does it use, and how fresh is it?
A scanner is only as good as its feed. Ask the vendor to name its sources. Public ones worth knowing are the CISA Known Exploited Vulnerabilities catalog and OSV, which describes itself as an open database that aggregates advisories from sources such as GitHub Security Advisories, PyPA and RustSec across 38 ecosystems.
Then ask how quickly a new advisory reaches your results, and what happens when two sources disagree about affected versions. A vendor who has thought about this will have a concrete answer. A vendor who says "we use AI" has not.
How does it decide what matters?
A raw count of CVEs is not a risk score. Ask what the vendor layers on top. Three signals are public and worth demanding:
- Presence in the KEV catalog, which lists vulnerabilities exploited in the wild.
- An exploit probability such as EPSS, a model that estimates the chance a CVE is exploited in the next 30 days.
- Whether your code calls the vulnerable function at all, usually called reachability analysis.
The last one cuts the most noise, and it is also the easiest to oversell. Ask which languages it covers, whether it uses static call graphs, and what it does when the analysis cannot decide. The honest answer is "we flag it as unknown," not "we are always right."
Can it say "not affected" in a form others can read?
Customers and auditors increasingly ask whether a flagged vulnerability matters in your product. The CycloneDX VEX capability exists to convey exploitability of a component in the context of the product that uses it. CISA publishes minimum elements for VEX documents.
Ask whether the tool can import and export SBOMs and VEX statements, and in which formats. Lock-in often hides in the export button.
Where does it run, and where does your code go?
Ask what leaves your network. Some tools upload source, some upload only a manifest, some run fully local. Ask whether a mode without any AI component exists, because some regulated teams need one. Ask who the subprocessors are. Get the answer in writing.
What does it not do?
Every scanner has limits. Ask for the three most common reasons customers see missed findings or false positives. Ask about license risk, malicious packages and secrets, and whether those are in the same product or a separate purchase. A vendor unwilling to name a weakness is telling you something.
What to do
- Pick two or three representative repositories, including one ugly one.
- Run every shortlisted tool on the same commits and diff the package lists.
- Take ten findings each tool labeled critical and check by hand whether the code path is reachable.
- Request a sample SBOM and VEX export and open it in a second tool.
- Ask for the data sources, update timing and data handling in writing.
- Price the integration time, not only the license.
If you want a no-account baseline, the free in-browser scan at gold.safeguard.sh/scan checks up to 100 dependencies and is a quick way to see how a tree looks before a trial.
Quick check
What is the single best test of an SCA tool?
Run it on a repository you know well and compare its package list with what your build resolves. If it misses packages, nothing else it reports can be trusted.
Is a lower CVE count always better?
No. A low count can mean good prioritization or poor coverage. Check the package list first, then the ranking.
Do I need reachability analysis?
It helps cut noise, but treat it as a ranking aid with known limits, not proof that an unreachable finding is safe forever.
Sources
- CISA Known Exploited Vulnerabilities catalog
- OSV, Open Source Vulnerabilities
- FIRST EPSS
- CycloneDX VEX
- CISA minimum requirements for VEX
Last reviewed: 18 October 2026.
