Evaluate a software composition analysis tool by running it on two or three of your own repositories, checking its findings against a list you verified by hand, and scoring coverage, false positives, fix guidance and CI fit. Two weeks is enough. Ignore demo repositories, which are tuned to look good.
We name no vendors. The method applies to any legacy or modern SCA product, including ours.
What should you prepare before the trial starts?
Pick the test set. Choose one repository you know is messy, with a monorepo or many transitive dependencies, one ordinary service and one container image. Include at least one ecosystem that is awkward for you, such as a private registry or a vendored directory.
Build ground truth. For the messy repository, list ten components by hand with exact resolved versions from the lockfile. Pick three known vulnerable packages from recent advisories, including one deep transitive one. CISA's Known Exploited Vulnerabilities catalog is a handy source of confirmed, real world cases to test against, since each entry is a vulnerability that has been exploited.
Write the success criteria down, with numbers, before you see any results.
Which criteria matter?
Coverage. Does the tool find all ecosystems in the repo, and resolve transitive dependencies to the versions actually installed? Compare against your ten components. Missing a deep dependency is a more serious failure than a cosmetic gap.
Accuracy. Count false positives in a sample of twenty findings: wrong version matched, vulnerability does not apply to this package, or already fixed. Count false negatives against your three planted cases.
Prioritization. Does it help you decide what to fix first? Look for signals such as whether the vulnerable code is actually reachable and whether exploitation has been observed. The FIRST page for EPSS describes a data driven way to estimate exploitation likelihood, and tools that use it, or the KEV catalog, give you more than a severity number.
Fix guidance. Does it name the first safe version, show a diff or open a pull request? Apply one suggested fix and see whether the build and tests pass.
Workflow fit. Time a CI run. Check the output formats your pipeline needs, such as SARIF or JUnit, and see whether the gate can be tuned without code. Check who sees what: roles, projects and audit logs.
Data handling. Find out whether source code is uploaded or only a dependency list and results. Some tools scan locally and send results only. Match this to your own rules.
Honest limits. Ask what the tool does not cover. Note whether the answer matches what you find.
How do you run it without wasting time?
Week one: install, scan the three targets, review coverage against ground truth. Week two: triage a sample, apply fixes, wire a CI job and measure noise. Keep a simple log of minutes spent per task, because setup effort is a cost.
If you want a no-commitment first look at a dependency list, a free in-browser scan at gold.safeguard.sh/scan checks up to 100 dependencies. It is not a replacement for a full trial on your repositories.
What mistakes ruin trials?
Using the vendor's sample project. Judging by the number of findings, since more findings often means more noise. Skipping the CI test. Letting a champion on one team decide alone. Not recording results, so the decision rests on memory.
NIST's Secure Software Development Framework gives a shared vocabulary for what these tools support: finding and addressing vulnerabilities in released software. Cite it to stakeholders who ask why the tool is needed.
What to do
- Choose three test targets, including one messy and one awkward ecosystem.
- Build ground truth: ten components, three known vulnerable packages.
- Set numeric success criteria before testing.
- Run each candidate on the same targets in the same week.
- Sample twenty findings per tool and count false positives.
- Apply one suggested fix and run your tests.
- Time a CI run and check the output formats you need.
- Record scores in one table and decide with the table, not the demo.
Quick check
How long should an SCA trial last?
Two weeks is usually enough when you prepare ground truth first. Longer trials mostly add noise.
Is more findings a better result?
Not by itself. Check accuracy and prioritization, since extra findings may be false or unreachable.
Sources
- Known Exploited Vulnerabilities Catalog, CISA, fetched 18 October 2026
- Exploit Prediction Scoring System (EPSS), FIRST, fetched 18 October 2026
- NIST SP 800-218, Secure Software Development Framework, fetched 18 October 2026
Last reviewed: 18 October 2026.
