Evaluate a supply chain security tool by running it on your own repositories for two weeks, against a baseline you already know. Score accuracy, noise, fix quality, data formats and effort to adopt. A demo on the vendor's sample project proves very little about your dependency tree.
Most trials fail for a dull reason: nobody wrote down what a good result looks like before the tool ran. This plan fixes that. It works for any tool in the category, and it does not name or rank vendors.
What should you decide before the trial starts?
Pick three repositories: one typical service, one old and messy one, and one that ships to customers. Then write three lists.
- Known problems. Dependencies you already know are vulnerable, with CVE IDs. Add one or two you know are not reachable from your code.
- Known clean. A few packages you know are fine. A tool that flags them has a false positive problem.
- Must-haves. The formats and workflows you cannot live without, such as CycloneDX or SPDX export, pull request checks, or ticket creation.
Add a time box and an owner. Two weeks and one engineer is enough. Without an owner, trials drift into a vendor-led slideshow.
Which questions matter in week one?
Week one is about the data.
- Does it find the dependency tree you actually have? Compare its component count with your lockfile. Check transitive packages, not just direct ones.
- Does it find the known problems? Count hits against your list. Note how each finding is dated and sourced.
- How does it rank findings? CISA's KEV catalog lists vulnerabilities with evidence of exploitation, and FIRST's EPSS publishes a daily 30 day exploitation probability. A tool that uses neither, or cannot say what it does use, gives you a harder triage job.
- Can it say a finding does not apply? The VEX idea, described by CISA as an attestation that a product is or is not affected by a vulnerability, matters here. Ask whether a not-affected status can be recorded and exported.
- What leaves your machine? Ask exactly: source code, manifests, or only results. Get the answer in writing.
Which questions matter in week two?
Week two is about work. Take five real findings and push them all the way through.
- Does the suggested fix compile and pass your tests, or does it only bump a version?
- How many clicks or commands from finding to pull request?
- Does the pull request check fail for the right reasons, and can a developer see why?
- Can you suppress a finding with a reason and an expiry date, and does that survive the next scan?
- What does an audit export look like? Ask for the file and open it.
Then try one failure on purpose: scan a repository with a private package name that also exists in the public registry. See what the tool says. A tool that treats every package name as public can miss dependency confusion.
What does a simple scoring sheet look like?
Give each criterion 0 to 3 and write one sentence of evidence. Weight the rows by what hurts you most.
- Coverage of your ecosystems
- Accuracy against known problems and known clean
- Noise: findings per repository that nobody would act on
- Fix quality
- Evidence and export formats
- Data handling and deployment options
- Time from install to first useful result
If you want a zero-signup baseline to compare against, Safeguard's free Gold scan at gold.safeguard.sh/scan runs in your browser on up to 100 dependencies. It is one more data point, not a verdict.
What are the honest limits of a trial?
Two weeks will not show how a tool behaves after a year of rule changes, or how its vendor handles a bad false positive at midnight. Ask for two customer references that look like you, and ask what the vendor's support did on its worst day. Also remember that a tool tuned to look good on public sample repositories can behave differently on your private code.
What to do
- Write the known-problem, known-clean and must-have lists before installing anything.
- Run every candidate on the same three repositories in the same week.
- Record component counts, finding counts and time spent per finding.
- Push five findings through to merged fixes.
- Score with the sheet and have two people score separately.
- Ask each vendor to put data handling answers in the contract, not just the demo.
Quick check
How long should a security tool trial last?
Two weeks is enough for one engineer to test accuracy, fixes and workflow on three repositories. Longer trials tend to lose focus unless they have a written scorecard.
What is the fastest way to spot a noisy tool?
Compare its findings with your known-clean list and ask it to explain five findings. If nobody can say why a finding matters here, triage will eat your week.
Sources
- CISA, Software Bill of Materials (SBOM) and VEX
- CISA, Known Exploited Vulnerabilities Catalog
- FIRST, Exploit Prediction Scoring System
- NIST SP 800-218, Secure Software Development Framework
Last reviewed: 18 October 2026.
