To choose a software composition analysis tool, write the criteria down before you see a demo, weight them by what hurts your team most, and score each tool only on evidence from your own repositories. A scorecard will not pick for you, but it stops the loudest demo from winning.
Start with the criteria, not the vendors
Most evaluations drift because the criteria appear after the demos, shaped by whatever looked impressive. Reverse that. Agree on a short list first. Eight to ten rows is plenty.
A workable starting set:
- Ecosystem and lockfile coverage for the languages you actually ship
- Accuracy: missed vulnerabilities and wrong matches, measured on your code
- Prioritization: how well it separates urgent from noisy
- CI and pull request fit
- Fix guidance or automated fix pull requests
- License findings and policy
- SBOM import and export
- Data handling and deployment options
- Reporting, access control and audit trail
- Cost model fit and exit options
Set weights before scoring
Give each row a weight from 1 to 5, and cap the number of fives at three. If everything is a five, nothing is. A small team with a single language might weight CI fit and fix guidance highly and reporting low. A regulated team that answers customer questionnaires often weights SBOM export and audit trail higher.
Write the weights down and have the people who will live with the tool agree to them. Engineers, security and whoever owns procurement usually weight differently, and it is better to find that out now.
Score on evidence, not on claims
For each row, define what a 1, 3 and 5 look like in plain words. For example, on accuracy: 1 means it missed a known vulnerable package you seeded, 3 means it found everything but with several wrong matches, 5 means it found everything with few wrong matches. Then fill the cells from your trial notes, with a link or screenshot for every score.
Keep two columns apart: what the vendor said and what you observed. Only the second one is scored.
Add knockout rows
Some criteria are pass or fail. If a tool cannot read a package manager you depend on, or cannot run where your data policy requires, no weighted total rescues it. Mark these as knockouts and check them first, which saves days of trial time.
Leave room for judgment
Add up the totals, then look at the gaps. A tool that wins by two points but scores a one on developer experience may lose in practice, because tools nobody opens do not reduce risk. Treat the scorecard as input to a discussion, and note any score you would change after hearing the group.
Where Safeguard fits: Safeguard covers SBOM and dependency scanning with reachability analysis. You can run the free scan at gold.safeguard.sh/scan on one of your repositories and score it with the same sheet as everything else.
Quick check
How many criteria should a scorecard have?
Eight to ten is enough for most teams. More rows spread attention thin and make the weights meaningless.
Should price be a scored row?
Score fit of the cost model, such as whether it scales with developers, repositories or scans. Compare actual quotes separately once you have a shortlist.
Last reviewed: 19 October 2026.
