Build the scorecard before you see a demo. List six to eight criteria, weight them by what hurts your team most, and score each tool only on evidence from your own repositories. A scorecard turns a sales conversation into a comparison you can defend to finance and to engineering.
Decide the criteria first
Most teams score what vendors show on a slide. Start from your own pain instead. Ask the people who triage findings today what wastes their time. The answers usually fall into a short list:
- Coverage of the languages and package managers you actually ship
- Accuracy of the findings, meaning few false positives and few misses
- Noise reduction, such as whether the tool can tell if vulnerable code is called
- Fit with your pipeline and pull request flow
- Quality of fix guidance
- SBOM output in a standard format
- License findings, if legal asks for them
- Cost model and what drives it
Keep the list short. Twelve criteria means nobody remembers the weights.
Weight by consequence
Give each criterion a weight from one to five, and make the group agree before scoring. A team drowning in alerts should weight noise reduction high. A team facing a buyer demand for SBOMs should weight format and completeness high. Write the weights down and do not change them after the demos.
Score on evidence, not claims
Each score needs an artifact behind it. For accuracy, that is a spreadsheet of findings from one real repository, reviewed by a human. For coverage, it is a list of your manifests with a yes or no for each. For pipeline fit, it is a run of the tool inside a test branch.
If a vendor cannot let you test a criterion, score it zero for now. You can raise it later with proof.
Use a three level scale
A one to ten scale invites false precision. Use three levels:
- 0: not supported, or could not be tested
- 1: works with caveats you can name
- 2: works in your environment, with evidence
Multiply by the weight and add up. The total matters less than the conversation about where the gaps are.
Add two gates that are not scored
Some things should disqualify a tool rather than lose points. Typical gates are data handling that your security review will not accept, and no way to run in your pipeline at all. Decide your gates up front, and apply them before scoring so you do not waste a pilot.
Record the decision
Keep the scorecard, the weights, and the evidence in one folder. When someone asks in a year why you chose a tool, you can answer in five minutes. When the contract renews, you can rerun the same scorecard against the same repositories.
For where the data in an SCA tool comes from, see questions to ask about SCA vulnerability data.
Safeguard offers SBOM and dependency scanning with reachability analysis, and a free scan at https://gold.safeguard.sh/scan lets you test one repository against your own scorecard.
Quick check
How many tools should be on the scorecard?
Two to four. More than that and the pilot work grows faster than the insight you gain.
Who should set the weights?
The people who live with the output: engineering, security, and whoever answers customer questionnaires. Agree on them before any demo.
Last reviewed: 19 October 2026.
