A good scorecard for a vendor's AI feature covers six areas: the job it does, the evidence it shows, how it handles your data, the controls around it, its maturity, and how you exit. Score each from 0 to 3 against written criteria, and record who gave each score.
The six areas
| Area | 0 | 3 |
|---|---|---|
| Job | Vague claim | Named task with a measurable outcome |
| Evidence | None shown | Traceable reasons on your own data |
| Data handling | Unclear | Documented inputs, retention, training use |
| Controls | None | Approvals, limits, audit trail, off switch |
| Maturity | Roadmap only | Generally available with a track record |
| Exit | Locked in | Your data and settings export cleanly |
How to score fairly
Score in pairs. Two people grade the same feature independently, then discuss differences. Use evidence you saw yourself, not the vendor's summary. If you did not see it, the score is capped at 1 until you do.
Keep the weights simple. Many teams weight evidence and controls double, because those two areas show whether a feature is safe to run. Decide weights before the demos.
Add a column for unknowns
Create a column for "asked, not yet answered." It is more useful than a low score, because it separates a bad answer from no answer. Send that list to the vendor with a date.
Use the scorecard for decisions, not only comparison
Set a floor. For example, no feature that can change code is adopted without a 3 on controls. Setting a floor stops a high score on one area from hiding a gap in another.
Revisit after adoption
Scores age. After six months, repeat the evidence and controls rows with fresh data. NIST's AI Risk Management Framework, voluntary in nature, treats management as ongoing rather than one time, and a six month rescore is a modest way to follow that spirit.
A note on fairness
Give each vendor the same scorecard and the same trial rules. It protects you from bias and makes your decision easier to defend later to your own leadership or auditors.
Try it yourself
Click through it at your own pace, or open it full screen.
Pick one feature on screen and try scoring it on the six rows above.
Where Safeguard fits
Safeguard marks capabilities as generally available or early access, and you can try the free scan at gold.safeguard.sh/scan to fill the evidence row yourself.
Quick check
How many vendors should I score?
Two or three is practical. More dilutes the hands on testing.
Who should score?
An engineer, a security reviewer, and someone from privacy or procurement.
Sources
- NIST, AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
Last reviewed: 19 October 2026.
