Decide ten things before any demo: what you need to cover, who will act on results, what deadlines apply, which signals rank work, where tickets go, what must integrate, how you will test, what you will measure, who maintains it, and what you will stop doing. Clear answers make demos comparable and shorten the evaluation.
Why prepare first
Demos are built to impress. Without your own requirements, you will compare polish instead of fit. A written list also protects you from buying a category when you needed a process.
The ten questions
1. What is in scope? List applications, languages, package ecosystems, container images and deployment targets. Include the oddities such as a legacy service or a mobile app.
2. Who acts on findings? Name the team per application. If you cannot, fix that before buying.
3. What are your deadlines? Write the target time to fix for each priority tier, based on capacity.
4. Which signals will rank work? For example the CISA Known Exploited Vulnerabilities catalog, EPSS from FIRST, reachability and exposure. Say which are required.
5. Where does work live? Name the ticket tracker and whether tickets should be created automatically.
6. What must it connect to? Source hosts, CI systems, registries, chat, and identity provider. Mark each as required or nice to have.
7. How will you test it? Use your own repositories. Pick three applications of different sizes and one with known problems.
8. What will you measure? Choose a few metrics up front, such as coverage, owner assignment and time to fix, so you can judge after a trial.
9. Who will maintain it? Rules, exceptions and integrations need an owner after the purchase.
10. What will you retire? If a new tool does not replace anything, you add overhead. Name what it should replace.
Turn the answers into a scorecard
Make a simple table with your requirements as rows and the candidates as columns. Score each by what you saw in your own data, not by claims. Keep notes on any claim you could not verify in the trial.
| Requirement | Must or nice | Result in trial |
|---|---|---|
| Lockfile and image scanning for listed ecosystems | Must | |
| Reachability signal | Nice | |
| Ticket creation with deadline | Must | |
| Exception with expiry | Must | |
| Export of findings and SBOMs | Must |
Questions to put to any supplier
- Which of these requirements did you meet without custom work?
- What data does the output contain per finding? Check it against exact component, version and path.
- How are scan data and results stored, and can I export everything if I leave?
- What happens when your scanner is down?
NIST SP 800-218 offers a shared vocabulary for secure development practices that buyers can use with suppliers, which is useful when wording requirements.
Common mistakes
- Evaluating on a demo project instead of your own code.
- Counting findings as a quality signal.
- Skipping the ownership question.
- Buying for a future state you have not reached.
A worked example
A mid sized company drafts its answers before inviting any supplier. A sample of their one page brief reads as follows.
- Scope: 40 repositories, three languages, container images built in CI, one legacy service.
- Owners: each repository mapped to a team in a catalog.
- Deadlines: tier one in a short window, tier two longer, others in normal cycles.
- Signals: KEV, EPSS, reachability and exposure are required. Severity alone is not enough.
- Work location: tickets in the existing tracker, created automatically for tiers one and two.
- Integrations: source host and CI required, chat nice to have.
- Test plan: three of their own repositories, including the legacy one.
- Measures: coverage, owner assignment, time to fix.
- Maintainer: one named platform engineer.
- Retire: one existing scanner and two manual reports.
With this page in hand, every demo becomes an apples to apples test, and the team can say no quickly to products that miss a must.
Step by step: run the evaluation
- Draft the ten answers in a shared document.
- Review with engineering and security leads and fix disagreements.
- Invite two or three candidates to a time boxed trial with the same brief.
- Run the trial on your own repositories and record results in the scorecard.
- Collect developer feedback on messages and noise.
- Check export and exit by exporting everything and reading it.
- Decide with the metrics, not the demo.
Example scoring approach
| Criterion | Weight | How to score |
|---|---|---|
| Coverage of your ecosystems | High | Percent of your repositories scanned correctly |
| Data quality per finding | High | Fields present on a sample of 20 |
| Fit with your tracker | Medium | Tickets created correctly in the trial |
| Noise level | High | Share of findings a developer marks useful |
| Exit and export | Medium | Time to export all data in a usable format |
Weights should reflect your priorities, so set them before the trials.
Red flags during evaluation
- Reluctance to run on your own code.
- Counting findings as proof of quality.
- No way to explain why a finding is ranked where it is.
- Unclear answers about data storage and export.
- Claims that cannot be tested in the trial.
Write them down as they appear. They tend to matter more after purchase than the features highlighted in the demo.
Try it yourself
Open the demo and look at how direct and transitive components are listed for one project.
Click through it at your own pace, or open it full screen.
Where Safeguard fits
Safeguard offers dependency scanning with reachability analysis, SBOM generation, policies and gates, and a free scan at gold.safeguard.sh/scan that you can run on your own code while you work through your list.
Quick check
Should I run a trial on a vendor's sample project?
Use your own repositories. A sample project cannot show how a tool handles your ecosystems and scale.
What if we cannot name owners for every application?
Resolve that first. Any tool will surface findings that nobody is assigned to fix.
How long should a trial last?
Long enough to see at least one full cycle of scan, ticket, fix and rescan on your own code, which is often two to four weeks.
Sources
- CISA Known Exploited Vulnerabilities Catalog
- FIRST, EPSS
- NIST SP 800-218, Secure Software Development Framework
Last reviewed: 19 October 2026.
