Ask fifteen questions in four groups: scale and performance, ownership routing, policy and inheritance, and operations. For each, listen for a concrete answer you can verify in a trial. Vague answers such as "it scales" or "we integrate with everything" mean you should test the claim yourself.
Scale and performance
- What is the largest repository you have scanned, and what happened? A real story with limits beats a number.
- How do scans behave when 50 pull requests open at once? Look for queueing, concurrency controls and clear status.
- How do you handle source control API limits? Good answers mention reading rate limit headers, backoff and caching. GitHub, for example, documents hourly primary limits and secondary limits on concurrency and content generating requests.
- Can you scan a monorepo by path and report per package? See does the scanner handle monorepos.
- What do you do when a scan cannot finish? It should report partial results as partial, not as clean.
Ownership routing
- Where does ownership come from? Catalog, CODEOWNERS, identity groups, manual mapping, or a mix.
- Is ownership re-resolved on every scan? Copy once means stale after the first reorg.
- Can I see findings with no owner as a metric? If not, ownership gaps are invisible.
Policy and inheritance
- How do org, unit and team policies combine? Ask for the rule: lower layers tighten freely, loosening needs approval.
- Can I preview the effect of a policy change before enforcing? A dry run that lists newly failing repos.
- How do exceptions work? Named approver, reason, expiry, and a report of everything active.
Operations
- What do audit logs record, and can I export them? Policy changes, approvals, role changes, token creation.
- How are SBOMs produced and exchanged? Formats, depth, and whether NTIA minimum elements are present. The NTIA elements include supplier, component name, version, unique identifiers, dependency relationship, author of SBOM data and timestamp.
- How do I get data out? API coverage, webhooks, export. You will build dashboards.
- What happens at the end of the contract? Data export and deletion terms, in writing.
Scoring answers
Use a simple three level rating for each question:
| Rating | Meaning |
|---|---|
| Verified | You saw it work in your own trial |
| Claimed | Documented or demonstrated by the vendor only |
| Unclear | No concrete answer yet |
Only "Verified" should move a decision. Ask for time boxed trial access so more answers can be upgraded.
How to run the conversation
Send the questions in advance and ask for written answers. Written responses are easier to compare across vendors and can become part of the contract record. Then spend live time on the five you care about most, and ask for a screen share of each rather than a description.
Pair this list with a scale test, described in how to scale test an SCA tool before buying, and with a clear definition of what success means, such as the criteria in pilot vs proof of value vs proof of concept.
What not to ask
Skip questions that invite superlatives, such as who is fastest or most accurate. They produce marketing, not evidence. Ask for behavior you can observe.
Safeguard provides SBOM and dependency scanning, reachability analysis, policies and gates, and a free scan at gold.safeguard.sh/scan, so several of these answers can be checked on your own code.
Quick check
How many questions are too many?
Fifteen is manageable. Prioritize the five where a wrong answer would hurt you most.
Should answers be written?
Yes. Written answers are comparable and can be referenced later in the contract.
Sources
- GitHub Docs: Rate limits for the REST API
- NTIA: The Minimum Elements for a Software Bill of Materials
Last reviewed: 19 October 2026.
