Ask an AI code security vendor twelve things: where your code goes, whether a no model mode exists, which models run, how accuracy is measured, how noise is cut, who approves fixes, what is logged, how evidence is exported, what certifications are real, what the limits are, how pricing scales and how you leave. Get answers in writing.
Data and models
1. Where does our source code go, and for how long? A good answer comes with a diagram: what leaves your network, who receives it, how long it is kept and how it is deleted. Vague phrases such as "secure cloud" are not an answer.
2. Is there a mode with no model calls? Some teams cannot send code to a model. A good vendor can say which features work without one and which do not, and does not pretend the results are identical.
3. Which models run, and can we choose? Ask whether models are fixed or configurable, whether you can bring your own, and whether on premises or local models are supported. Your own model risk policy may require a named, approved model.
4. Is our code used to train anything? Ask for the contract language, not a marketing line.
Quality of findings
5. How do you measure accuracy, and on what data? A good vendor can describe a test set and its limits. Be careful with a single headline number. Run your own repository in a trial and compare.
6. How do you reduce noise? Ask about reachability, exploitability signals and deduplication. Ask what happens with cases the analysis cannot decide. A conservative answer, such as treating unknowns as potentially reachable, is safer than hiding them.
7. What does it miss? Every method has blind spots, for example logic flaws or code reached through reflection. A vendor that cannot name any is not being straight.
Fixes and control
8. Who approves a change? If the tool proposes or applies fixes, ask whether a person must approve, whether changes land as pull requests, and whether you can turn automatic changes off.
9. What does the tool log? You want a record of scans, findings, actions and the identity behind each. Ask if it exports to your own store.
Evidence and trust
10. Which claims are certifications and which are plans? Ask for the certificate or report, its scope and its date. Listen for words like "aligned" or "designed to meet." Those describe intent. NIST's SP 800-218A, the secure development profile for generative AI, is a useful reference for what secure practice looks like for AI systems.
11. How does it map to our frameworks? Ask which controls the output supports, which are decided automatically and which still need your own evidence. A tool can support evidence. It does not make you compliant.
Commercial
12. What does leaving look like? Ask how to export findings, SBOMs and history in open formats, and what happens to your data after you stop.
How should you score the answers?
Make a simple table in your own notes. For each question, mark the vendor's answer as verified in a demo, stated in writing, stated verbally or unclear. Weight data handling and the no model option most heavily if you are regulated. A single unclear answer on question 1 is worth pausing for.
What to do in the trial
- Send the twelve questions before the first call.
- Run the product on a repository you know well.
- Watch network traffic during a scan and compare with the diagram from question 1.
- Export logs and read them as an auditor would.
- Ask for the documented limits again at the end.
How does Safeguard answer?
We answer only what we can verify. Safeguard supports Full AI, Zero AI, or bring your own model, so you choose whether and how models are used. Its CLI includes a local runner mode that pulls the repo on your machine, scans locally and pushes only encrypted results. Its reachability analysis classes findings as reachable, conditionally reachable, unreachable or unknown, with unknown treated as potentially reachable. Safeguard's documentation says it does not hold a FedRAMP authorization. We make no other certification claims in this post, and you should ask us these same twelve questions.
Quick check
What is the most important question?
Where your source code goes. It determines what you can approve, what regulators will ask and which other answers matter.
Should we trust accuracy numbers from a vendor?
Treat them as claims to test. Ask how they were measured, then run the tool on your own repository.
Sources
- NIST SP 800-218A: SSDF Community Profile for generative AI
- Safeguard docs: reachability analysis
- Safeguard docs: compliance
Last reviewed: 18 October 2026.
