When a security vendor advertises AI, ask what the AI does, which data it sees, how accuracy was measured, who reviews its output, and what happens to your bill and your data at scale. Treat every answer as a claim to test in the trial.
What does it actually do
- Which features use a model: triage, explanation, fix suggestions, search, report writing? List each one.
- For each feature, what is the non-AI fallback? If the model is unavailable, does the product still work?
- Is the AI feature generally available today, in beta, or planned? Score only what is available.
Vendors sometimes group three small helpers under one "AI engine" label. A list by feature makes the real scope visible.
Data and models
- Which models power each feature, and are they third party, self hosted, or built in house?
- What data from our environment is sent to the model: code, findings, metadata?
- Is our data retained by the model provider or used for training?
- Can we turn AI features off for the whole organization or per project?
Pair these with the questions in data handling and deployment.
Accuracy and review
- How did you measure accuracy? Ask for the test set description, not a headline percentage.
- What does the tool do when it is unsure? Does it say so, or does it produce a confident answer anyway?
- Does the AI take any action on its own, such as opening pull requests or changing finding status? Can all actions be set to require approval?
For question 10, the safest default is that automated changes need a human to approve them. A vendor who ships with automation on for every category deserves a harder look at their controls.
Cost and scale
- Is AI usage included in the license, metered, or capped? How do we predict the bill?
- What happens at the cap: do features stop, slow down, or overage charge?
Ask for a written example for your repository count. Do not accept "it depends" without the factors that it depends on.
How to test in the trial
Choose five findings where you know the correct fix. Ask the AI feature to explain and propose a fix. Review each by hand and record whether the fix compiles, passes tests, and solves the actual problem. Then try one adversarial case: a finding that looks risky but is not reachable, and see if the tool notices.
Track the time reviewers spend. An AI suggestion that takes longer to verify than to write is not saving anything.
Red flags
- Accuracy numbers with no description of the test set
- No way to see which model produced an output
- No audit trail of AI initiated actions
- "Our AI replaces your security team" language, which usually signals a gap in the controls
Safeguard offers autonomous remediation with fix pull requests, which is off by default with every category set to manual approval. Ask every vendor for the equivalent control documentation.
Quick check
Is a vendor's accuracy percentage reliable?
Only if you know the test set and method. Ask for both, then run your own sample in the trial.
Should AI features be on by default?
For anything that changes code or finding status, a manual approval default is safer. Verify that you can control it.
Last reviewed: 19 October 2026.
