To evaluate an AI security claim, ask what the AI decides, what evidence shows it works on your kind of data, what it can see and do, and what happens when it is wrong. Treat a claim as untested until you have run it on your own samples and read the failure modes.
What does the claim really say?
Start by translating marketing into a testable sentence. "AI powered detection" can mean a large model, a classic classifier, rules with a new label, or a person reviewing queues. Ask for a one paragraph description of where the model sits in the workflow, what input it takes and what output it produces. Then ask which parts are deterministic code.
What evidence is credible?
Be wary of single headline numbers. A claim such as "finds 95 percent of issues" is meaningless without the test set, the definition of an issue and the false positive count. Better evidence includes:
- A description of the benchmark and how it was built, ideally with samples you can inspect.
- Results broken out by category and severity.
- False positive and false negative rates, not just detection.
- Independent testing or a proof of concept on your repositories or traffic.
Run your own bake off with a few hundred items you already know the truth for. Include hard cases that earlier tools missed.
What can the AI see and do?
NIST's AI Risk Management Framework asks organisations to map context and impact, so ask the same of a vendor. What data leaves your environment? Is it used to train models? Where is it processed and for how long is it kept? Can you run the model in your own environment or use your own model? If the product acts, not just advises, which actions are allowed and who approves them? OWASP's excessive agency guidance names excessive functionality, permissions and autonomy as root causes, which is a useful frame for any agentic feature.
What happens when it is wrong?
Every model makes mistakes. Ask how the vendor surfaces uncertainty, how you correct a bad result, whether corrections feed back, and what the rollback path is when an update makes things worse. For automated fixes or blocking actions, ask for a dry run mode and an audit trail.
What about the vendor's own security and governance?
Ask about prompt injection defences in the product itself, since tools that read your code or tickets are a target. Ask how model and prompt changes are tested before release, whether versions are pinned and how you are notified of changes. Check attestations in the normal way: read the scope and dates of any report and what it excludes, rather than accepting a logo.
A constructed example: a vendor claims its assistant "automatically fixes vulnerabilities". Your questions reveal that it opens pull requests, never merges, runs your tests first and logs each change. That is a reasonable design. If it instead pushes straight to main with write access to every repository, the same claim describes a different risk.
What to do
- Rewrite each claim as a testable sentence.
- Ask where the model sits in the workflow and what is deterministic.
- Request the benchmark description and false positive data.
- Run a proof of concept on your own labelled samples.
- Review data flows, retention, training use and deployment options.
- Review permissions and approval steps for any action the AI can take.
- Ask how updates are tested and how you can roll back.
- Record the answers so renewals can be checked against them.
Quick check
Is a vendor's accuracy figure enough to buy on?
No. Without the test set, definitions and false positive rates, the number tells you little. Test on your own data.
What is the most important question to ask?
What can the system do when it is wrong? The answer about permissions, approvals and rollback usually matters more than the average accuracy.
Sources
Last reviewed: 18 October 2026.
