Claims about AI security coverage are often broader than the product behind them. The fix is not distrust, it is translation: turn each claim into a test you can run. Below are five claims buyers hear often, why each is incomplete on its own, and the question that makes it checkable.
Myth 1: "We cover AI because we use AI"
Fact: using machine learning inside a tool says how it finds things, not what it can see. A scanner with an AI engine and no model file detection still will not list your models.
Ask: "Which component types appear in the inventory? Show me one of each."
Myth 2: "We scan models"
Fact: scanning a model can mean several things. It might mean detecting that a file exists, checking its format for code execution risk, recording its source, or evaluating its behavior. These are different jobs. OWASP's LLM Top 10 lists supply chain and data and model poisoning as separate risks, which hints at how many questions sit inside the word "scan".
Ask: "Which of these do you do: detect, check format, record provenance, test behavior? Show each on my sample."
Myth 3: "We cover MCP"
Fact: coverage can stop at the package name of the server. The protocol's tools specification says tools can change and that clients must treat annotations as untrusted unless the server is trusted. Real coverage includes which tools a server exposes and when they change.
Ask: "Install this test server. Show me its tools, then change one and show me the alert."
Myth 4: "We stop prompt injection"
Fact: the OWASP list puts prompt injection first, and it is a risk class rather than a bug you patch once. Tooling can reduce exposure by limiting what an agent can reach, but a claim to stop it completely deserves a request for the test method.
Ask: "What is the test set, how often is it updated, and what is the miss rate on it?" If they cannot answer the last part, treat the claim as marketing.
Myth 5: "We are compliant, so you are"
Fact: a vendor's report covers the vendor. It does not make your use of the product compliant. See whether a vendor's SOC 2 report covers its AI features.
Ask: "Which systems and features are in scope of the report, and which are not?"
Turn claims into a ledger
| Claim | Test | Result | Evidence file |
|---|---|---|---|
| Models covered | Seeded repo with model file | Found or missed | Export |
| MCP covered | Test server, change one tool | Alert or none | Screenshot and log |
| Prompt injection | Ask for test method | Documented or not | Document |
Keep the ledger shared among evaluators. See how to keep a vendor claims ledger during an evaluation for the method.
What a candid answer sounds like
Candid vendors name what they do not cover. That sentence is a good sign. A vendor that claims full coverage of a fast moving area is either very new to it or not being precise.
Safeguard lists what it supports on its site, and that list is a fair thing to hold us to. We offer SBOM and dependency scanning, reachability analysis, MCP Guardrail, agent identity and asset discovery. You can run the free scan at gold.safeguard.sh/scan and judge what you see.
Quick check
Should we ignore any vendor that makes a broad claim?
No. Treat each broad claim as a hypothesis, write the test, and score the result. Many vendors pass when the question is specific.
How many claims should we test?
Pick the five that matter most to your risk. Testing everything dilutes the effort.
Sources
Last reviewed: 19 October 2026.
