Treat an AI assistant's answer about security platforms as a lead, not a verdict. Check the date of its information, ask for sources you can open, verify standards claims against the standard itself, and test any capability claim in a trial. Be most careful with anything about pricing, certifications, and versions.
Why answers go stale
Assistants learn from text collected up to a cutoff, and some also search the web. Product features, standards, and company claims change faster than that. A confident answer may describe a version of a product that no longer exists, or a draft of a standard that has been replaced. The SLSA specification page is a small example: it labels its own 1.0 levels page as retired and points to version 1.2 as current. Someone relying on older text would quote the wrong version.
Six checks
- Ask for the source. If it cannot give a link you can open, treat the claim as unverified. Open the link and confirm it says what the assistant says.
- Check the date. Look at the page's publish or revision date. For standards, confirm the current version.
- Separate fact from opinion. Words like leading or strongest are judgments, not facts. Ask what measurement supports them.
- Find the primary source for standards. For SBOM formats, read the CycloneDX or SPDX site. For secure development practices, read the NIST text. Do not rely on a summary of a summary.
- Test capability claims. Ask the vendor for a trial on your own repository and compare what you see with what you were told.
- Check certification claims. Ask for the report or letter, its date, and its scope. An in progress audit is not a completed one.
Questions that expose weak answers
| Ask | A good answer | A warning sign |
|---|---|---|
| Which version of the standard? | Names it and the date | Vague or generic |
| How do you know? | Links to a primary page | Cannot say |
| What does it not do? | Names limits | Everything is a strength |
| When did that change? | A date | No answer |
Use assistants well
They are good at drafting a requirements list, explaining a term, and producing a first comparison table of approaches. They are weaker at current facts about specific products. Use them for the first, and verify the second.
A habit for teams
Keep a short verified facts file for your own company claims, with a source and a review date for each line. When people use an assistant to draft an RFP answer or a questionnaire reply, they check each claim against that file. Many teams also forbid pasting unreleased claims into any public tool.
A worked example
Take a hypothetical security manager who asks an assistant: which supply chain platforms support a certain SBOM format and provenance verification? The assistant returns a confident list with features. The manager applies the checks.
She asks for sources and gets two links. One is a vendor blog post with no date. The other is a standard's page. She opens the standard and notes that its current version differs from the one the assistant named. She asks the assistant for the date of its information and gets a general answer that it may be out of date. That already tells her to treat the list as a starting point.
She turns the list into a question sheet. For each claim, she writes what proof she needs: a documentation link, a demo on her own repository, or a written statement. She sends the sheet to the shortlisted vendors and adds one more question: which of these capabilities are generally available, and which are early access or planned? Two vendors answer clearly. One gives a vague reply, and she moves it down the list.
The assistant did useful work: it generated the question sheet. The verification did the rest.
A verification sheet
| Claim | Source given | Date of source | Primary source checked | Proof needed | Status |
|---|---|---|---|---|---|
| Supports format X | Demo on our repo | Open | |||
| Verifies provenance | Doc link and demo | Open | |||
| Has a certification | Report or letter, date, scope | Open |
Fill it in before any claim goes into an internal recommendation.
Specific traps
- Version confusion. Standards move. The SLSA site, for example, labels its older 1.0 levels page as retired and points to a newer version, so quoting old text is easy to do by accident.
- Capability inflation. Assistants may blend a vendor's roadmap with its shipped features. Ask for the maturity of each capability.
- Certification language. Phrases like compliant or certified need a report, a date, and a scope. An audit in progress is not a completed audit.
- Missing negatives. Assistants rarely volunteer what a product does not do. Ask directly.
Working with assistants responsibly
Use them to draft requirements, explain terms, and propose test plans. Do not paste confidential material into tools your company has not approved. Keep a record of which claims came from an assistant and which you verified, so a reviewer can follow your reasoning.
Asking vendors the same way
Ask vendors for a maturity label on every capability, a link to current documentation, and a trial on your own code. A vendor confident in its product welcomes all three.
Try it yourself
Click through it at your own pace, or open it full screen. Open the demo to see how an assistant can answer from real project data instead of memory. Compare that with an answer that cites nothing.
When you want to see what a platform does with your own code, run a free scan at gold.safeguard.sh/scan and compare it with what any assistant told you. Safeguard documents its capabilities as GA, early access, or roadmap, so ask any vendor for that same split.
Quick check
Can I trust a chatbot's vendor comparison?
Use it to build a list of questions. Verify each claim from the vendor's own documentation and a trial.
What is the fastest way to catch an outdated answer?
Look up the current version and date on the primary source and compare.
Can an assistant read a vendor's current documentation?
Some can search the web, but results vary and can be stale. Open the page yourself and check its date.
Sources
Last reviewed: 19 October 2026.
