The main red flags are vague answers about where code goes, safety that exists only as prompt wording, no administrator controls, no exportable logs, no way to limit commands or network access, and pressure to skip a trial. Any one deserves a follow up; several together mean pause.
Ten signs worth a pause
- Vague data answers. "We take privacy seriously" is not a retention statement. Ask for a written description of what is sent, stored, and for how long.
- Safety lives in the prompt. If the tool's protection is an instruction to the model, an attacker or a model error can override it. Look for controls enforced in code or the operating system.
- No central administration. If every developer's local settings decide the policy, your policy is a suggestion.
- No exportable logs. Without records of actions, you cannot investigate or prove a control worked.
- All or nothing permissions. A tool that offers only "ask every time" or "never ask" will push people toward "never ask".
- Honors any project config without a trust step. The MCP security guidance warns about malicious startup commands in client configuration. A tool that runs whatever a cloned repository says deserves scrutiny.
- No way to scope tool servers. Connected servers should be listable, restrictable, and reviewable.
- Opaque updates. Silent changes to defaults make last quarter's review meaningless. Ask how changes are announced.
- Pressure to skip the trial. A vendor that resists a test on a throwaway repository is telling you something.
- Claims that sound absolute. Words like "fully secure" or "can never leak" are not evidence. Real security statements name scope and limits.
How to test for each
| Red flag | Quick test |
|---|---|
| Prompt-only safety | Plant an instruction in a file telling the agent to ignore its rules, and see if the tool blocks the action itself |
| No admin control | Try to enforce a setting for a test group |
| No logs | Run a task and attempt to export what happened |
| Trusts project config | Clone a repository with a harmless startup command in its settings and see if the tool asks first |
| Opaque updates | Read the changelog for the last three versions, looking for permission or default changes |
Fair play for vendors
A red flag is a reason to ask, not proof of bad faith. Some gaps close with compensating controls: a container for isolation, a proxy for logging, a wrapper for network limits. Write the compensating control into the decision. Equally, a polished answer is not proof of a good product, so test.
Weigh by your risk
A team working on public open source code can tolerate more than a team handling restricted data. Decide ahead of time which flags are blockers for your environment and which are conditions. Our post on an evaluation scorecard shows how to turn this into gates and weights.
What good looks like
Good answers are specific, dated, and testable. They come with documentation, a named contact, and a willingness to let you try the claim. When a vendor answers a hard question with a clear limit ("we do not do X, here is how customers cover it"), that candor is a positive sign.
The OWASP Top 10 for LLM Applications is a useful shared vocabulary for these conversations, especially Prompt Injection, Excessive Agency, and Supply Chain.
Safeguard can help at the dependency and pipeline end of this: SBOM and dependency scanning, reachability analysis, policies and gates on pull requests, and agent identity. You can try a free scan at gold.safeguard.sh/scan.
A worked example: reading a vendor response
A vendor answers a written question about data handling. Compare two answers.
| Question | Answer one | Answer two |
|---|---|---|
| Where do prompts and files go? | "We take privacy very seriously and use industry standard protection." | "Prompts and the files you include are sent to our processing service in the regions listed on our documentation page. Retention is configurable per organization. Training use is off by default for business accounts, and the setting is visible to administrators." |
| Can we disable it for a team? | "Yes, contact support." | "Administrators can disable the tool for a group in the console. Changes take effect at the next session." |
| What do logs show? | "Comprehensive logging." | "Per action records with user, time, tool, and result, exportable in a standard format. Prompts are not included by default." |
Answer one has three red flags: no specifics, no testable claim, no named control. Answer two is specific, scoped, and checkable. You still verify it by test, but it gives you something to test. The shape of an answer is a quiet signal about the product behind it.
A short severity guide
| Flag | Severity for sensitive code | Common compensating control |
|---|---|---|
| Vague data answers | High | Written follow up, contract terms |
| Prompt-only safety | High | Container, network limits |
| No admin controls | High | Wrapper with central configuration |
| No logs | High | Proxy and wrapper capture |
| Trusts project config silently | High | Sandbox, trust prompt, review of configs |
| Opaque updates | Medium | Version pinning, change review |
| All or nothing permissions | Medium | External allowlist |
| Pressure to skip the trial | Medium | Hold the line, or walk away |
Questions that surface flags early
- What happens to a prompt after it is sent? Walk me through it.
- Which settings are enforced by your software and which are suggestions to the model?
- Show me how an administrator limits what the tool can run.
- What does your log record, and how do I export it?
- What do you recommend customers do about prompt injection, and what is out of scope for you?
Honest vendors answer question five with limits. Treat candor as a positive signal.
Handling a flagged tool you still want
Sometimes the product is good and the flags are fixable. Write a short exception: the flag, the compensating control, the owner, and the review date. For example, "No central logging, compensated by a proxy capturing hosts and commands, owned by platform, review in ninety days." That sentence is easy to audit later.
When to walk away
Walk away when a flag cannot be compensated and the data is sensitive, when the vendor will not allow a trial, or when answers change between conversations. Time spent on a tool that cannot meet your needs is better spent testing one that can.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through and notice what a dependency inventory shows about packages a tool adds.
Quick check
Is one red flag a deal breaker?
Not always. Ask whether a compensating control closes it, and record the exception with an owner and a date.
Should we skip tools without logs?
For anything touching sensitive code, yes, unless you can add logging through a wrapper or proxy.
Can a good demo hide these flags?
Yes. Demos show the happy path. Plant a hostile instruction, request an export, and enforce a setting, and the flags tend to appear.
Sources
Last reviewed: 19 October 2026.
