Before signing with a guardrails vendor for AI agents and software delivery, ask where its rules are enforced, what happens when it fails, how you write and test policy, what evidence it keeps, and where your data goes. Insist on seeing each answer demonstrated on your own systems.
Guardrails here means rules that allow, hold or block actions in the path of work: a release, a package pull, an agent's tool call. The questions below are grouped by what can go wrong.
Enforcement
1. Where exactly does enforcement happen? In the build, in CI, at a registry, at a proxy in front of tool servers, or in the agent runtime? Anything outside the enforcement point is not covered. Ask for a diagram of your own setup, not a generic one.
2. What can bypass it? A developer who builds locally, an agent calling a tool directly, a second pipeline nobody told the vendor about. A candid vendor will list some.
3. What happens when the product fails or is slow? Does work stop (fail closed) or continue (fail open)? Ask whether you can choose per rule. Ask for the added latency measured in your environment, since a figure from marketing says little.
Policy
4. How do we write and change rules? Code, a UI, or both? Can rules live in version control with review? A policy that cannot be reviewed cannot be audited.
5. Can we run a rule in report mode first? You want to see what would be blocked before turning enforcement on.
6. How do exceptions work? Each should have an approver, a reason and an expiry. Ask to see the register.
7. How does it decide what is serious? Severity alone creates noise. Ask whether it uses reachability, exploitation evidence from public sources such as CISA's Known Exploited Vulnerabilities Catalog, and your own asset importance. Reachability tells you if the vulnerable code is actually used.
AI agent specifics
8. How does it identify agents and their access? Can it tell which agent called a tool, who owns it and what level of access it holds? OWASP's LLM06, Excessive Agency, in the 2025 Top 10 for LLM Applications, describes the risk of an LLM system having more functionality, permissions or autonomy than it needs. Ask how the product limits each.
9. Can it hold high impact actions for human approval? And how are approvals recorded?
Evidence and data
10. What records does it keep, and for how long? Blocked events, overrides, approvals and policy changes should be exportable. NIST SP 800-53 control CM-8 expects an accurate component inventory, so ask whether the product maintains one and how it reconciles with yours.
11. Where do our code, prompts and logs go? Ask where they are stored, who at the vendor can read them, whether they train any model, and whether you can keep them in your environment. Read the attestations the vendor shows you for their scope, and do not accept a logo as proof.
Exit and cost
12. What do we get at exit, and what are the commercial terms? Ask for policy export in a usable format, and for term, renewal, overage and scope rules in writing. Counsel should review the legal text.
How to run the evaluation
Send questions 10 to 12 in writing first. Watch questions 1 to 9 demonstrated live on a copy of one real pipeline and one real agent. Plant harmless policy violations, such as a dependency your policy forbids, and an agent tool call outside its list, then check whether each is stopped, logged and explained.
Score each question as proven, claimed or evaded. Weigh failures in questions 2 and 3 heavily, since a guardrail that quietly fails open provides false comfort.
What to do
- Draw your own enforcement points before talking to vendors.
- Send the data, retention and exit questions in writing.
- Test failure behavior and bypass paths in a live trial.
- Run rules in report mode before enforcement.
- Verify exceptions have approvers and expiry dates.
- Check attestation scope, and confirm export of policy and evidence.
Where Safeguard fits
Safeguard provides policies, gates and guardrails at build, CI and release, an MCP Guardrail proxy for MCP servers, and agent identity tracking, all generally available. Use these same questions on Safeguard and on any other option you consider.
Quick check
Which question is the most revealing?
What happens when the product fails. A clear, tested answer shows operational maturity, while a vague one is a warning.
Should the first rule block immediately?
No. Run it in report mode, review what it would have blocked, then enforce.
Sources
- OWASP Top 10 for LLM Applications 2025
- NIST SP 800-53 Rev. 5, control CM-8 (OSCAL catalog)
- CISA Known Exploited Vulnerabilities Catalog
Last reviewed: 18 October 2026.
