Trial AI agent and delivery guardrails by running them in report mode on one team, replaying constructed test cases, and enforcing only in the second week. Score what was blocked correctly, what was blocked wrongly, how developers reacted and how fast exceptions were handled.
What are you testing?
Guardrails act at several points: in the IDE, in pull requests and CI, at release, and in front of the tools an agent can call. A trial should pick the points that matter most for you and test those. Define in advance what a good result is, for instance: known bad changes are caught, normal work is not delayed noticeably, and every block explains itself.
Who should take part?
One team of three to eight engineers who use AI coding tools or agents, plus a security owner and someone who can change policy during the trial. Choose a team that ships often, so you get evidence quickly, and tell them the trial is a test of the tool, not of them.
Week one: report mode
Day 1: scope and rules. Pick five to eight rules. Examples: block a dependency with a known exploited flaw, require review for changes to the pipeline file, stop an agent from calling a tool outside an allowed list, flag a secret in a commit, require a signed release. Write each in plain words.
Days 2 to 5: observe. Set every rule to report only. Developers work as usual. At the end of each day, list what would have been blocked and review a sample with the team. Mark each as correct, wrong or unclear.
Meanwhile, prepare constructed test cases in a test repository with approval. Examples, all harmless: a pull request adding a package version with a known flaw, a test file with a fake key that matches a secret pattern, an agent configured in a sandbox to call a tool that is not on its list. None should touch production.
Week two: enforce and measure
Day 8: enable enforcement for the rules whose report mode results were mostly correct. Leave the doubtful ones in report mode.
Days 8 to 12: measure the following.
| Measure | How to count |
|---|---|
| Correct blocks | Blocks the team agrees were right |
| Wrong blocks | Blocks overturned |
| Delay | Extra minutes on affected pull requests |
| Explanation | Did the block say what, why and what to do? |
| Exception time | Time from waiver request to decision |
| Agent limits | Did the sandbox agent get stopped, and was it logged? |
Day 13: team survey. Ask three questions: what annoyed you, what helped, what would make you bypass it.
Day 14: decide. Compare against your criteria. A tool that blocks correctly but explains badly is fixable. A tool with no waiver path is not.
Try it yourself
What are the common traps?
- Enforcing everything on day one, which turns the team against it.
- Using only synthetic tests and ignoring real traffic.
- Skipping the exception path.
- Counting blocks as success.
- Testing on a quiet team that produces too few events.
NIST's SSDF, SP 800-218, treats these practices as part of the development life cycle, so judge the trial by how it fits daily work.
What should the exit criteria say?
Write them before you start. For example: at least most of the test cases are caught, wrong blocks are fewer than correct ones, every block explains the reason, and a waiver is decided within a working day. Add a hard stop: if developers need to bypass the tool to ship, pause the trial and fix the cause. Also record what you would need before a wider rollout, such as single sign on, audit export or more rule types. That list becomes your contract and rollout plan.
Where Safeguard fits
Safeguard supports policies, gates and guardrails enforceable at build, CI and release, and an MCP Guardrail that is a proxy for MCP servers. Tool calls through its MCP server are unlimited on every plan. Autonomous remediation is off by default with categories starting on Manual, so the trial can leave it off. You can run this plan with your own repositories.
What to do
- Pick one team and five to eight rules written in plain words.
- Run all rules in report mode for the first week and review samples.
- Prepare harmless test cases in a test repository.
- Enforce only the rules that proved accurate.
- Measure correct blocks, wrong blocks, delay and exception time.
- Survey the team and decide against criteria set on day one.
Quick check
Why start in report mode?
It shows what would be blocked without disrupting work, so you can fix noisy rules before enforcing them.
What is the most important trial result?
Whether correct blocks outnumber wrong ones and whether developers can get a fair, fast exception when a block is wrong.
Sources
- NIST SP 800-218, Secure Software Development Framework
- OWASP Top 10 for LLM Applications, 2025
- Model Context Protocol authorization specification
Last reviewed: 18 October 2026.
