Ask vendors how a gate decides, what it does when it fails, who can override it and how every decision is recorded. The goal is to learn whether a policy is a rule you can trust or a noisy blocker developers learn to bypass. Test the answers in a real pull request.
What a gate does
A pull request gate is a check that runs when code is proposed for merging. It can block the merge, warn the author, or observe and report. A policy is the rule the gate applies, such as "no new critical findings with a fix available." Gates and policies are only useful when developers trust them.
Twelve questions for the RFP
Decision logic
- Which signals can a policy use: severity, whether a fix exists, whether the vulnerable code is reachable, exploit status, license, age?
- Can policies differ by repository, team, branch or environment?
- Can a policy distinguish new findings introduced by the pull request from existing ones?
Behavior
- Can each policy be set to block, warn or observe, and changed without redeploying?
- What happens if the check cannot complete, for example during an outage? Does it fail open or closed, and can we choose?
- How long does the check take on a repository of our size, and how is that reported?
Overrides and exceptions
- Who can override a block, how is the reason captured, and can we require a second approver?
- Can exceptions expire automatically and be reviewed on a schedule?
Evidence
- Is every gate decision, override and exception stored with who, when and why?
- Can we export that history for an audit?
Rollout
- Can we run a policy in observe mode first and see what it would have blocked?
- How do developers see the reason for a block, and the steps to fix it, inside the review?
What good answers look like
| Topic | Strong answer | Weak answer |
|---|---|---|
| New versus existing | Separate counts, with a baseline | Everything flagged on every run |
| Outage behavior | Configurable, documented | "It won't fail" |
| Overrides | Reason required, logged, reviewable | Admin toggle with no record |
| Rollout | Observe mode with a report | Switch on and see |
Test it on a real change
Open a pull request that adds a dependency with a known vulnerability, then check:
- Does the check appear in the review within a reasonable time?
- Is the message clear enough that a developer can act without asking security?
- Does an override leave a record with a reason?
- Does observe mode show what would have been blocked?
A worked example: one policy, tested
Consider the policy: "Block merges that add a new critical finding with a fix available. Warn on everything else." Here is how a team might test it.
| Test case | Expected result | What to record |
|---|---|---|
| Pull request adds a dependency with a known critical vulnerability and a fixed version | Check blocks, message names the package and the fixed version | Time to result, clarity of message |
| Pull request adds a dependency with a critical vulnerability and no fix | Check warns, does not block | Whether the rule distinguishes fix availability |
| Pull request changes only documentation | Check passes quickly | Time and any needless blocking |
| Existing critical finding in untouched code | Not blocked, shown as existing | Whether new and existing are separated |
| Override by a developer | Requires a reason, logs who and when | Record and notification |
| Override by a second approver requirement | Cannot merge without second approval | Whether it is enforced |
| Vendor service unavailable | Behaves as configured, fail open or closed | Documentation and actual behavior |
Seven small pull requests, a couple of hours, and you will know more than any questionnaire could tell you.
Rollout in three stages
- Observe. The gate runs and reports but never blocks. Collect a week of data on what it would have blocked.
- Warn. Developers see messages. Collect feedback on clarity and false alarms.
- Block on a narrow rule. Start with the clearest rule, such as new critical findings with a fix. Expand once trust has grown.
Ask each vendor how their product supports every stage and how switching between stages is recorded.
Metrics to track in the first month
| Metric | Why it matters |
|---|---|
| Check duration | Slow checks erode trust |
| Block rate | Too many blocks suggests noisy rules |
| Override rate | A high rate suggests the rule is wrong or unclear |
| Time from block to fix | Shows whether messages help |
| Exceptions past expiry | Shows whether the review process works |
Questions about policy as code
Some teams want policies in version control. Ask whether policies can be defined in files, reviewed in pull requests and applied automatically, and whether the history of changes is kept. Ask what happens when a policy file and the interface disagree.
Questions about agents and automation
If automated tools, including AI coding assistants, open pull requests in your repositories, ask whether the gate treats those pull requests the same way and whether the origin of a change is recorded. Ask what controls limit the actions automated identities can take.
Try it yourself
Click through it at your own pace, or open it full screen. Watch what the gate reports and where the override record would appear, then compare with what each vendor shows you.
Quick check
Should a new gate block merges on day one?
Usually not. Start in observe mode, review what it would have blocked, then tighten.
Why ask about outage behavior?
A gate that fails closed can stall every merge, and one that fails open can silently skip checks. You should choose and know which.
How do we stop developers from learning to ignore the gate?
Keep rules narrow, measure override and block rates, and fix noisy rules fast. A gate that cries wolf trains people to bypass it.
Safeguard's GA capabilities include policies, gates and guardrails, with autonomous remediation and fix PRs off by default and every category on Manual. Ask it the same twelve questions and check the answers on a repo of your own with the free scan at https://gold.safeguard.sh/scan.
Last reviewed: 19 October 2026.
