Replay five to ten real past pull requests through each candidate's gate in report only mode, with the same rules. Record which ones each tool would have blocked, warned on or passed, and read the messages as a developer would. Compare decisions, not features lists.
Why compare gates on real changes
A gate is where a scanner meets your developers every day. Feature lists say "blocks risky pull requests," but that hides the details: what counts as new, how fast it responds, and how clear the message is. The only way to see this is to run each gate on changes your team actually made.
Gates also carry risk. A gate that blocks the wrong changes hurts delivery, and one that never blocks adds nothing. See evaluating pull request gates for the modes.
Pick the pull requests
Choose a mix from your history, such as:
- A pull request that added a new dependency.
- One that bumped a vulnerable version to a fixed one.
- One that bumped a safe package to another safe version.
- One that changed nothing in dependencies, such as a docs edit.
- One that added a vulnerable dependency, if you can find or construct it on a branch.
Include at least one from each major ecosystem. You are testing both sides: does it catch what it should, and does it stay silent when nothing changed?
Fix the rules first
Write one policy in plain words and apply the closest equivalent in each tool. For example:
| Rule | Plain statement |
|---|---|
| Block | New dependency with a critical advisory that has a fix available |
| Warn | New dependency with a high advisory |
| Pass | Existing findings not introduced by this change |
| Pass | Changes without dependency edits |
If a tool cannot express a rule, record the difference. The matching exercise is similar to the parity checklist.
Run in report mode
Configure each gate to report without blocking, so nothing in your real pipeline changes. Where a tool offers a dry run or observe mode, use it. Re-open or replay each pull request on a throwaway branch so the tool receives it fresh.
Record per pull request and per tool:
| Field | Note |
|---|---|
| Decision | Block, warn or pass |
| Time to decision | Seconds or minutes |
| New findings only? | Did it separate new from old? |
| Message clarity | Rate with a developer's help |
| Fix hint | Version or action suggested |
| Comment volume | Number of comments posted |
The last field matters because chatty bots tire people out. See limiting scanner comments on pull requests.
Compare against the expected answer
Before running, write what you would want to happen for each pull request. Then compare. Count agreements and disagreements per tool. A tool that blocks an unrelated docs change fails even if it catches the vulnerable one.
Show the gate's message to two developers without context and ask what they would do next. If both answer correctly in under a minute, the message works.
Other behaviors to test
- What happens on a re-run of the same pull request: same result?
- What does the gate do when the vulnerability data is down? That is covered in fail open or fail closed.
- Can a developer request an exception, and who approves it?
Try it yourself
Click through it at your own pace, or open it full screen. Watch what the gate shows for a clean change and for one that adds a risky package.
Safeguard's policies, gates and guardrails can run in observe mode first, and the free scan at gold.safeguard.sh/scan is a quick look at the findings a gate would act on.
Quick check
Should the gate block during the trial?
No. Run in report mode until you trust the decisions, then enable blocking on one repository.
How many pull requests is enough?
Five to ten, chosen to cover the cases above. More only adds review time.
Last reviewed: 19 October 2026.
