Test the gate on purpose. On a schedule, run a test pipeline that includes a known vulnerable component and confirm that the release stops, the right rule fires and the event is logged. A gate that has never failed in a test is an assumption, not a control.
Why gates quietly stop working
Gates fail silently for ordinary reasons. A scanner update changes output format and the parser reads zero findings. A pipeline refactor drops the step. A policy exception meant for one release stays on. A network change makes the vulnerability data stale. In each case the pipeline stays green, and nobody notices until an auditor or an incident does.
A simple test design
- Create a small test repository or pipeline that mirrors the real gate configuration.
- Include a dependency version with a long-known, well-documented vulnerability that your policy should block.
- Run the pipeline and expect it to fail on the correct rule.
- Add a second case: a clean build that should pass.
- Add a third case: a waived finding with a valid waiver should pass, and with an expired waiver should fail.
- Check that each run writes the expected log entries.
| Test case | Expected result |
|---|---|
| Known vulnerable component | Gate fails, rule named |
| Clean build | Gate passes |
| Valid waiver | Gate passes, waiver logged |
| Expired waiver | Gate fails |
| Scanner unavailable | Fails closed, or alerts per policy |
| Data source stale | Alert raised |
How often
Run the tests on every change to the pipeline or policy, and on a fixed schedule such as weekly. Treat a test failure as a production incident for the control, since it means releases may be shipping unchecked.
Check the negative path too
It is easy to test that bad builds fail. Also test that good builds pass, or developers will learn to distrust the gate and work around it. Measure how often legitimate releases are blocked for the wrong reason.
Keep the evidence
Store the test results as evidence of control operation. Auditors who want to see that a control operated effectively over a period value dated test runs more than a policy statement. NIST SP 800-218 practice PW.8.2 speaks of scoping, designing, performing and documenting tests including recording and triaging issues, and the same discipline applies to tests of your own controls.
Watch for bypass paths
List every way code can reach production without the gate: manual deploys, admin credentials, a separate hotfix pipeline. Test those paths too, or at least know that they are logged.
Quick check
Is it safe to plant a vulnerable dependency?
In an isolated test pipeline that never deploys, yes. Never run the test case against production.
How do we know the scanner data is fresh?
Have the pipeline check the data timestamp and alert when it is older than your threshold.
Sources
Safeguard supports policies and gates in pull requests, and you can try a free scan at gold.safeguard.sh/scan.
Last reviewed: 19 October 2026.
