During a security tool trial, ask how the vendor handles outages, upgrades and degraded service, and observe the tool's behaviour when the service fails. The key question is simple: if the vendor is down, do your builds stop, pass without checks or queue? Know the answer before you rely on it.
Why this matters more for security tools
A security check in your pipeline is a dependency. If it is unavailable and set to block, it can halt releases. If it is set to pass on failure, you may ship unscanned code without noticing. Neither is wrong, but each must be a deliberate choice.
Questions to ask the vendor
- Is there a public status page with incident history? How far back does it go?
- What do you publish after an incident, such as a post incident summary?
- How are upgrades delivered, and how much notice do customers get?
- Can upgrades be paused or scheduled in our environment, if the tool runs there?
- What happens to queued scans during an outage?
- Are there stated service targets for my plan, and what do they cover?
- How do you notify customers of an incident?
Write the answers in your kickoff notes. Check that claims are backed by public pages rather than only by conversation.
Observe during the trial
You cannot cause a real outage, but you can watch and test how the tool fails.
| Test | What to look for |
|---|---|
| Block the tool's network access from CI | Does the build fail, pass or hang, and is the message clear? |
| Revoke the token | Is the error actionable? |
| Watch the status page | Were any incidents reported during your window, and how were they handled? |
| Note any upgrades | Did anything change without notice? |
| Time scans at different hours | Is performance steady? |
Record the results in the evidence log.
Decide your failure policy
Before you buy, choose what the tool should do when it cannot run. A common approach:
- Warn only for new tools: pass on failure with an alert while you learn the pattern.
- Block on critical repositories: fail closed where the risk is high, with a documented override.
- Always alert: whichever choice you make, a failed check must produce a visible alert.
Check that the tool supports the policy you want and that the override path is audited. See evaluating pull request gates.
Upgrade questions for self hosted components
If any part runs in your environment, ask about upgrade frequency, breaking changes, rollback and how long each version is supported. Ask who performs upgrades and how much staff time it takes. See support periods and end of life dates.
Ask about the vendor's own resilience
Ask how the vendor protects its own build and release process, since a compromised security tool is a serious risk. See the blast radius if your appsec vendor is compromised.
Keep expectations realistic
No service is available all the time. A good answer is not a promise of perfection. It is a clear description of what the vendor does, what it tells you and what you can do while it recovers.
Safeguard's pull request gates can be run in warn or block modes, which lets you choose the failure policy to match your risk. A free scan is a way to see output with no pipeline dependency at all.
Quick check
Should a security check fail open or closed?
It depends on risk. Many teams start fail open with alerts, then fail closed on critical systems with an audited override.
Can I test an outage during a trial?
You can simulate one by blocking network access or revoking a token in a test pipeline and observing the behaviour.
Last reviewed: 19 October 2026.
