Test the tool inside a real pipeline on a test branch, and measure four things: run time, how it fails a build, how clear the message is, and what it does when the vulnerability data is unreachable. A tool that is accurate but slow or noisy in CI will be turned off within a month.
Run time
Time a scan on your largest repository, cold and with caching. A dependency scan that reads lockfiles should be quick. If it needs to build the project, expect longer. Compare the added minutes to your current pipeline budget. Developers notice anything over a couple of minutes on a pull request.
How it decides to fail a build
Ask what controls the exit code. You want policy you can set, not a fixed behavior. Useful dials include:
- Severity thresholds
- Only new findings introduced by this change
- Exploit signals such as known exploitation
- Whether the finding is reachable
- A time window for known issues you have accepted
Failing a build on every old finding blocks everyone for reasons unrelated to their change. Failing only on new ones keeps the rule fair.
Messages developers can act on
Read a failing run as a developer would. Does it name the package, the version, the path that pulled it in, and the fixed version? Does it link to the advisory? A bare list of identifiers creates support tickets.
Pull request experience
Check whether results appear as a comment or status on the pull request. Check whether it posts the same finding repeatedly on every push. Duplicate comments are a common reason teams mute a bot.
Failure modes of the tool itself
Break things on purpose:
- Block network access to the vulnerability data. Does the build hang, fail closed, or pass with a warning?
- Scan a repository with no lockfile.
- Scan a repository with two lockfiles.
- Rotate or remove the token.
Decide which behaviors you can live with. Failing closed on an outage may be right for a regulated pipeline and wrong for a busy team, so make sure it is configurable.
Secrets and permissions
Check what access the integration needs. Read only access to repositories is a better starting point than write. Document what the tool sends out of your network, and confirm that against your security review.
Local and IDE parity
If developers cannot run the same check locally, they find out about issues late. Ask whether the CLI and editor extensions use the same rules as the pipeline.
Safeguard has a CLI (curl and PowerShell installers) and policy gates for pull requests, so you can try the same rules locally and in CI. The free scan at https://gold.safeguard.sh/scan is a quick first look.
Click through it at your own pace, or open it full screen.
Quick check
Should SCA block every pull request with a finding?
Usually not. Block on new, serious, relevant findings and report the rest.
Last reviewed: 19 October 2026.
