Measure a scanner's effect on your pipeline before buying: added minutes per run, failure rate unrelated to code, behaviour with caches, and what happens when the scanner or its service is unreachable. Developers judge a security tool by how it changes their feedback loop, and slow or flaky checks get bypassed.
Decide your budget first
Pick a number before testing. For example, a pull request check should add no more than a few minutes, and the main branch scan can take longer. Without a budget, every result sounds acceptable. Involve the developers who wait on this pipeline when you choose it.
What to measure
- Added time on a pull request. Run ten typical pull requests with and without the scanner. Record median and slowest.
- Cold versus warm. First run, run with a cache, and run after the cache expires. Many tools look fast only when warm.
- Scaling. Try your largest repository and your widest monorepo, not only a sample project.
- Parallel load. Trigger many pipelines at once. Rate limits and shared queues show up here.
- Flakiness. Count failures not caused by a code problem: timeouts, network errors, authentication expiry. Even a small rate is costly across hundreds of runs.
- Output quality in context. Are results posted where developers read them, such as the pull request, or in a separate portal nobody opens?
Test the failure modes
This is the part most trials skip.
- What happens if the scanner service is down? Does the build fail, pass or hang? See the block or pass question.
- What if advisory data fails to update?
- What if a scan exceeds its time limit?
- Can an engineer bypass the check in an emergency, and is the bypass logged?
Write down the intended behaviour for each, then verify it in the trial.
Blocking policy during the trial
Run in report mode first. Then enable blocking on one volunteer team for new critical findings only. Gather feedback weekly: what blocked, whether it was right, how long it took to resolve. Count false blocks, because each one costs trust.
Security of the integration itself
The scanner runs with access to your code and often your secrets. Check what permissions it needs, how tokens are scoped and whether it runs in an isolated job. See the pipeline hardening checklist.
Reading the results
A scanner that adds little time, rarely fails for the wrong reason, and puts clear results in front of developers wins on experience. A scanner with better detection but a bad pipeline fit may lose to a simpler one that people actually keep switched on.
Where Safeguard fits: policies, gates and guardrails let you choose what blocks a pull request, and you can try the scanner's output with the free scan.
Try it yourself
Click through it at your own pace, or open it full screen.
Watch where the result is shown to the developer and what a failing check says.
Quick check
Should security scans run on every commit?
Run fast checks on pull requests and deeper scans on merge or on a schedule. Match depth to the feedback time developers will tolerate.
Last reviewed: 19 October 2026.
