Yes, and the gap can be large. A first run may download databases, resolve packages and warm caches, while later runs reuse them. To compare speed fairly, time every tool cold and warm, on the same machine, several times each, and report ranges rather than one number.
What affects scanner time
Several things sit behind the stopwatch, and none of them is raw detection speed.
- Vulnerability data. A tool may download or update its database on first use, or query a service each time.
- Package resolution. Resolving transitive dependencies can hit a registry over the network.
- Caches. Previously fetched data is reused on a second run.
- Machine load. A busy laptop or shared CI runner adds noise.
- Network conditions. Remote lookups vary through the day.
OSV-Scanner's documentation, for example, describes an offline mode that uses locally cached databases, which shows that where the data lives changes what a run involves. See the offline mode notes. Other tools differ in their own ways, so read each one's documentation first.
Procedure
- Use one machine, or one CI runner type, for all candidates. Close other heavy work.
- Use the same frozen repository. Pick one small and one large from your set.
- For each tool, do a cold run. Clear that tool's cache or use a fresh container so nothing is stored. Record wall clock time.
- Do a warm run immediately after, with the cache in place. Record the time.
- Repeat five times, alternating tools, and note the date and hour.
- Record the commands used, as in the configuration parity list.
| Tool | Repo | Cold min to max | Warm min to max | Notes |
|---|---|---|---|---|
| A | small | range | range | network lookups |
| B | small | range | range | local database |
Measure the whole job, not just the scan
For a pipeline, the relevant time includes install, database update, scan, and upload of results. Time the steps separately. A tool that scans in seconds but needs a long database download on a fresh CI runner may be slower in practice than one with a slower scan and a bundled database. How runs fit into a pipeline is covered in scanner CI speed and blocking test plan.
Check where hosted tools do the work too. If analysis happens remotely, your wall clock includes queue time. Note whether that queue time varied.
Report honestly
- Give ranges. Averages hide the slow outliers developers remember.
- State machine, network and date. Timings do not transfer to other environments.
- Do not extrapolate from one repository to your largest monorepo. Time that too.
- Mention memory use if a tool strains the runner.
What to do with the results
Speed is one criterion among several. Decide in advance how much a minute of pipeline time is worth to your team. A pull request check that adds a few minutes is felt every day, while a nightly scan can take longer without anyone noticing. Match the measurement to where the tool will run.
The question of behavior when a tool is slow or unavailable is a separate one, handled in the post on scanner outages in CI.
Safeguard has a CLI, and its free scan at gold.safeguard.sh/scan is a way to see a hosted run's time from the outside. Apply the same cold and warm procedure to it.
Quick check
How many runs are enough?
Five per condition shows the spread without taking all day. Add more if the range is wide.
Should I compare with vendor benchmark numbers?
Treat them as claims to check. Run your own repositories on your own machines.
Sources
- OSV-Scanner usage: documents an offline mode that uses locally cached databases.
Last reviewed: 19 October 2026.
