Report the median and a high percentile such as the 95th, not just the average. Scan times are skewed: most runs are quick and a few are very slow. The average blends them into a number nobody experiences. Developers remember the slow runs, so measure them directly.
Why averages mislead
Imagine a scanner that finishes quickly on most pull requests and takes much longer when a lockfile changes or a cache is cold. The average lands somewhere in the middle. No developer ever waits the average time. They wait the quick time or the long time.
The median tells you the typical experience. A high percentile tells you how bad the unlucky experience gets. Together they describe the shape. The gap between them is itself a useful number: a wide gap means something inconsistent is going on, such as cache misses or noisy neighbours on shared runners.
What the monitoring documentation says
The Prometheus documentation on histograms and summaries makes two points that apply here. First, a histogram stores counts in buckets, and quantiles are calculated later from those buckets. Second, pre-calculated quantiles cannot be aggregated. You cannot average the 95th percentile of ten jobs to get the 95th percentile of all jobs. If you want a percentile across many repositories, store the raw durations or histogram buckets and calculate at query time.
That is the practical rule: keep durations, not just summaries.
Steps to collect it
- Record the start and end time of the security job for every run. Most CI systems already expose job duration through their API.
- Store one row per run with repository, branch type, scanner version, duration and outcome code.
- Choose buckets that match decisions you care about, such as under a minute, under five minutes and over ten.
- Calculate the median and 95th percentile weekly, by repository group.
- Compare the same repository against itself over time, not against other teams.
Slice before you compare
A pull request scan, a scheduled full scan and a release scan do different work. Mixing them in one distribution is the fastest way to produce a meaningless chart. Tag every run with its trigger and report each separately.
Also separate waiting from working. If a job spent most of its time queued for a runner, the scanner is not slow. See the companion post on queue time versus run time.
What to do when the tail is long
- Check cache hit behaviour. A cold cache often explains the slowest runs.
- Look at repository size and whether the scan covers generated files or vendored code.
- Check for retries inside the job, such as network calls to a vulnerability feed.
- Consider running the full scan on a schedule and a faster incremental check on pull requests.
Resist the temptation to set a target on the percentile and then reward teams for hitting it. People will find ways to hit a number. Use it to find problems.
Quick check
Is the average ever useful?
For capacity planning, total minutes used is more useful than the average. For developer experience, use the median and a high percentile.
How many runs do I need for a stable percentile?
A high percentile needs a decent number of runs. With a handful of runs a week, report the slowest few runs individually instead.
How Safeguard fits
Safeguard's dependency scanning and SBOM generation can run in your pipeline, and the free scan at gold.safeguard.sh/scan lets you see a result before you wire anything up.
Sources
- Prometheus documentation, Histograms and summaries (quantile calculation and aggregation limits): https://prometheus.io/docs/practices/histograms/
Last reviewed: 19 October 2026.
