Scale test an SCA tool by scanning a deliberately uneven sample of your own repositories, including your largest monorepo, and recording wall clock time, failures, memory and API calls for each. Judge the tool on the slowest and ugliest repos, not the average one.
Why vendor benchmarks do not transfer
A demo repository has a clean lockfile and a few hundred packages. Your estate has a decade of manifests, generated code, vendored folders and one repository nobody dares to clone. Scale problems show up in the tail, so your test set should be weighted toward the tail.
Build the sample
Pick 20 to 40 repositories. Include:
- Your largest monorepo by file count.
- The repo with the most lockfiles.
- A repo with a private registry dependency.
- A legacy repo with no lockfile at all.
- A few typical services in each language you run.
- A repo with a huge transitive tree.
Write down what you expect before you run anything. Surprises are the point.
What to measure
| Metric | How | Why it matters |
|---|---|---|
| Wall clock per repo | Timestamps from trigger to results available | Pull request feedback must fit your CI budget |
| Failure rate | Count scans that error, time out or return partial results | Silent partial results are worse than failures |
| Peak memory and CPU | Runner metrics | Sizing and cost of the runners |
| Source control API calls | Request counts and rate limit headers | Large estates hit limits |
| Result completeness | Compare dependency counts to a known lockfile | Proves coverage |
The API limit check
If the tool pulls from GitHub, remember the limits. GitHub documents 5,000 requests per hour for authenticated users, higher for some Enterprise Cloud apps, and secondary limits such as 100 concurrent requests and 80 content generating requests per minute. It also notes the response headers, such as x-ratelimit-remaining, are authoritative. Ask the vendor how the tool backs off and whether it reads those headers. Then watch the counters yourself during the test.
Run it three ways
- Cold start. First scan of everything. Measures onboarding.
- Warm rerun. Same commit again. Should be much faster if caching works.
- Burst. Trigger 30 scans at once, as a Monday morning merge wave would. Look for queueing and failures.
Read the results honestly
Report the 50th, 90th and worst times, not only the mean. Record every failure with its cause. A tool that finishes 38 of 40 repos and tells you clearly which two failed and why is more usable than one that finishes 40 with two quietly incomplete.
Write the outcome in a one page summary your procurement and platform teams can both read. Related: How to shard scans of a large monorepo.
Try it yourself
Click through it at your own pace, or open it full screen.
Note how the inventory reports counts per project. That is the number to compare against your own lockfiles.
Safeguard offers dependency scanning with a free scan at gold.safeguard.sh/scan if you want a quick first data point.
Quick check
How many repositories should a scale test include?
Twenty to forty, chosen to include your worst cases, is enough to expose most problems.
What is the most important number to record?
The failure and partial result count. Speed means little if results are incomplete.
Sources
Last reviewed: 19 October 2026.
