For most teams, six to ten repositories chosen for variety is enough to compare dependency scanners. Cover each ecosystem you use, at least one large and one small project, and one old, neglected repository. Beyond that, extra repositories add effort faster than they add information.
Variety matters more than volume
Fifty similar repositories tell you about one situation fifty times. Two repositories in different ecosystems tell you about two situations. A scanner's weaknesses tend to cluster by ecosystem, build tool and repository age, so your sample should span those differences.
This is a judgment about test design, not a statistical guarantee. With a handful of repositories you cannot claim a measured accuracy rate for the whole company. What you can do is find clear failures and clear strengths in the situations you actually have.
A selection grid
Build a grid of the dimensions that matter to you and make sure each cell is covered at least once.
| Dimension | Cover at least |
|---|---|
| Ecosystem | Every language or package manager you run in production |
| Size | One small service, one large monorepo or app |
| Age | One recently active, one stale |
| Lockfile | One with, one without |
| Dependencies | One with private packages, if you have any |
| Criticality | One system leadership cares about |
If you have three ecosystems, a sensible set is two or three repositories per ecosystem, with the grid satisfied across them. You will end up between six and ten almost by itself. The post on choosing a test corpus goes deeper on picking, and choosing first repositories covers rollout order.
Why not use everything
Running every candidate on every repository sounds thorough, but it has costs.
- Adjudicating findings is manual. Every extra repository multiplies review time.
- Setup problems appear early and repeat. You learn the pattern from the first few.
- Large sets invite shortcuts, such as skipping verification, which harms quality more than a smaller sample would.
A better use of time is verifying findings in a smaller set deeply. Ten repositories with every one sided finding adjudicated beat fifty with raw counts.
When to go bigger
Add repositories in two situations.
- A tool fails on a repository. Find out whether it is a one off or a pattern by adding two more of the same type.
- A tie. If two tools are indistinguishable on your set, add repositories from the cells where they differ most, not random ones.
What a small sample can honestly say
You can say: "On these eight repositories, covering these ecosystems, tool A identified every package in the reference lockfile and tool B missed one ecosystem entirely." That is a useful, checkable statement.
You cannot say: "Tool A is 97 percent accurate across our estate." Avoid any number that suggests a population rate, and always show the denominator, such as "412 of 430 packages."
Write down why each was chosen
For every repository, note which cell of the grid it covers and why it is in the set. If someone challenges the result, you can show the sample was designed, not cherry picked. Pair this with a pre-registered decision rule so the sample cannot be bent after the results arrive.
Safeguard's free scan at gold.safeguard.sh/scan takes a single repository, which makes it easy to run across your grid one at a time, and its dependency scanning and SBOM output give you per repository inventories to compare.
Quick check
Is one repository enough?
It is enough to rule out a tool that fails badly, but not to choose between two good ones. Use at least one per ecosystem.
Should I include open source projects I do not own?
Only as a supplement. Your own repositories reflect your build and layout, which is the point of the test.
Last reviewed: 19 October 2026.
