Pilot a supply chain security platform on three repositories: one well maintained, one messy, and one legacy. Define what you will measure before connecting anything, run for two to four weeks, and compare inventory accuracy, noise, and time to a safe fix. Three varied repos teach more than thirty similar ones.
Pick three repos on purpose
Choosing your tidiest projects flatters any tool. Choose contrast.
- The clean repo. Current lockfile, tests, active owner. This shows the ideal case.
- The messy repo. Monorepo, mixed languages, or vendored code. This tests coverage.
- The legacy repo. Old dependencies, few tests. This tests how the tool handles a long list of old findings.
If one of the three produces a container image, scan the image as well as the source, because the two often disagree.
Write the scorecard first
Decide in advance what a good result looks like. A simple scorecard works.
| Question | How to check | Pass looks like |
|---|---|---|
| Is the inventory complete? | Compare to a manual list from the owner | Few or no missing packages |
| Are versions exact? | Spot check ten against the lockfile | All match |
| Is the noise manageable? | Count findings before and after reachability | Clear drop, with reasons |
| Can we act? | Time one real fix end to end | Within a normal sprint |
| Is the evidence usable? | Export an SBOM | Opens cleanly in a standard viewer |
For the SBOM check, confirm the file claims a recognized format. CycloneDX is an OWASP standard that Ecma International published as ECMA-424, and SPDX is recognized as ISO/IEC 5962:2021. Either is a reasonable target.
The four week schedule
- Week 1. Connect with read only access. Capture the first inventory and first findings. Do not fix anything yet.
- Week 2. Triage. Ask the repo owners to mark each of the top twenty findings as real, not real, or unsure, and compare with the tool's ranking.
- Week 3. Fix three findings, ideally one per repository. Record time spent and whether the tool helped.
- Week 4. Review results with the owners and the security lead against the scorecard.
Traps to avoid
- Counting raw findings as a score. More findings is not better coverage. It may be more noise.
- Letting the vendor pick the repos.
- Skipping the legacy repo because the results look ugly. That is where your real backlog lives.
- Treating the pilot as free production use. Set an end date.
What to hand the decision maker
One page: the scorecard, three honest observations from the repo owners, and the open questions you could not answer in four weeks. Say plainly what the pilot did not test, such as scale.
A worked example
Take a hypothetical engineering group that picks three repositories. The first is a small API service with a current lockfile and active owner. The second is a monorepo with JavaScript, Python, and a vendored library. The third is a seven year old internal tool with few tests.
Week one produces three inventories. For the API service, the owner confirms the list matches. For the monorepo, the inventory misses the vendored library, because it was copied in without a package record. For the legacy tool, the inventory is complete but long, with many old dependencies.
In week two, the owners review the top twenty findings from each repository. The API service owner marks most as real. The monorepo owner marks a number as unreachable, with a call path check to back it up. The legacy tool owner marks many as real but unfixable without a large upgrade.
In week three, the team fixes one item in each. The API service takes minutes. The monorepo item takes an afternoon because a transitive dependency must be updated. The legacy tool item is replaced by an exception with a mitigation and a 60 day review. These are invented numbers for illustration, but the pattern is realistic: effort varies widely, and the pilot should show you that range.
Scoring template
Give every evaluator the same sheet and ask for a score from one to five with a note.
| Criterion | Weight | Repo A | Repo B | Repo C |
|---|---|---|---|---|
| Inventory accuracy | High | |||
| Noise after reachability | High | |||
| Clarity of fix guidance | Medium | |||
| Pull request experience | Medium | |||
| SBOM export quality | Medium | |||
| Effort to set up | Low |
Add up the scores only after each person has written their notes. Notes carry the reasons, and the reasons matter more than the totals.
Questions for the repo owners at the end
- Did the tool find anything you did not know about?
- Did it report anything you consider wrong, and how often?
- Would you want it on every pull request?
- What would make you turn it off?
The fourth question is the most revealing. Whatever annoys the owners during four weeks will annoy everyone at scale.
Writing the recommendation
Keep it short. State the result for each criterion, the three strongest observations from owners, the risks you did not test, and the next step. Say clearly whether the pilot supports a wider trial, a second tool test, or stopping. A pilot that ends in a clear no is a good use of four weeks.
After the pilot
Do not leave pilot access open. Remove it, export what you learned, and keep the scorecard. If you evaluate a second tool later, reuse the same three repositories and the same sheet so the results compare fairly.
Try it yourself
Click through it at your own pace, or open it full screen. Open the demo and compare the number of findings before and after the reachability step, which maps to the noise row on your scorecard.
Safeguard offers dependency scanning, SBOM output, and reachability analysis, and the free scan at gold.safeguard.sh/scan can serve as your week 1 baseline.
Quick check
How many repos is enough for a pilot?
Three varied repos is enough to compare tools on inventory, noise, and fix effort. Scale is a separate question to test later.
Should developers be involved?
Yes. The repo owners judge whether findings are real, which is the strongest evidence you will get.
What if the vendor wants to choose the repos?
Decline. Choosing the repos yourself, with contrast, is what makes the pilot trustworthy.
Sources
Last reviewed: 19 October 2026.
