Yes. Split your repositories into a visible set used for setup and questions, and a hold-out set revealed only for the final run. If a tool performs well on both, it handles your code. If it only does well on the repositories it saw, it was tuned.
What tuning looks like in a trial
During a comparison, you will talk to each vendor or project community. You send logs, you share a failing repository, someone adjusts a setting or adds support for a file format. That is useful and normal. It also means the tool improves on exactly the cases you showed it.
Whether that help would be there for every future repository is a different question. A hold-out set answers it by testing on code nobody has coached the tool on.
How to split
| Set | Share | Use |
|---|---|---|
| Working set | About two thirds | Setup, debugging, questions, support tests |
| Hold-out set | About one third | One final run per tool, no coaching |
With eight repositories, that is five visible and three hidden. Make sure the hold-out set mirrors the grid, with at least one repository for each major ecosystem. Chosen badly, the hold-out set tests nothing. The selection grid from an earlier post helps here.
Decide the split before you start, by a method that is not your preference. Sort the repositories by name and put every third into the hold-out set, or draw lots within each ecosystem. If you pick by feel, you will hide the easy ones and show the hard ones, or the reverse.
Keeping it hidden
- Do not put hold-out repositories in any shared screen, ticket or call with a vendor.
- Do not connect them to any trial tenant until the final run day.
- On final day, give every tool the same hold-out checkout, frozen, as explained in the freeze post.
- Run each tool once with settings already agreed. No tweaking between runs.
If a trial environment needs a repository connected in advance, create a copy under a neutral name and connect that. Remove it afterward.
Reading the results
Compare each tool's quality on the working set and hold-out set using the same measures, such as inventory match and adjudicated precision.
- Similar on both: the tool is stable on your code.
- Better on working set: some of its performance came from help or tuning. Decide whether you would get that help after buying.
- Worse on hold-out and the gap is large: be cautious. It may mean the result depends on specialist attention.
- Better on hold-out: possible, since sets are small. Do not over read it.
With a small sample, small differences are noise. Look for large gaps and for failures that make sense, such as an unsupported ecosystem.
Be fair about support
Help from a vendor is part of what you are buying, so do not punish a tool for needing it. The point is to know how much of the result depended on it. The vendor support test covers evaluating support separately from detection.
When it is not worth it
If you have fewer than five repositories, a hold-out set leaves too little to learn from. In that case, hold back a single repository and use it as a final sanity check, or rotate: use one repository only after all setup is complete.
Safeguard's free scan at gold.safeguard.sh/scan needs no setup, so it is easy to run on the hold-out set at the end under the same rules as every other candidate, alongside its dependency scanning and SBOM output.
Quick check
Does a hold-out set mean vendors cannot help me at all?
No. They can help with the working set. The hold-out run is simply done without coaching.
What if a tool needs a repository connected in advance?
Connect a renamed copy, then remove it, or schedule the hold-out run to include setup time as a measured step.
Last reviewed: 19 October 2026.
