Write your criteria first, pick two or three real repositories, plant a few known issues, run every tool on identical inputs, label a fixed sample of findings, repeat the runs, and record the decision. The order matters, because choosing criteria after seeing results invites bias.
Step 1: Write the criteria before touching a tool
List what you need: languages, build systems, pipeline integration, noise tolerance, fix guidance, reporting, and cost of ownership. Weight them. Put the weights in a file with the date. Our SCA scorecard guide shows a format.
Step 2: Choose the test inputs
Pick two or three repositories that represent your work: one typical service, one awkward legacy project, and one that mixes languages. Freeze a commit for each so every tool sees the same code.
Step 3: Plant ground truth
Add a few known issues to a branch: an outdated dependency with a known advisory, a hard-coded secret, an injection flaw. Keep a private list. This gives you a known answer to check for missed issues. Use a test branch, never production.
Step 4: Fix the settings
Document the version of each tool and the configuration. Either use defaults for all, or give each tool equal tuning time. Unequal effort is the most common source of unfair results.
Step 5: Run and label
Run each tool at least three times. Take a fixed sample of findings per tool, for example the first 30 by severity. Have two engineers label each finding as real, not real, or unclear without knowing which tool produced it. Resolve differences by discussion.
Step 6: Measure what matters
Compute, per tool: planted issues found, real findings, false alarms in the sample, variation between runs, time to set up, scan time and how many findings had usable fix guidance. Our post on precision and recall explains the two core numbers.
Step 7: Write the record
Summarize the method, the numbers, the weights and the decision, plus doubts. Share it with the people who must live with the choice. If a vendor or reviewer later challenges the result, the record shows what was done.
Try it yourself
Click through it at your own pace, or open it full screen.
Look at the dependency list and count how many entries you could check by hand. That tells you how big a sample to label.
Where Safeguard fits
Safeguard offers SBOM and dependency scanning, reachability analysis and policy gates. You can include it as one candidate using the free scan at gold.safeguard.sh/scan.
Quick check
How long should a bake-off take?
A focused test can fit in a few weeks. Longer tests rarely add information and tend to drift.
Should the vendors see my test files?
Share the setup, not the answer key. Keeping the planted issues private prevents tuning to the test.
Last reviewed: 19 October 2026.
