To evaluate an SBOM tool, run it on three of your own repositories and check ten things: depth of transitive dependencies, format validity, identifier quality, license data, ecosystem coverage, speed, CI fit, update behavior, export options and how it handles what it cannot see. A vendor demo repo proves very little.
Why your own repositories matter
Demo repositories are clean on purpose. Yours have a monorepo with three package managers, a vendored library nobody remembers, a Dockerfile that installs things with curl, and a lockfile that drifted. Pick three repositories that differ: one that is typical, one that is ugly, and one that ships to customers. Run every candidate on the same three and keep the raw output files. Comparing files beats comparing screenshots.
The ten tests
-
Transitive depth. Count components in each output and spot check five deep dependencies against the lockfile. A tool that lists only direct dependencies will look fast and tell you little.
-
Format validity. Validate the output against the CycloneDX or SPDX schema with a neutral validator, not the vendor's own viewer.
-
Identifiers. Check that components carry package URLs and versions, not just names. Without them, matching to advisories gets fuzzy.
-
Licenses. See whether license fields are filled in, and whether unknowns are marked as unknown rather than guessed.
-
Ecosystems. List every language and file type you use, including the dull ones such as a Gradle build or a Python requirements file, and confirm each one produces components.
-
Speed and resource use. Time a cold run and a repeat run on the largest repository.
-
CI fit. Does it run headless, return a usable exit code, and avoid needing a browser login in a pipeline?
-
Update behavior. Change a dependency, rerun, and see how the diff is presented.
-
Export and ownership. Can you export everything in a standard format and take it with you if you leave?
-
Honesty about gaps. Ask what the tool does when it cannot resolve a package. The right answer is a clear marker, not silence.
Scoring without fooling yourself
Write the pass bar before you run anything. For example: components within a small tolerance of a hand-checked lockfile count, schema validation passes with zero errors, and every ecosystem you use is covered. Have two people score independently. If one tool wins on a test you do not care about, that win should not count. Keep a short note on setup effort too, because the tool that takes a week to wire into CI has a cost the output file will not show.
Two more habits help. Run the tools in the same environment, on the same commit, so differences come from the tool and not the checkout. And keep a list of surprises, such as a component one tool found that another missed. Chase each surprise to the lockfile and decide who was right. Those small investigations teach you more about a tool's real behavior than any score does, and they leave you with evidence you can show a skeptical colleague.
Try it yourself
Click through the demo and watch how the results change as you move through each step.
Click through it at your own pace, or open it full screen.
Where Safeguard fits
Safeguard runs SBOM and dependency scanning with reachability analysis, and you can try it on a repository with the free scan.
Quick check
Do I need to test both CycloneDX and SPDX output?
Test whichever your customers or regulators ask for. If nobody has asked, pick one, confirm the tool can export the other, and move on.
Last reviewed: 19 October 2026.
