To measure SBOM accuracy, build a small repository where you already know every dependency, generate an SBOM, and count two things: components the tool missed and components it invented. Express them as recall and precision. Repeat per ecosystem, because accuracy rarely transfers from one language to another.
Why a known answer beats a feature list
A product page can say "supports many ecosystems". It cannot tell you whether the tool lists the transitive dependency your build actually installed. The only way to learn that is to compare output against a list you trust. Most teams skip this and discover the gaps when a customer or auditor finds one.
Build the answer key
Create one small project per ecosystem you ship. Keep each one boring and fully pinned.
- Declare five direct dependencies.
- Install them and write down the full resolved tree from the package manager itself, direct and transitive.
- Add the awkward cases you really have: a dev dependency, an optional dependency, a package from a private source, and a vendored copy.
- Save the list as a plain file of name and version pairs. This is your answer key.
Do the same for one container image you build. For the image, list OS packages with the distribution's own package command, and language packages from inside the image.
Score the output
Generate the SBOM and normalise names and versions so the two lists are comparable. Then count.
| Term | Meaning | Formula |
|---|---|---|
| True positive | In the key and in the SBOM | counted |
| False negative | In the key, missing from the SBOM | counted |
| False positive | In the SBOM, not in the key | counted |
| Recall | How much of the truth the tool found | true positives divided by true positives plus false negatives |
| Precision | How much of the output is real | true positives divided by true positives plus false positives |
Recall protects you from blind spots: a missing component is a vulnerability you cannot see. Precision protects your time: extra components create findings that waste triage effort. Report both, because a tool can score high on one by sacrificing the other.
Check the fields, not just the names
Names and versions are the start. A second pass should compare identifiers and relationships.
- Is each component identified in a way that matches vulnerability data, such as a package URL with the right type and version? The Package URL specification defines the parts: type, namespace, name, version, qualifiers and subpath.
- Does the SBOM say which component depends on which, or is it a flat list?
- Are dev and test dependencies marked, so you can filter them?
Read the misses
Group every false negative by cause. Common groups are: no lockfile, a vendored copy, a binary with no metadata, a build step that downloads something at install time, and a layer the tool did not unpack. The group tells you whether the fix is a tool setting, a change to your build, or a limit you must disclose.
Keep it as a regression test
Store the answer keys in the repository. Rerun the comparison whenever you upgrade the generator or change your build. A tool update that quietly drops a class of component shows up as a recall drop within minutes.
Try it yourself
Click through it at your own pace, or open it full screen. Click into a component and compare direct and transitive entries against what you would put in your own answer key.
Where Safeguard fits
Safeguard builds a dependency inventory and SBOM from your projects, and you can run the free scan at gold.safeguard.sh/scan against a known-answer repository as one of your test inputs.
Quick check
How many test projects do I need?
One small project per ecosystem you ship, plus one container image. Depth matters more than count.
Is a perfect score realistic?
Often not for binaries or vendored code. Aim to understand and document each miss rather than hide it.
Sources
Last reviewed: 19 October 2026.
