A dependency scanning proof of concept needs about seven pass marks, each measurable and written before the trial starts: inventory completeness, finding accuracy, prioritization quality, scan time, developer workflow, fix success, and evidence export. Set thresholds from your own baseline, not from a vendor brochure.
Why write them first
Without pass marks, every result can be argued either way. Ten thousand findings looks like great coverage to one person and unusable noise to another. A pre-agreed table removes the argument and also protects the vendor from moving goalposts. If you want the day by day plan around this table, see the two week plan.
The table
| Criterion | How to measure | Example pass mark (set your own) |
|---|---|---|
| Inventory completeness | Compare scanner package list with your build's installed packages | Misses on a named sample stay under a threshold you set |
| Finding accuracy | Hand check a random sample of findings | Share of true positives above your threshold |
| Prioritization | Does the top of the list match what your team would fix first? | Your two senior engineers agree with most of the top ten |
| Scan time | Time a full scan and a pull request scan | PR scan fits your CI budget |
| Developer workflow | Developer can act on a result without asking security | Each trial developer says they would keep it on |
| Fix success | Apply suggested upgrades, run tests | Most suggested fixes build and pass |
| Evidence export | Export an SBOM and a findings report | Files open in your downstream tools |
The right column is deliberately a template. A fintech with a strict change window and a three person startup will not pass the same marks.
How to set a threshold
Start from today's reality. If your team already triages manually, time that for ten findings. If you already generate an SBOM, count how many packages it lists for one repo. The scanner should beat your baseline or add something you do not have.
Define a floor, not just a target. A floor is the line below which you stop. For example, if the scanner cannot detect your main language's lockfile format, that is a stop, whatever else it does well.
Weighted or pass fail?
Use pass fail for the floors (coverage, data handling) and a simple 1 to 5 score for the rest. Do not average everything into one number. A tool that scores perfectly on dashboards and fails on lockfiles is still a fail.
Common mistakes
- Counting findings as a success metric. More findings is not better. Accurate and actionable is better.
- Testing only the repo the vendor suggests.
- Forgetting the human criterion. A scanner nobody opens protects nothing.
- Leaving out exit conditions, so the PoC never ends.
What to do when results are mixed
Mixed results are normal. Rank the failed criteria by how hard they are to fix. A noisy default policy can be tuned in a day. A missing ecosystem cannot. Extend only for the fixable gaps, with a date.
Try it yourself
Click through to see how filtering by whether vulnerable code is actually reachable changes the size of the list, then ask what share of your own findings would survive that filter.
Click through it at your own pace, or open it full screen.
Quick check
How many criteria is too many?
More than ten gets hard to score honestly. Seven to eight, with two or three as hard floors, is workable.
Who should approve the criteria?
The person who will sign the decision, plus the developers who will live with the tool. Get both before day one.
Safeguard offers SBOM and dependency scanning with reachability analysis, and you can try a free scan at gold.safeguard.sh/scan before you plan a longer test. The CLI, desktop app and IDE extensions are listed on safeguard.sh/download.
Last reviewed: 19 October 2026.
