Build a small project with a hand picked set of dependencies whose status you have verified from advisories. Run each scanner against it and score misses and extras against your answer key. A known answer set turns "which tool is right" into a measurement you can repeat.
Why a test project beats comparing real repos
On a real repository you never know the truth. Every disagreement is an argument. On a project you built, you wrote the answer key first, so each disagreement is scored. You also control scope, files and versions, so the usual causes of drift are removed.
Step 1: choose a small set of cases
Aim for 10 to 20 entries. Each one should test a different behavior.
| Case | What it tests |
|---|---|
| A direct dependency with a reviewed advisory on your exact version | Basic detection |
| A transitive dependency with an advisory | Tree resolution |
| A version just above the fixed version | False positive check |
| A version just below the fixed version | Boundary handling |
| A development only dependency | Scope policy |
| A package with mixed case or punctuation in its name | Name normalization |
| A pre-release version | Version ordering |
| A package with no known advisory | Clean control |
Pick versions from advisories you have read yourself, and write down the advisory ID and the fixed version for each. The OSV schema is a useful reference for what the fixed event means.
Step 2: write the answer key
For each entry, record package, version, expected result (finding or no finding), advisory ID, and your source. Keep it in a plain file next to the project. Do not copy results from either tool into it.
Step 3: run both tools the same way
Pin the scope, point both at the same lockfile, and record the tool versions and the date. Run each tool against the identical commit.
Step 4: score the results
Count four numbers per tool.
- Hits: expected findings that were reported.
- Misses: expected findings that were not reported.
- Extras: reported findings you did not expect.
- Clean passes: controls correctly left quiet.
Investigate each miss and extra. An extra is not always a defect, since the advisory data may have changed after you wrote the key. Update the key with a dated note if so.
Step 5: repeat on a schedule
Advisories change, so rerun every quarter and after any tool upgrade. Keep the results file. Over time it tells you whether a tool change helped.
Limits of the method
Fifteen entries cannot represent your whole estate. The test project catches systematic behaviors, such as boundary handling, naming and scope. It does not measure coverage of your languages, private packages or unusual build setups. Treat it as a calibration step, not a final verdict.
A second limit is staleness. A key that is a year old will mislead. Date every row and retire entries whose advisories have been revised.
Keep it safe
Use real vulnerable versions only in an isolated test project that never ships and is never deployed. Mark the repository clearly and keep it out of production build pipelines.
Quick check
How many cases do I need?
Ten to twenty is enough to expose systematic differences. More cases add upkeep without much new information.
Can I publish the scores?
Be careful. Results depend on versions, dates and your own key, so describe the method and date rather than ranking tools.
Sources
Last reviewed: 19 October 2026.
