Evaluate a malicious package detection tool by running it on your real lockfiles for two weeks, replaying known public reports as a test set, and scoring detection, time to alert, false positives and workflow fit. Write the scorecard before you start, so the vendor demo cannot set the criteria.
Why trial with your own data?
A demo uses packages picked to look good. Your lockfiles show what the tool does with the dependencies you actually have, including odd internal packages and old versions. That is where false positives and gaps appear.
How do you build a test set safely?
Use public reports rather than writing malware. The OpenSSF Malicious Packages repository publishes reports in the OSV format and covers typosquatting, account takeover, malicious binaries and dependency confusion. It also moves fully false positive reports into a withdrawn directory, which is a reminder that reports can be wrong (OpenSSF).
Sample a small set of reported packages across your ecosystems. Do not install them on a machine that matters. The goal is to check whether the tool flags the package name and version when it appears in a manifest, not to run it.
Many removed packages are no longer downloadable, so check which ones the tool can still analyse. Note that gap in your results.
What should the scorecard hold?
Score each tool one to five on:
- Detection: how many test set entries it flags, by threat type.
- Timing: how soon after you add a package to a manifest does it report?
- Noise: how many of your own, known-good packages does it flag? Count each one.
- Evidence: does each finding explain itself well enough for an engineer to decide in five minutes?
- Blocking: can it stop an install or a merge, and can you configure the threshold?
- Fit: does it work with your package managers, CI, proxy and ticketing?
- Operations: setup time, permissions requested, and how it handles private packages.
Weight the rows by what matters to you. If you ship to regulated customers, evidence may weigh more than speed.
What does a good trial run look like?
Week one: connect to a handful of representative repositories, import the test set, and log every finding without acting on it.
Week two: turn on blocking in a non-critical pipeline. Track how many blocks were correct, how many were wrong, and how long each decision took.
At the end, review every false positive with the vendor. A tool that explains its mistakes and fixes them is more useful than one that never admits to any.
What are the traps?
Do not let a trial run only on a clean project. Do not count flagged vulnerabilities as malicious findings. And do not accept a score without the reason. Ask whether results change after a rerun, since non-deterministic verdicts are hard to trust in CI.
Check data handling too. If the tool uploads source or manifests, learn where they go and who can read them.
What to do
- Write the scorecard and weights first.
- Pick three to five repositories across your main ecosystems.
- Build a test set from public reports, and keep it away from live machines.
- Run in observe mode for a week, then block mode in a low risk pipeline.
- Log every finding as correct, false positive or unclear, with time spent.
- Review all false positives with the vendor.
- Decide on the scorecard, not on the demo.
Quick check
How long should a trial last?
Two weeks is a workable minimum: one to observe, one to test blocking. Longer helps if your release cadence is slow.
Can we test with real malware?
You do not need to. Public reports and flagged names are enough to test detection, and avoid running hostile code.
Sources
Last reviewed: 18 October 2026.
