Write down how you will decide before any scanner has run. Once results exist, people see patterns that fit what they already wanted, and the loudest or most senior opinion wins. A rule agreed in advance turns the comparison into a measurement.
What goes wrong without one
Picture the meeting after a trial. Tool A found more issues. Tool B had a nicer interface. Someone liked B's support call. Each person now picks the criteria that make their favorite win. That is not dishonest, it is how people reason, and it is why serious studies in many fields register their method first.
A decision rule written beforehand removes the room to redefine success afterward.
What to put in the rule
Keep it to one page. Include five parts.
- The question. For example: "Which candidate identifies our real dependencies and vulnerabilities most completely with the least triage effort across our ecosystems?"
- The measures. Name them: inventory match against the lockfile, precision after adjudication, one sided true positives, setup time, triage minutes per finding, fit with your pipeline.
- Weights or priorities. Either a weighted score or a strict order. A weighted scorecard is the common way.
- Pass and fail lines. For example: "Any tool that cannot read our main ecosystem fails."
- The tie rule. What happens if scores are within a stated margin. See scoring scale and tie breakers.
A template
| Section | Example wording |
|---|---|
| Question | One sentence |
| Test set | List of repositories and frozen SHAs |
| Measures | Six measures, each with how it is computed |
| Must pass | Hard requirements, such as ecosystem support and export format |
| Scoring | Weights summing to 100 |
| Tie rule | Prefer lower triage effort, then easier integration |
| Who decides | Named person, and who is consulted |
| Date signed | Date, before first scan |
Make the rule hard to quietly change
- Store it in a document with a visible date, such as a commit in a private repository or a dated page.
- Share it with everyone on the evaluation team, and optionally with the vendors, so no candidate is surprised by how it is judged.
- If you must change the rule, write the change down with the reason and the date, and state whether results had been seen. A change made after seeing results is a different thing from a change made before.
Handle surprises honestly
Sometimes a result shows the rule missed something important. A tool might fail in a way you did not foresee. That is fine, but handle it openly: add the new criterion in a clearly marked section, apply it to every tool, and report results with and without it. If the winner changes depending on it, say so. Readers can then judge for themselves.
Do not tune for the favorite
If a stakeholder already likes a tool, ask them to write what result would change their mind. Put that in the rule too. It is a small act of intellectual honesty and it makes the final readout much easier to defend. For the readout itself, see the one page trial readout.
Share the plan, not only the winner
When you report the outcome, include the rule as written beforehand, the data, and the verdict. Readers who see all three can follow your reasoning, which is the point.
Safeguard encourages teams to try its dependency scanning and SBOM output against their own repositories, and the free scan at gold.safeguard.sh/scan can be part of any test you design this way.
Quick check
Is a scorecard the same as a decision rule?
A scorecard is one part of it. The rule also covers the test set, hard requirements, ties and who decides.
Do vendors need to see the rule?
Not required, but sharing it keeps the process fair and avoids arguments about moving goalposts.
Last reviewed: 19 October 2026.
