To compare two security tools without bias, run blind triage. Export findings from both on the same code, remove tool names and styling, shuffle the list, have reviewers rate each finding, then reveal the source. Brand loyalty and interface polish stop influencing the accuracy score.
Why bias creeps in
Reviewers who know which tool produced a finding judge it differently. A familiar logo gets the benefit of the doubt. A tool that was your champion's pick gets patient triage, while the other gets quick dismissals. Even a nicer dashboard can make the same finding feel more credible.
Blind triage is cheap and removes most of this.
The method
- Freeze the target. Both tools scan the same commit of the same repositories on the same day.
- Export raw findings. Use CSV or JSON. Keep: component or file, line or version, issue identifier, description.
- Normalise. Put each finding into a common format with identical columns and wording style. Remove severity labels if the tools use different scales, or map them to one.
- Merge and deduplicate. If both tools report the same issue on the same component, it becomes one row with two hidden sources.
- Shuffle. Randomise the order and assign each row an anonymous ID.
- Triage blind. Two reviewers judge each row independently using fixed verdict categories: true and reachable, true but not reachable, false positive, unclear.
- Resolve disagreements. Where reviewers differ, a third person decides.
- Reveal. Map IDs back to tools and calculate.
What to calculate
| Measure | How |
|---|---|
| Precision | Real findings divided by findings reported, for each tool |
| Unique true findings | Real issues only one tool reported |
| Overlap | Real issues both reported |
| Review time | Average minutes per finding for each tool's rows |
Recall, the share of all real issues each tool caught, can only be estimated from the pooled set of confirmed findings, so describe it as a rough guide.
Pitfalls
- Unequal configuration. If one tool ran with defaults and the other was tuned, the comparison is unfair. Record settings for both.
- Different scope. One tool may scan more file types. Note it rather than hiding it.
- Reviewer fatigue. Cap each session at an hour. Fatigue produces false verdicts.
- Tiny samples. Under 30 or so findings, differences may be luck. Say so.
What blind triage cannot judge
It measures finding quality. It does not measure workflow fit, support, speed, integration or price. Score those separately in your wrap up scorecard, ideally by people who have used each tool, and keep the weights fixed before you see the blind results. See a wrap up scorecard that survives scrutiny.
Bias that remains
The target code matters. If your repositories happen to favour one tool's strengths, that is a valid result for you, but it may not generalise. Ask whether your sample reflects the languages and build styles you will use in future.
Reviewers also bring bias about the issues themselves: some people dismiss anything that is not a remote code execution. Agree categories and examples before you start, so a verdict means the same thing for everyone.
If you want a third data point, Safeguard's free scan can be run on the same project, and its output can enter the same anonymous pool like any other.
Quick check
Do I need two reviewers per finding?
It is better, because disagreements show where categories are vague. With limited time, double review a random subset.
Can I blind the comparison if findings look very different?
Yes. Normalise to the same columns and plain wording, and remove tool specific labels before shuffling.
Last reviewed: 19 October 2026.
