To score a trial blind, export results from each tool, strip names and logos, label them A, B and C, and have reviewers rate the same sample of findings and fixes using a written rubric. Reveal the identities only after scores are recorded. This reduces brand bias and the pull of a good demo.
Why blind scoring helps
People rate output differently when they know the source. A familiar name gets the benefit of the doubt, and a polished interface can mask weak results. Blind review focuses everyone on what the tool found and whether it was right. It is a standard approach in many fields, and it adapts easily to a security evaluation.
It will not hide everything. Some formats reveal the tool, and reviewers may guess. That is acceptable, since the aim is to reduce bias, not to remove it.
Setup in six steps
- Pick the sample. Choose three to five repositories, and fix the commits.
- Run each tool. Use the same inputs and export results in a neutral format, such as a spreadsheet with the same columns.
- Anonymise. Remove tool names, logos, product specific wording and identifiers. Replace them with A, B and C. A person outside the review group holds the key.
- Draw a review sample. From each tool, take a set of surfaced findings and a set of suppressed or filtered ones, in equal numbers.
- Rate independently. Each reviewer scores items without discussion, using the rubric below.
- Reveal and discuss. Compare scores, unblind and then add the parts blind scoring cannot cover, such as usability and support.
A simple rubric
| Question | Scale |
|---|---|
| Is this finding real for this code? | Yes, no, unsure |
| Would a developer know what to do next? | Clear, partly, unclear |
| Is the priority sensible? | Agree, disagree |
| For fixes: would you merge it after review? | Yes, with changes, no |
| For suppressed items: was suppression justified? | Yes, no, unsure |
Ask reviewers to note why for every "no" or "unsure". Those notes are often the most valuable output.
What blind scoring cannot judge
Some qualities need the full product: setup effort, integration fit, support, data handling and contract terms. Score those separately and unblinded in your scorecard. Combine the blind results and the open results with weights you set before the trial.
Keep it light
You do not need many reviewers. Two or three engineers, each spending a couple of hours on a sample of around 60 items per tool, give you a useful signal. Record agreement between reviewers. If they disagree often, your rubric needs tightening.
A caution on small samples
Blind scores from a small sample are a signal, not a verdict. If two tools score close together, do not pick a winner from the gap. Expand the sample, or let the open criteria such as support and data handling decide. Treat a large and consistent gap as meaningful, and a small one as a tie.
Where Safeguard fits
If Safeguard is one of your candidates, its exports go into the same anonymised pool as everyone else's. You can generate a first set of results with the free scan at gold.safeguard.sh/scan.
Quick check
Will reviewers guess which tool is which?
Sometimes. Normalise formats and wording to reduce it. Partial blinding still helps.
Who should hold the key to the labels?
Someone outside the review group, such as a project coordinator, until all scores are submitted.
Last reviewed: 19 October 2026.
