Build a weighted scorecard in five steps: list must-have criteria that are pass or fail, weight the remaining criteria before you see any product, define what each score means in words, score blind where you can, and attach evidence to every number. The point is to make the decision explainable, not to produce a magic total.
Step 1: separate must-haves from preferences
Some criteria are gates. If a tool cannot meet them, the score does not matter. Examples: it cannot run in the region you need, it cannot export logs, or it needs credentials you will not grant. List these first and mark them pass or fail. Everything else goes into the weighted part.
Step 2: weight before you look
Agree weights in a meeting before any demo. Weights that are set after seeing products tend to drift toward the product people already like. Use a simple scale, with weights adding to 100.
| Criterion group | Example weight | Notes |
|---|---|---|
| Coverage of your actual risks | 30 | Mapped to your inventory, not a generic list |
| Enforcement and control quality | 25 | Can it block, or only report |
| Evidence and audit records | 15 | Logs you can hand to an auditor |
| Fit with your workflow | 15 | Where it plugs in, setup effort |
| Data handling and deployment | 10 | Where data goes, who can see it |
| Exit and portability | 5 | Export of rules and logs |
These numbers are an illustration. Change them to match your own risk.
Step 3: define the scores in words
Use a four point scale and write what each point means for each criterion. For example, for audit records:
- 0: no record of tool calls
- 1: record of the call without arguments
- 2: arguments and identity, no outcome
- 3: arguments, identity, outcome and time, exportable
Anyone reading a 2 then knows what it stands for. Without this, one scorer's 3 is another's 2.
Step 4: score blind where possible
Have scorers write notes before they see the totals. Where a trial result can be anonymised, such as a list of findings, remove the product name for the first pass of review. At minimum, have two people score independently and discuss differences.
Step 5: record the evidence
Next to each score, write the test that produced it and where the artefact is stored. When the decision is questioned in six months, you can show why.
Reading the result
Check three things before accepting the top total. Did any must-have fail? Is the lead larger than the disagreement between scorers? Does the winner score at least a 1 on every criterion, or is a high total hiding a hole? If the answer to the last question is no, treat it as a risk you accept in writing.
For the trial itself, see how to run a proof of concept.
Quick check
How many criteria should a scorecard have?
Between eight and fifteen. Fewer misses things, more makes each weight so small it stops mattering.
Should price be on the scorecard?
Keep cost separate. Score capability first, then compare cost among the options that clear your bar.
Safeguard policies and gates can be one of the options you test, and the free scan gives a first result without a commitment.
Last reviewed: 19 October 2026.
