To compare AI assistants fairly, score each product on the same four feature areas, scheduled tasks, file generation, desktop app and remote sandbox, using weighted criteria and an evidence level for every score. Weight by your own risk and use case, and count claimed answers for less than tested ones.
Build it before the demos
Decide the criteria and weights before any vendor presents. Otherwise the smoothest presenter sets your scorecard. Use a simple 0 to 3 scale per criterion, and a multiplier for evidence: tested counts full, documented counts two thirds, claimed counts one third, unknown counts zero.
The criteria
| Area | Criterion | What a 3 looks like |
|---|---|---|
| Scheduled tasks | Run location is clear and documented | Device versus cloud stated, behavior when off described |
| Scheduled tasks | Identity and permissions per task | Dedicated or user scoped, narrow, revocable |
| Scheduled tasks | Failure alerts and logs | Alerts reach a named person, logs exportable |
| File generation | Output handling | Formulas and macros neutralized or disabled, tested |
| File generation | Storage, links and deletion | Private links, retention stated, admin delete |
| Desktop app | Permission model | Scoped folders, no clipboard or screen by default |
| Desktop app | Update and management | Signed updates, pace control, enforced settings |
| Remote sandbox | Isolation and network control | Named isolation, restricted egress, tested |
| Remote sandbox | State and secrets | Fresh by default, secrets short lived |
| Cross cutting | Admin controls and audit logs | Org wide switches, per feature, exportable logs |
| Cross cutting | Data handling terms | Retention, training use, sub-processors documented |
Add or remove rows to match your use. Ten to fifteen criteria is plenty.
Weights
Weights express your priorities. A team that mainly wants coding help might give the remote sandbox and desktop permissions the most weight. A team in a regulated sector might weight data handling and logs highest. Keep the total at 100 and write down why. When two stakeholders disagree on a weight, that disagreement is useful, so settle it before scoring.
Evidence levels in practice
For each score, record where it came from.
- Tested: you saw it work or fail in a trial.
- Documented: the vendor's written documentation or contract says so.
- Claimed: someone said so in a call.
- Unknown: no answer.
Many scorecards fail because a smooth demo scores like a tested control. Tests such as the marker file check, the private window link check and a forced failure take little time and make scores honest. The related posts on file generation and desktop permissions list tests you can reuse.
Add gates, not just scores
Some answers should disqualify, whatever the total. Examples: no admin way to disable a feature, no logs, public links by default with no control, or a vendor who cannot state where data is stored. A product with a high total and a failed gate does not proceed until the gap closes in writing.
Keep the comparison fair
Use the same test script for every product, run by the same people in the same week. Record the product version tested, since features change quickly. Avoid ranking on features you will not use. And leave out price from this card, since commercial terms belong in a separate review.
After the decision
File the scorecard as evidence of your selection process. Safeguard's compliance and evidence automation can hold it alongside the rest of your vendor review records.
A worked example: scoring two unnamed products
Call them Product One and Product Two. Use five criteria and weights that total 100 for a team that mostly writes code: sandbox isolation and network control (30), desktop permission model (25), scheduled task identity and alerts (20), file output handling (10), admin controls and logs (15).
Scores run from 0 to 3, multiplied by an evidence factor: tested 1.0, documented 0.67, claimed 0.33, unknown 0.
| Criterion | Weight | P1 score | P1 evidence | P1 points | P2 score | P2 evidence | P2 points |
|---|---|---|---|---|---|---|---|
| Sandbox | 30 | 3 | Tested | 30.0 | 3 | Claimed | 9.9 |
| Desktop | 25 | 2 | Tested | 16.7 | 3 | Documented | 16.8 |
| Scheduled | 20 | 2 | Documented | 8.9 | 2 | Tested | 13.3 |
| Files | 10 | 1 | Tested | 3.3 | 2 | Tested | 6.7 |
| Admin and logs | 15 | 3 | Documented | 10.1 | 1 | Claimed | 1.7 |
Points are weight times score divided by 3, times the evidence factor. Product One totals about 69 and Product Two about 48, even though Product Two claimed more. The gap comes from evidence, and from the admin controls. This is the scorecard doing its job: it rewards what you could verify.
Step by step: build and run your own
- Choose criteria from the table, then add or drop rows for your use.
- Set weights as a group and write one sentence for each.
- Define gates, which are fail conditions that no score can offset.
- Write the test script for each criterion.
- Run the same script for each product in the same week.
- Record score, evidence level and version.
- Compute totals, apply gates, and write the open questions.
- Present the results with the evidence column visible.
Common mistakes
- Weights set after seeing results. This lets preference steer the math.
- Scoring features, not controls. "Has a sandbox" is not a score. "Egress denied by default, tested" is.
- Mixing price into the security total. Keep commercial terms separate.
- Reusing last year's scorecard. Products change, so refresh criteria each cycle.
- Hiding the evidence. If readers cannot see how a number was earned, they will argue about the number.
Using the scorecard after purchase
Keep it. It becomes the baseline for renewal: re-run the tests, compare, and ask the vendor about any drop. It also gives you a ready answer when someone asks how the tool was selected.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how controls map to evidence, and which records you would keep for your own review.
Quick check
How many vendors should we score?
Two or three is manageable. More than that and the testing quality drops.
Should price be a criterion?
Handle commercial terms in a separate review so the security and capability scores stay clean.
What if two products tie?
Look at the gates and open questions first, then at which vendor answered in writing and fastest. Ties often break on responsiveness.
Last reviewed: 19 October 2026.
