Score AI coding agents in two passes. First, pass or fail on must have security controls. Second, rate the survivors on weighted criteria such as isolation, logging, administration, and developer experience. Keep the weights agreed before anyone sees a demo, so enthusiasm does not rewrite the rules.
Pass one: gates
Gates are things a tool must do for your team to consider it at all. Examples that suit many security-conscious teams:
- Written statement of where prompts and code are sent and how long they are retained.
- A way to require approval for command execution, enforced by the tool.
- An option to keep sensitive repositories out of scope.
- Exportable records of actions.
- A documented way for an administrator to disable the tool for a user or group.
If a candidate misses a gate, record whether a compensating control could close it. If not, stop there.
Pass two: weighted criteria
Here is a template. Change the weights to fit your risk, but fix them in advance.
| Criterion | Weight | What a 5 looks like |
|---|---|---|
| Data handling controls | 20 | Configurable retention, clear terms, regional options you need |
| Execution control | 20 | Granular allowlists, centrally managed |
| Isolation support | 15 | Runs cleanly in containers, no hidden ambient access |
| Audit and logging | 15 | Per action records, exportable, tied to identity |
| Administration | 10 | Central policy, user groups, quick disable |
| Extension governance | 10 | Controls over connected servers and plugins |
| Developer experience | 10 | Developers choose it voluntarily |
Multiply score by weight, add up, and divide by the total. Resist adding twenty criteria. Seven to ten is enough to discriminate.
How to get honest scores
- Test, do not ask. For each criterion, run a task. Our post on choosing an AI coding agent lists five that reveal behavior.
- Score blind where possible. Have two people score independently, then compare. Large gaps point to unclear evidence.
- Record evidence with each score. A link to a screenshot, a log excerpt, or a documentation page. Scores without evidence get relitigated.
- Include a skeptic. Someone whose job is to find the failure improves the result.
Use ranges, not false precision
A score of 3.7 against 3.8 is noise. Treat results within half a point as ties, and break ties with the factors the scorecard cannot capture: contract terms, support, and your team's appetite for the migration effort.
Keep the scorecard alive
Tools change monthly. Set a date to re-score the chosen tool and one alternative every six months. Also trigger a re-score on events: a major feature, an ownership change, or a security advisory involving the tool. The OWASP Top 10 for LLM Applications was reissued in 2025, which is a reminder that the threat list itself moves, so revisit your gates too.
Share the outcome
Publish the result internally with the criteria, the evidence, and the exceptions. People accept a decision more readily when they can see how it was made. It also gives the next evaluation a head start.
Safeguard can help at the dependency and pipeline end of this: SBOM and dependency scanning, reachability analysis, policies and gates on pull requests, and agent identity. You can try a free scan at gold.safeguard.sh/scan.
A worked example: scoring two candidates
Two candidates reach the second pass. The team uses the template weights from above. Scores run from 1 to 5.
| Criterion | Weight | Candidate A | Candidate B |
|---|---|---|---|
| Data handling controls | 20 | 4 | 3 |
| Execution control | 20 | 5 | 3 |
| Isolation support | 15 | 4 | 5 |
| Audit and logging | 15 | 3 | 4 |
| Administration | 10 | 4 | 2 |
| Extension governance | 10 | 3 | 3 |
| Developer experience | 10 | 4 | 5 |
Weighted totals work like this. For A: 20 times 4 is 80, 20 times 5 is 100, 15 times 4 is 60, 15 times 3 is 45, 10 times 4 is 40, 10 times 3 is 30, 10 times 4 is 40. The sum is 395, which divided by 100 gives 3.95. For B: 60, 60, 75, 60, 20, 30, 50 sum to 355, or 3.55.
A leads by 0.4, which is within the range where you should look closely rather than declare a winner. The detail that matters is that B scores 2 on administration. If central policy is a gate for your team, B might fail pass one regardless of the total. This is exactly why the gates come first.
Handling disagreements between scorers
Two scorers will not always agree. When scores differ by two or more points on a criterion, discuss the evidence rather than averaging. Often one person tested something the other did not. Write the resolution next to the score. This habit also improves the next evaluation, because the evidence notes show what to test.
Weights are a values statement
Changing the weights changes the winner, so agree them before you see results. A regulated team might raise logging and administration. A team with strict data residency needs might double the data handling weight. A small startup might give developer experience more weight, because a tool nobody uses protects nothing. Record why the weights are what they are.
Evidence log
| Criterion | Evidence kept |
|---|---|
| Data handling | Vendor statement with date, screenshot of the setting |
| Execution control | Recording of the chained command test |
| Isolation | Notes from the container run |
| Logging | Sample export file |
| Administration | Screenshot of group policy |
| Extensions | List of connected servers and how they are restricted |
| Developer experience | Short survey of pilot users |
After the decision
Share the scorecard with leadership as a one page summary. Note the exceptions, the compensating controls, and the date when you will re-score. If a candidate with a lower total is chosen for a good reason, such as a contract term, say so openly.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through and notice how one set of evidence supports several requirements at once.
Quick check
Why separate gates from weighted scores?
Weighted scores let a strong feature hide a weak control. Gates stop a missing must-have from being averaged away.
How many people should score?
At least two independent scorers, from different functions, then a short reconciliation meeting.
What if two tools score within half a point?
Treat it as a tie and decide on the factors the scorecard cannot capture, such as contract terms, support, and migration effort.
Sources
Last reviewed: 19 October 2026.
