A useful SAST scorecard has eight criteria, a three-level rating for each, and a note for evidence. Use the criteria OWASP publishes for source code analysis tools, rate every vendor on the same sheet after a pilot, and decide on weights only after the team agrees what matters.
The eight criteria
OWASP lists these selection criteria for source code analysis tools. They make a good skeleton.
- Language support. Does it cover the languages and versions you ship?
- Vulnerability coverage. Does it map to the OWASP Top Ten or an equivalent list you care about?
- Accuracy. False positive and false negative rates, and benchmark results if available.
- Framework understanding. Does it model the libraries and frameworks you actually use?
- Build and binary requirements. Does it need buildable code, or can it analyze what you have?
- Workflow fit. IDE plugins, CI integration, ease of setup.
- Cost. License model: per user, organization, application or lines of code.
- Output interoperability. Standards such as SARIF.
How to rate
Use three levels, not ten.
- Meets: works on our repositories without special effort.
- Partial: works with workarounds or on some repositories.
- Fails: does not work or evidence is missing.
Next to each rating, write the evidence in one line: "scanned all five pilot repos, two needed a build change" is evidence. "Sales said yes" is not.
Turning ratings into a decision
Avoid starting with percentages. Weighted sums look objective, but the weights are opinions, and picking them first lets the loudest person decide the result.
Instead, mark which criteria are must-haves. A tool that fails a must-have is out, whatever else it offers. Typical must-haves: supports your main language, runs in your CI, exports a format you can use.
Among the survivors, compare on the remaining criteria and on two practical signals from the pilot: the share of findings your triagers labeled real, and developer willingness to keep it on.
Add two rows OWASP does not list
Operational questions matter in the first year. Add them.
- Support quality. How fast did the vendor respond during the pilot?
- Data handling. Where does your code go during a scan and what is retained?
Run the scoring session
Hold one meeting with the people who ran the pilot: the triagers, one or two developers and whoever owns the budget. Fill the sheet together, criterion by criterion, and ask each person to say what evidence they saw. Disagreements usually come from different samples of the pilot data, so look at the same findings. Keep the meeting under an hour and write the result down before leaving the room, including any criterion where the group could not agree.
Keep the sheet alive
Review the scorecard after six months in production. Update it with what actually happened: scan failures, noise, fix rates. It becomes your evidence for renewal or replacement.
Quick check
Should I weight the criteria?
Only after agreeing on must-haves. Weights set early often hide a disagreement instead of settling it.
How many vendors should reach the pilot?
Two or three. More than that dilutes triage attention and slows the decision.
Sources
Last reviewed: 19 October 2026.
