Success criteria to write before a security tool proof of concept

Success criteria are measurable statements you write before the trial, such as "finds at least 8 of the 10 planted flaws" or "adds under 3 minutes to a pull request check." They protect you from moving the goalposts when a demo impresses, and they give vendors a fair target.

Why before, not after

Once a vendor has spent a month in your environment, people become attached. Stakeholders see a nice dashboard and quietly redefine success. Criteria written in advance, signed off by the buyer and the team lead, stop that.

Good criteria have three parts

A subject, a number, and a method of measuring. "Good accuracy" fails all three. "Finds at least 8 of 10 known flaws in the test repository, counted from the exported report" has all three.

Examples by area

Detection

• Finds at least a set number of the known issues in your test repositories
• Reports each known vulnerable dependency with the correct package and version
• Detects a newly published vulnerability in an already scanned project within a time window you choose, without a new scan

Noise

• No more than a set share of top priority findings are judged false or unreachable by your reviewers (see measuring false positives)
• Duplicate findings across repositories are grouped, not repeated

Speed

• Scan of your largest repository completes within a stated time
• The pull request check adds no more than a stated number of minutes

Measure these on your own runners, not on the vendor's machines.

Developer adoption

• A pilot team uses the tool in pull requests for two weeks, and a stated share of developers rate it usable
• Developers can suppress a finding with a reason without asking a security engineer

Integration

• Single sign-on works with your identity provider
• Findings flow to your ticket system with the fields you need
• A report can be exported in the format your auditors accept

Operations and trust

• Vendor answers support questions within the time they promised
• The vendor passes your data handling gate

How to choose thresholds

Anchor them in your current pain. If your last tool produced 200 alerts per week and your team could handle 30, then "noise under 30 actionable alerts per week" is concrete. If you have no history, use the pilot to set a baseline, then decide, but write down that you are doing so.

Avoid thresholds that only one vendor can meet because it matches their product design. Ask someone outside the evaluation team to read the list and spot bias.

Include a stop rule

Write what causes a vendor to stop early: failing a gate, missing setup by a stated day, or being unable to scan your main language. Stopping a weak candidate in week 1 saves your team's time.

Record results the same way

Use one sheet. One row per criterion, one column per vendor: result, evidence link, pass or fail. Then feed the pass or fail totals into the weighted rubric.

Quick check

Last reviewed: 19 October 2026.