Run an AI security tool proof of concept by fixing the scope, writing pass criteria before you start, testing on your own systems, and deciding against that written bar at the end. Four weeks is enough. The most common mistake is letting the vendor choose the data and the questions.
Week by week
| Week | Focus | Output |
|---|---|---|
| 0 | Scope and criteria | One page: systems, users, pass bar, deal breakers |
| 1 | Setup | Tool connected to one repo, one team, one agent or server |
| 2 | Planted tests | Results for known problems you seeded |
| 3 | Real use | Notes from the people who live with it daily |
| 4 | Decision | Scored table and a yes, no or extend |
Write the criteria first
Put three kinds of criteria on the page.
- Must pass. Two or three things without which the answer is no. Example: it must show tool arguments in the audit record.
- Should pass. Things that improve the case but do not decide it.
- Measured. Numbers you will collect: setup time, number of findings you judged correct, number you judged noise, time to triage one finding.
Keep a rule against changing the criteria after week one unless everyone agrees and notes why.
Seed known problems
A tool that finds things you already know about is easier to judge than one that finds things you cannot check. Plant a few. Add an unlisted tool server, a dependency with a known advisory, a prompt carrying an instruction hidden in a document, and an over-broad token in a test project. Then see which ones the tool finds, which it misses, and how it describes each.
For findings it raises that you did not plant, have an engineer verify a sample. Count how many were real, how many were wrong, and how many were true but irrelevant.
Involve the people who will use it
Ask two developers and one security analyst to use the tool for their normal work in week three. Give them three questions: what did it help you do, what slowed you down, and what would make you turn it off? A tool nobody wants to keep installed will not protect anyone.
Decide with the data
At the end, score each criterion as pass, partial or fail. Do not hide a deal breaker behind a good average. If you extend, set the date and the single question the extension must answer.
Related reading: the weighted scorecard method and, for judging scan results, how to test reachability claims in a proof of concept.
Try it yourself
Click through it at your own pace, or open it full screen.
Run through the flow and note which of your criteria you could judge from it and which need your own data.
Safeguard offers a free scan that can serve as a quick first data point before a larger proof of concept.
Quick check
How many systems should a proof of concept include?
One team, one repository and one agent or server is plenty. A narrow scope you can measure beats a wide one you cannot.
What if the vendor insists on running the demo?
Accept a guided session for setup, then run the planted tests yourself.
Last reviewed: 19 October 2026.
