Build the test set from your own code: 30 to 60 small changes, each labelled by a human as vulnerable or safe, including near-identical safe twins of every vulnerable case. Run the AI reviewer on all of them and count what it flags wrongly, what it misses, and whether its explanations are correct.
Why a vendor demo is not a test set
A demo repo is chosen by the vendor, so the tool has usually seen something like it. Your code has your frameworks, your wrappers and your habits. A finding that looks obvious in a tutorial looks different when input passes through three internal helper functions before reaching a query.
A test set you own also gives you something repeatable. You can run it again after a model update, after a configuration change, or against a second tool, and compare like with like.
What goes in the set
Aim for a mix. A rough split that works for a first pass:
| Case type | Share | Purpose |
|---|---|---|
| Real past vulnerabilities, reduced to a small diff | 30 to 40 percent | Does it catch what actually hurt you? |
| Safe twins of those cases | 30 to 40 percent | Does it stay quiet when the fix is present? |
| Plain, harmless changes (renames, tests, docs) | 15 to 25 percent | Measures background noise |
| Tricky safe code that looks dangerous | 10 percent | Probes for false alarms |
Safe twins matter most. A tool that flags every string concatenation near a query will catch every SQL injection case, and look great, until you notice it flags the parameterised version too. The weakness class is easy to name: CWE-89 describes building an SQL command from external input without neutralising special characters. The safe twin uses a parameterised call and should pass quietly.
Include hard-coded credentials too (CWE-798), using obviously fake values, so you never put a real secret into a trial.
How to build and label it
- Pull 10 to 15 past security fixes from your history. Reduce each to the smallest diff that shows the flaw.
- Write the fixed version as the safe twin.
- Add plain changes from recent merged pull requests.
- Have two people label each case independently. Settle disagreements in writing. A case nobody can agree on is a bad test case, so drop it.
- Record the expected answer and the line where the problem sits.
Strip real secrets, customer data and internal hostnames before the set leaves your network. If the trial is a hosted service, assume the set will be transmitted.
How to score
For each case, record flagged or not, whether the flagged line is right, and whether the explanation is accurate. Then compute four counts: caught vulnerable cases, missed vulnerable cases, wrong flags on safe cases, and wrong flags on plain changes. Keep the raw table, because a leadership summary hides the interesting misses.
Do not chase a single percentage. With 40 cases, one case moves the number by 2.5 points. Read the misses one by one and ask whether they matter for your risk.
Keep it alive
Add every real escape (a bug that reached production) as a new case. Rerun the set when the vendor announces a model change or you change settings. A set that grows with your incidents becomes the most honest evidence you have about the tool.
Try it yourself
Open the demo and look at how a static code scan reports a finding with its location and explanation. Use it as a model for what to record per case.
Click through it at your own pace, or open it full screen.
Quick check
How many test cases are enough?
Thirty to sixty is enough to expose obvious problems and compare two tools. It is not enough to quote a precise rate, so treat the result as a screen, not a benchmark.
Can I use public vulnerable sample apps instead?
You can add a few, but a model may have seen them. Your own reduced cases are a fairer test.
Should real secrets be in the set?
No. Use fake values that match the format, and strip internal names before anything is sent to a hosted tool.
Safeguard can scan a repository for dependency and code issues, and the free scan at gold.safeguard.sh/scan is a quick way to get a second opinion on a repo while you build your own set.
Sources
Last reviewed: 19 October 2026.
