How to choose the code you test security tools on in a pilot

Use a mix of three kinds of code in a pilot: a few of your own representative repositories, one or two repositories with planted flaws you control, and a public benchmark. Each answers a different question, and relying on only one gives a misleading result.

Your own repositories

Real code shows how a tool handles your languages, build quirks, private registries, and monorepo layout. It also reveals the noise level you will live with.

Choose repositories that vary:

• Your most important service
• A legacy project that nobody wants to touch
• A recently created project using current frameworks
• One repository with a complicated build, such as multiple languages or generated code

Get approval from the owners and check each vendor's data handling terms before sharing anything sensitive. If the code cannot leave your network, ask whether the tool can run locally.

The weakness of real code is that you do not have a complete list of what is wrong with it. You cannot measure misses without an answer key, so build one for a subset, as described in measuring false positives and misses.

Planted flaws

Create a small repository, or branch, with flaws you add yourself and record in a sheet: a dependency pinned to a version with a known vulnerability, a hardcoded test credential, an injectable query, and so on. Because you know every item, recall is easy to compute.

Make a few of them tricky on purpose:

1. A vulnerable dependency that is imported but never called
2. A vulnerability in a transitive dependency three levels down
3. A flaw hidden behind a wrapper function
4. A fixed version that exists, and one that does not

This also tests whether reachability analysis and fix suggestions behave sensibly. Keep the planted repository private and do not reuse it forever. Vendors tune to well known test code.

Public benchmarks

Public suites give you a neutral reference that every vendor can run. The OWASP Benchmark, for example, is an open source web application for SAST, DAST, and IAST tools, with a stated goal that every planted flaw is genuinely exploitable. Its current release was v1.2 when we checked. Check that your language is covered: the page describes Java and Python test suites.

Benchmarks have a known weakness. Vendors can tune to them, and they rarely resemble your architecture. Use the result as a sanity check, and give it a small weight in your score.

Match the corpus to the tool type

• For dependency scanning, include lockfiles of different ecosystems and at least one with a very large dependency tree.
• For static analysis, include code in each language you use, with and without a framework.
• For dynamic testing, you need a running test application with authentication. See the authenticated DAST checklist.
• For container scanning, use images with an old base and images built from minimal bases.

Keep it fair

Give every vendor exactly the same code, the same time window, and the same instructions. Freeze the corpus before the trial starts. If you change it, tell everyone and rerun.

Sources

• OWASP Benchmark project

Quick check

Last reviewed: 19 October 2026.