Turn each false positive report into a permanent test: copy the smallest piece of code that triggers the wrong result, label what the correct result is, add it to the rule's test suite, then fix the rule until the test passes. Over time the suite becomes a regression corpus for that rule.
Definitions
- Regression corpus: a set of code samples with known correct outcomes, kept so old mistakes are not repeated.
- True positive: the rule reports real risk.
- False positive: the rule reports something that is not a problem.
- False negative: the rule is silent where it should have reported.
Both error kinds belong in the corpus. A fix for a false positive that creates a false negative is a common way rules decay.
The capture routine
- Collect the report. Ask developers to link the finding, not paste a description.
- Reduce it. Remove everything that does not affect the result until a few lines remain. This also helps you understand the cause.
- Remove private detail. Replace real names, hosts and values with neutral ones.
- Label it. Valid means no report expected. Invalid means a report is expected.
- Add it to the tests. In ESLint this is an entry in the valid or invalid list of RuleTester. In Go, it is a file in testdata, where a line with no want comment asserts that nothing is reported.
- Write a one line reason. A comment such as "value comes from a constant" saves the next person an hour.
- Fix and run everything. All earlier cases must still pass.
Keep a counter-sample
For every false positive you fix by narrowing a rule, add a nearby sample that should still be reported. This is the guard against over-correcting. If a developer says "it flags my safe call", the sibling test says "but it must still flag the unsafe one".
Structure on disk
| Folder | Contents |
|---|---|
| flagged | Code the rule must report |
| allowed | Code the rule must stay silent on |
| edge | Unusual syntax: aliases, nesting, generated code |
| from-reports | Cases traced to a real ticket, with the ticket id in a comment |
Public benchmarks
Some projects publish test suites for tools. The OWASP Benchmark is a Java and Python test suite designed to check the speed and accuracy of vulnerability detection tools, where the vulnerabilities it scores are meant to be exploitable. It is useful for general scanner evaluation, but a corpus drawn from your own tickets reflects your code and your frameworks, which no public suite can.
Measure with it
Because every case has a label, you can compute how many the rule gets right after each change. Track it over time. A drop after a change tells you the change did harm, before developers do.
Quick check
How many samples should a corpus have?
Start with one per report and grow naturally. A small corpus of real cases is worth more than a large synthetic one.
Should I include code from customers or private repositories?
Not as is. Reduce it and replace identifying details before it goes into a shared test suite.
Safeguard's scanning covers dependencies and SBOMs, and its reachability analysis helps show which findings your code can actually call. You can run a free scan on a repository.
Sources
- ESLint: Node.js API Reference (RuleTester)
- Go: golang.org/x/tools/go/analysis package documentation
- OWASP Benchmark Project
Last reviewed: 19 October 2026.
