Pick ten vulnerabilities your team already fixed by hand, rewind the code to before each fix, ask the AI tool to remediate, and compare its output with what you shipped. Score on whether it fixed the root cause, stayed in scope, and passed an honest test.
Why backtest
Vendor demos and public benchmarks show what a tool does on someone else's code. Your repositories have your frameworks, conventions and odd corners. A backtest uses ground truth you already have: a known flaw and a known good fix. It costs a few hours and tells you more about your own fit than reading marketing pages.
It is also a safe way to learn how the tool fails, before a failure reaches a production branch.
Design the set
Choose ten items with variety:
- Three dependency upgrades (patch, minor, one major).
- Four code fixes across different weakness classes, such as injection, access control and input handling.
- Two configuration changes.
- One that your team found hard.
Avoid picking only the easy wins. If every case is a one line version bump, the result will flatter the tool.
Run it cleanly
- Create a branch at the commit just before each original fix.
- Give the tool the same information a live run would have: the finding, not your solution.
- Do not coach. One attempt, plus at most one clarifying round if you allow it, applied equally to every case.
- Keep the original fix out of reach, including in the commit history the tool can see.
- Record the tool and version used, since results can change with updates.
Score each case
Use a simple sheet, one row per case.
| Criterion | Scoring |
|---|---|
| Root cause addressed | Yes, partly, no |
| Matches or beats the human fix on scope | Yes, no |
| An honest test exists (fails before, passes after) | Yes, no |
| Facts correct (versions, packages) | Yes, no |
| Review time to reach a decision | Minutes, noted |
| Outcome | Would merge, would edit, would reject |
OWASP's guidance on secure coding with AI names several failure modes worth scoring directly: out of scope edits, weakened or deleted tests, and outdated or nonexistent package versions. Add a column for each.
Interpret with care
Ten cases are a small sample. Do not report a percentage as if it were stable. Use the exercise to find qualitative patterns: which weakness classes went well, which went badly, which failure modes appeared. If two of the four code fixes weakened tests, that tells you something even without a statistic.
Also note the review time. A tool that produces a plausible fix you must spend an hour verifying may save little. Capture that cost honestly.
Turn results into policy
Use the findings to set approval tiers. Categories that went well can have lighter review. Categories that failed can be restricted to suggestions only, or turned off. See approval tiers for automated fix PRs.
Repeat the backtest when you change tools, models or major settings.
Safeguard's autonomous remediation is off by default with every category on Manual, which makes it straightforward to trial one category at a time against a backtest like this one. You can start with a free scan at gold.safeguard.sh/scan.
Quick check
Can I backtest with public vulnerabilities instead of my own?
You can, but your own history is better because it reflects your code and conventions. Public cases work as a supplement.
Is ten cases enough?
For a first look at failure modes, yes. For a reliable rate, no. Add cases over time.
Sources
Last reviewed: 19 October 2026.
