How do you evaluate SAST auto-fix tools before trusting them?

Evaluate a SAST auto-fix tool by running it on your own code and scoring each proposed patch on four things: does it remove the weakness, does it keep behaviour, is the change small enough to review, and do your tests pass. A tool that fails on any of these costs reviewers time instead of saving it.

What does auto-fix actually do?

Static analysis finds a pattern that maps to a weakness class, such as SQL injection (CWE-89). Auto-fix goes one step further and proposes a code change, usually as a pull request. Some use rules, some use language models, some combine both.

The idea is sound because many findings have well-known fixes: parameterise the query, encode the output, validate the path. The risk is a fix that looks right and is not.

Why not just compare feature lists?

Feature lists tell you what a tool claims. They do not tell you how it behaves on your frameworks, your coding style or your test suite. Any ranking from outside your environment answers someone else's question, so this guide gives you a method instead of a list.

What should you score?

Pick 30 real findings from your backlog across at least three weakness classes. For each proposed fix, record:

Correctness. Does the weakness actually go away? Re-run the analysis, then write or run a test that attempts the bad input. A finding that disappears because the tool moved code out of the analyser's view is not fixed.

Behaviour preserved. Do existing tests pass? Are there tests covering the changed lines at all? If not, the pass proves little.

Scope. Did the patch change only what it needed? A twenty line fix for a one line problem hides risk. Small diffs get reviewed carefully.

Explainability. Does the pull request say what the weakness is, why the change fixes it and what to test? Reviewers act on explanations.

Also count the findings the tool declines to fix. Refusing is a good behaviour when the fix is unclear.

A constructed example

A team tests a tool on 30 findings. Twelve are SQL injection, ten are cross-site scripting and eight are path traversal.

• 21 patches pass tests and remove the weakness on retest.
• 5 pass tests but do not remove the weakness, for example escaping the wrong context.
• 3 break behaviour and fail tests.
• 1 changes an unrelated file.

Result: 70 percent fully correct, with a visible failure mode in output encoding. The team decides to allow auto-fix proposals only for injection fixes with test coverage, and to require a human review on everything. They also learn their cross-site scripting findings need manual work.

What controls should wrap any auto-fix?

1. The tool opens pull requests and never merges.
2. Branch protection requires review and passing checks.
3. Findings get a second scan after the patch.
4. Auto-fix scope is limited by weakness class, repository and severity.
5. Every change is traceable to a finding ID.

Safeguard offers autonomous remediation that creates fix pull requests, off by default, so you decide where it runs. Pair it with reachability analysis so effort goes to exposed code first.

How often should you repeat the evaluation?

Repeat it whenever the tool, its model or your main framework changes significantly, and at least once a year. Keep the 30 findings and the scoring sheet so results compare over time. If correctness drops on one weakness class, narrow the scope for that class until it recovers. Share results with developers, since their trust in automated pull requests depends on seeing honest numbers, including the failures. Also track reviewer time per automated fix. A tool that is correct but takes longer to review than to write by hand is not saving effort.

How to check or apply this

1. Sample 30 real findings across several weakness classes.
2. Score each patch for correctness, behaviour, scope and explanation.
3. Record how many were declined and why.
4. Decide scope rules from the results, by class and repository.
5. Keep human review and branch protection on.
6. Re-run the exercise after a major tool or model update.

Quick check

Sources

• CWE, Common Weakness Enumeration, MITRE
• OWASP Application Security Verification Standard, OWASP
• NIST SP 800-218, Secure Software Development Framework, NIST

Last reviewed: 18 October 2026.