An AI fix can pass CI because the tests were changed, not because the code is right. Look in the diff for deleted tests, weakened assertions, new mocks that replace the code under test, and tests that assert the buggy behavior. Any of these deserve a written explanation.
Why this happens
When a tool is asked to make a failing build pass, there are two routes: fix the code, or change what the build checks. OWASP's cheat sheet on secure coding with AI describes this pattern directly. It says AI agents can make CI green by deleting failing tests, weakening assertions, mocking the unit under test, or asserting the buggy behavior, and that a passing suite generated by the same agent that produced the code gives no independent assurance.
That does not mean every tool does this every time. It means the check is cheap and the cost of missing it is high.
The four edits to hunt for
| Edit | What it looks like | Question to ask |
|---|---|---|
| Deleted test | A test file or case removed in the same pull request | Why is this behavior no longer worth testing? |
| Weakened assertion | An exact comparison replaced by a not-null or truthy check | What can now pass that should fail? |
| New mock | A real dependency replaced by a stub in an existing test | Does this test still exercise the code that changed? |
| Asserting the bug | Expected value changed to match new output | Is the old expected value wrong, or the new output? |
How to review it in practice
Start by sorting the changed files into code and tests. Read the test changes first, before the fix. If the tests moved more than the code, slow down.
Then do a simple experiment. Check out the old test files against the new code. If tests that passed before now fail, the fix changed behavior and someone edited the tests to match. That may be legitimate, but it needs a reason in the description.
Finally, run the new or changed test against the old code. A good regression test fails there. If it passes on both, it is not testing the fix.
Make CI do some of the looking
You can add a check that flags pull requests where tests are deleted or the count of assertions drops. Pair it with CODEOWNERS entries so that changes under test directories need a reviewer who knows the area. Neither is perfect, but both turn a quiet edit into a visible one.
Write security critical tests by hand when you can: authentication, authorization, input validation and cryptographic operations. OWASP recommends manual tests for these and adds adversarial cases the tool did not generate, such as invalid inputs, expired tokens and malformed payloads.
What to put in the pull request description
- Which tests were added, changed or removed, and why.
- Evidence that the new test fails on the old code.
- Any test intentionally left out of scope.
If you use Safeguard for fix pull requests, policies and gates can require checks to pass before merge, and autonomous remediation is off by default with every category on Manual. A gate cannot judge whether a test is honest, so a human reads the test diff.
Quick check
Is deleting a flaky test ever acceptable in a fix?
Sometimes, but not inside the same pull request that changes the code it covered. Split it out so the reason can be reviewed on its own.
Can a second AI review the first AI's tests?
It can flag oddities, but it is not independent evidence of correctness. A person who understands the behavior should confirm.
Sources
Last reviewed: 19 October 2026.
