Mutation testing makes small automatic changes to your code, called mutants, and runs your tests against each one. A mutant that no test catches shows a gap. It is a direct way to check AI written tests, because it measures whether they fail when behavior changes.
How it works
A mutation tool edits the source in tiny ways: swapping a plus for a minus, flipping a boundary, deleting a statement. For each edit it runs the tests. If a test fails, the mutant is killed. If every test still passes, the mutant survived, and that points to behavior nobody checks.
The survival rate tells you something line coverage cannot. Coverage records that a line ran. Mutation testing records whether anything would have noticed if the line were wrong.
Why it fits AI generated tests
Generated tests are cheap to produce and often reach high coverage quickly. Martin Fowler warns that coverage is of little use as a numeric statement of how good tests are, and that when a level of coverage becomes a target, people will try to attain it. An assistant will happily hit a target without testing much.
Mutation testing pushes back on that. If the generated tests let many mutants survive, you have a measured reason to rewrite them.
The cost problem and one answer
Running the full suite for every mutant is slow. A paper on mutation testing at large scale describes a way around it. It mutates only changed code during code review, filters out mutants likely to be irrelevant to developers, caps the number per line and per review, and picks mutants based on how each kind of mutation has performed before. The authors report an evaluation covering more than 24,000 developers on more than 1,000 projects, with orders of magnitude fewer mutants.
You can borrow the idea at small scale. Run mutation testing only on the files in a pull request, and only for the functions the AI touched.
A practical setup
- Pick a mutation tool for your language and run it locally first.
- Limit it to changed files in the pull request.
- Triage survivors by hand. Some mutants are equivalent to the original code and cannot be killed, so a perfect score is not the goal.
- For each real survivor, add or tighten an assertion.
- Do not turn the score into a hard gate on day one. Use it as a review aid first.
What it cannot do
Mutation testing checks tests against the code as written. It cannot tell you that the requirement itself is wrong. It also will not find missing features. Keep human review of design and intent.
Quick check
What does a surviving mutant mean?
The tool changed the code and every test still passed. Either the behavior is untested, or the mutant is equivalent to the original and cannot be caught.
How do you keep mutation testing fast?
Mutate only the code changed in the pull request and cap the number of mutants, as the large scale approach in the cited paper does.
Sources
Last reviewed: 19 October 2026.
