A generated test is only worth keeping if it fails when the code is wrong. To check, break the code on purpose and see whether the test notices. If it still passes, it was decoration, however neat it looks and whatever it adds to your coverage number.
Why this check comes first
Google's public code review guidance includes one question for the tests in any change: will the tests actually fail when the code is broken? It is the right question for generated tests too. A model asked to write tests for a function often reads the function and asserts what it already does. That produces a test that agrees with the code, bugs included.
The five minute audit
For each generated test file, do this:
- Run the tests and confirm they pass.
- Make a small deliberate break in the code under test. Flip a comparison, remove a branch, return a constant.
- Run the tests again.
- If nothing fails, the test does not protect that behavior.
- Revert the break.
Repeat with two or three different breaks per function. This is manual mutation testing, and it needs no tooling.
Signs of a weak generated test
- It asserts only that no exception was thrown.
- It mocks the thing it claims to test, so the assertion checks the mock.
- It copies the implementation logic into the expected value.
- It has one happy path input and no boundary cases.
- It asserts on a whole serialized object when only one field matters, so it breaks on any change.
What good ones look like
A strong test states an expectation from the specification, not from the code. Give the model the requirement or a short list of cases in plain words, and ask it to write tests from that. Then compare its cases to your own list. Where the code and the requirement disagree, the test should fail, and that is the point.
Keep the human decision
Treat generated tests like any contribution. Google's guidance reminds reviewers that tests are code too and should be kept simple. Read each one, delete those that fail the audit, and keep the rest.
Use tools when volume grows
When you generate dozens of tests, automate the break and check loop with a mutation testing tool for your language. That is the next post in this series: mutation testing to check AI written tests.
A note on speed
The audit sounds slow, but it scales down. Ten minutes per generated file catches most weak tests. Do it before the pull request goes to review, so the reviewer sees tests that already passed the check. If the same weakness shows up repeatedly, such as missing boundary cases, add that to your prompt template.
Quick check
Is a passing generated test evidence that the code is correct?
No. A test written from the code can simply restate the code. It only counts if it would fail when the behavior is wrong.
What is the cheapest way to audit a generated test?
Break the code under test on purpose, rerun, and see whether the test fails.
Sources
Last reviewed: 19 October 2026.
