Auto retry makes a red build green, but it can also hide a real race condition or a timing dependent security bug. Treat every flaky result as a signal to classify, not noise to suppress, and be strictest with tests that guard authentication, authorization and cryptography.
What flakiness looks like at scale
A 2016 post on the Google Testing Blog defines a flaky result as a test that shows both a passing and a failing result with the same code. The authors reported that about 1.5 percent of all test runs in their corpus were flaky, and that almost 16 percent of their tests had some level of flakiness.
Those figures are from one large company in 2016 and are not a benchmark for your team, but the mechanism applies everywhere: with enough tests, something fails randomly on most builds.
The security risk of ignoring it
The same post notes that developers sometimes dismiss a failing result as flaky and later find it was a legitimate failure caused by the code. It describes the human tendency to ignore alarms once they cry wolf.
For security this matters because the bugs that behave randomly are often the worst ones. Race conditions in token redemption, session handling or balance checks produce intermittent failures first. A test that fails one run in twenty may be the only warning you get.
What the post says about mitigation
Google described several tactics: re-running only failing tests, re-running automatically on failure, and marking a test as flaky so it reports failure only after three failures in a row. It also described tools that quarantine tests with high flakiness. The authors point out that these reduce false alarms but encourage developers to ignore flakiness in their own tests, and that quarantining can mask a real race condition or other bug in the code under test.
A safer retry policy
- Retry once, and record every retry as a metric. A pass after retry is a yellow result, not green.
- Never auto retry tests tagged as security tests. Make a human look.
- Classify each flake: test bug, environment, or product bug. Track the share in the third group.
- Give quarantine a deadline, such as seven days, with an owner. An expired quarantine fails the build.
- Run a suspect test many times in a loop in a stress job to see whether it reproduces.
Common causes worth checking first
Shared state between tests, clocks and time zones, random seeds, network calls to real services, test order dependence and parallel runs that share a database. The post lists concurrency and reliance on non-deterministic or undefined behavior among the root causes.
Quick check
Should security tests be exempt from quarantine?
Keep them out of silent quarantine. If one must be disabled, require a named owner, a ticket and a date, and compensate with a manual check until it returns.
Sources
Last reviewed: 19 October 2026.
