A flaky security check is one that gives different results on the same code, for example failing and then passing when re-run on the same commit. Measure flake rate as the share of re-run jobs whose outcome changed with no code change. Fix the cause before developers learn to ignore the check.
Why flakiness is a security problem
Flaky tests are an old complaint. Flaky security checks are worse. If a check fails at random, developers re-run until it goes green, and the habit spreads to real failures. A gate that people retry out of reflex is no longer a gate.
Common causes
- A vulnerability data feed that is updated between two runs, so the same dependency is clean at 9am and flagged at 11am. This is not really flakiness, it is new information, but it feels the same to a developer.
- Network timeouts when the scanner fetches data or resolves packages.
- Unpinned scanner or rule versions that change between runs.
- Resource limits on shared runners.
- Non-deterministic ordering or sampling in the tool itself.
The first cause is worth separating. A changed result because of new advisory data is correct behaviour, and you should explain it in the failure message. The rest are real flakiness.
Define the measurement
On GitHub, a workflow run can be re-run for up to 30 days after the initial run, at most 50 times, and a re-run uses the same commit SHA and ref as the original. That makes re-runs a clean signal: same code, second attempt. GitLab and other systems offer retries in similar ways.
Use this definition:
- Take all security job attempts on the same commit.
- Keep commits that had at least two attempts.
- Flake rate is the share of those commits where attempts disagreed.
Record the scanner version, rule version and data feed date on every attempt. If the data feed date differs, classify the disagreement as a data change, not a flake.
Interpreting the rate
Do not chase a magic number. Look at the trend and the top offenders. A single job or repository usually causes most of the noise. Sort by the number of disagreeing commits and work from the top.
Check also what happens to the second attempt. If almost every retry passes, developers are right to retry, and your gate has a reliability problem. If retries usually fail again, the failures are probably real.
What to do about it
- Pin the scanner and rule versions in the pipeline, and update them deliberately.
- Cache data feeds, and show the feed date in the job log.
- Add bounded retries inside the job for network calls, and log each retry.
- If a check stays unreliable, move it to report-only mode, fix it and then re-enable it. Quarantine with an owner and a date, not forever.
- Tell developers how to report a suspected flake, and answer quickly.
Quick check
Should I auto-retry a failed security job?
Retry only for infrastructure errors such as network timeouts, and log the retry. Never auto-retry a job that failed because of a finding, since that hides the signal.
Is a new advisory failing an old commit a flake?
No. The code did not change but the knowledge did. Say so in the message and record it as a data change.
How Safeguard fits
Safeguard's policies and gates work on dependency and SBOM scan results, and pinned inputs make those results easier to reproduce. A free scan is available at gold.safeguard.sh/scan.
Sources
- GitHub Docs, Re-running workflows and jobs (30 day window, 50 re-run maximum, same SHA and ref): https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/re-running-workflows-and-jobs
Last reviewed: 19 October 2026.
