Retry only failures that can pass on a second try, such as timeouts and rate limit responses. Use exponential backoff with jitter, cap the attempts, and send exhausted jobs to a dead-letter queue. Make every job idempotent so a retry or duplicate never double counts findings.
Sort failures first
Not every failure deserves a retry. Retrying a permanent failure just burns workers.
| Failure | Transient? | Action |
|---|---|---|
| Source host rate limit | Yes | Wait for the reset or retry-after time, then retry |
| Network timeout fetching code | Usually | Retry with backoff |
| Worker crashed or was evicted | Yes | Job becomes visible again, retry |
| Repository not found or access revoked | No | Mark failed, notify owner |
| Unsupported or corrupt archive | No | Dead-letter with the error |
| Scanner crashed on this input every time | No after two or three tries | Dead-letter, attach logs |
Record a failure class on each attempt. That data later tells you whether most of your failures are your infrastructure or your inputs.
Backoff done properly
Exponential backoff means each wait is longer than the last, for example doubling. Jitter adds randomness so a thousand failed jobs do not all retry at the same instant. GitHub's own guidance for its API says to wait an exponentially increasing time between retries after a secondary rate limit and to stop after a set number of retries, and to honour the retry-after header when present. Apply the same logic to any external dependency your scanners call.
If the failure is a shared dependency, such as the source host being down, do not retry per job. Open a circuit breaker so all workers pause together and resume with a probe.
Idempotency
An idempotent job gives the same stored outcome whether it runs once or three times. Three tactics help.
- Give each job a dedupe key built from the target, scan type and config version.
- Write results with an upsert on that key rather than an append.
- Make side effects safe to repeat. A pull request comment should be updated in place, not posted again.
Without these, a visibility timeout that fires slightly early causes two workers to run the same job, and the developer sees duplicate comments and double findings.
Dead letters are a work queue
A dead-letter queue that nobody reads is a trash can. Assign an owner, alert on depth and age, and review it weekly. Each entry should hold the job, the failure class, the number of attempts and the last log excerpt. Add a replay action, so once you fix the cause you can push jobs back to the main queue.
Limits to set
- Maximum attempts, commonly three to five, per failure class.
- Maximum total time before a job is declared stale.
- Maximum backoff delay, so a job does not wait for days.
- A cap on replays from the dead-letter queue, to avoid loops.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how repeated checks over time produce a single, updating view rather than duplicate entries.
Safeguard handles job scheduling and retries as part of its hosted scanning, so teams consuming it do not build this. You can try the free scan at https://gold.safeguard.sh/scan.
Quick check
How many retries are enough?
Three to five for transient failures is common. Past that, a human or a code fix is needed.
Why add jitter?
Without it, many failed jobs retry together and recreate the overload that caused the failure.
Sources
Last reviewed: 19 October 2026.
