A scan job queue should hold small, idempotent jobs keyed by what is being scanned, dedupe repeat requests, support priorities, and send repeated failures to a dead-letter queue. Workers pull jobs, scan, and write results. Keep scanning logic out of the queue so you can change either side independently.
What a scan job is
A scan job is a request to analyse one target at one point in time. The target might be a commit, a container image digest or a lockfile. The job carries the target reference, the scan type, the policy version and a requester. It should not carry the code itself, only a pointer the worker can fetch.
Small jobs matter. One job per repository is easy to reason about, but a monorepo with forty services makes that job huge and slow. Many teams split by scan type and by component so a failure in one does not discard the rest.
Design rules that prevent most incidents
| Rule | Why it matters |
|---|---|
| Idempotent jobs | A retry or a duplicate delivery must give the same result, not double the findings. |
| Dedupe key | Key on target plus scan type plus config version, so ten pushes in a minute become fewer scans. |
| Visibility timeout | A job that a crashed worker held becomes available again after a set time. |
| Priority lanes | Pull request scans should not wait behind a nightly full rescan. |
| Dead-letter queue | A job that fails repeatedly stops blocking others and gets a human look. |
| Bounded retries | Infinite retries turn one bad input into permanent load. |
Most queue systems offer at-least-once delivery. That means duplicates will happen, so idempotency is a design requirement, not a nice extra.
Steps to build the first version
- Define the job schema: target reference, scan type, config version, priority, requested by, requested at.
- Compute a dedupe key from the target, scan type and config version.
- On enqueue, if a job with the same key is waiting, replace its priority with the higher one and skip the new job.
- Set a visibility timeout longer than your p99 scan time, and have workers extend it for long scans.
- Write results keyed by the same dedupe key, so a repeated write overwrites rather than appends.
- Route jobs that exceed the retry limit to the dead-letter queue and alert on its depth.
Things teams forget
Cancellation. When a pull request gets a new commit, the scan for the old commit is wasted work. Let the enqueue step mark older jobs for the same branch as superseded.
Back pressure. If the queue grows without limit, scans finish hours late and developers stop trusting them. Decide in advance what happens at a depth threshold: shed low priority work, slow producers, or add workers.
Poison inputs. A malformed archive or a repository with millions of files can crash a worker every time. Without a retry cap and a dead-letter path, that single job can occupy workers indefinitely.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how ongoing scans are triggered by change over time, which is the producer side of the queue described above.
Safeguard runs dependency and SBOM scanning with continuous monitoring, so you do not have to build this plumbing to get scans started. If you want to see a result first, try the free scan at https://gold.safeguard.sh/scan.
Quick check
Do I need exactly-once delivery?
No. Design for at-least-once delivery with idempotent jobs and result writes keyed by a dedupe key. That is simpler and more reliable than chasing exactly-once.
Should the queue hold the source code?
No. Store a reference such as a commit or image digest, and let the worker fetch it with short-lived credentials.
Last reviewed: 19 October 2026.
