Often not. Language models can produce different output for the same input, so an AI reviewer may flag a problem on one run and miss it on the next. Test for it: run every case several times, record how often findings change, and decide what level of variation you can live with.
Why the answers differ
Many models sample their output with some randomness. Even when a vendor turns that down, other things change between runs: a model version updated behind the scenes, different context pulled from the repo, truncated input on long diffs, or a retry after a timeout that uses a different path.
For a human, "sometimes it notices" is a quirk. For a control that you describe to auditors or customers, it is a problem, because you cannot say the check reliably ran.
The test, step by step
- Take 20 cases from your labelled test set, mixing vulnerable and safe.
- Run each case five times, same settings, ideally spread over a few days.
- Record per run: flagged yes or no, the line flagged, severity, and the wording of the explanation.
- Compute per case how many of five runs agree.
- Sort the cases by disagreement and read the worst ones.
You are looking for three patterns:
| Pattern | Meaning | Action |
|---|---|---|
| Same result five of five | Stable | Good |
| Flagged in some runs, missed in others | Unreliable detection | Treat that class as unverified |
| Always flagged, but severity or location changes | Stable detection, noisy detail | Fine for advisory use, bad for gating |
Decide what you will accept
Pick thresholds before you look at the data. For example: a gating check must be stable on at least 95 percent of cases, an advisory check on at least 80. Those numbers are examples to adapt, not standards. What matters is that you set them in advance.
If a class of issue is flagged only three runs in five, you can still use it as a hint. You cannot rely on it as the only safeguard. Pair it with a deterministic check, such as a rule-based scan or a dependency check that returns the same result every run.
Ways vendors reduce variance
Ask what the vendor does about it. Possible measures include fixed model versions, lower sampling randomness, running the check several times and keeping findings that repeat, and combining a model with rule-based analysis. None is magic, and each has a cost in time or spend. Ask for the behaviour, not a promise.
What to keep as evidence
Store the run table with dates, model or tool version if shown, and settings. When someone later asks whether the tool was reliable, you have data. When the vendor ships an update, rerun the table. See also regression testing after a model update.
Pair non-deterministic with deterministic
A practical design: let a deterministic scanner own the hard gates, such as known vulnerable dependencies, exposed secrets and policy violations. Let the AI reviewer add comments on logic and design issues, where a human reads and decides. That way a skipped finding in one run is not a skipped control.
Quick check
Is lowering randomness enough to make it repeatable?
It helps, but model updates, context selection and truncation can still change results. Test, do not assume.
How many repeats are enough?
Three to five runs per case shows obvious instability. More runs help only if you need a precise rate.
Safeguard pairs dependency and SBOM scanning, which returns the same result for the same inputs, with policies and gates that enforce on those results. See how a scan works at gold.safeguard.sh/scan.
Last reviewed: 19 October 2026.
