Write the red lines before you start. Good candidates: it will not answer data handling questions in writing, its verdict can be steered by text in a pull request, it skips code without saying so, it has no off switch or export, or it needs permissions far beyond reading code and commenting.
Why decide in advance
Scorecards reward a tool that is strong in many areas. They hide a single fatal flaw behind a good average. Red lines work differently: each one is a condition that ends the evaluation whatever the rest looks like. Setting them first keeps the decision honest when a demo is impressive and a deadline is close.
Eight red lines to consider
| # | Red line | How to check |
|---|---|---|
| 1 | Will not state in writing what it retains, where, and whether it trains on your code | Send the data handling questions |
| 2 | Verdict changes when the code is unchanged but a comment says it is safe | Run the planted comment test |
| 3 | Skips or truncates input without telling the user | Submit an oversized diff and read the output |
| 4 | Needs write access to code, secrets or settings to function as a reviewer | Read the permission list on install |
| 5 | Cannot be disabled per repository or entirely, quickly | Try it during the trial |
| 6 | No way to export findings and history, or to delete your data | Ask for a walkthrough, see exit and deletion |
| 7 | Cannot explain a finding with a location and reason a human can verify | Review 20 findings |
| 8 | Results too unstable to describe as a control | Run the repeat test |
You may add your own: a data residency limit, a language you cannot go without, a requirement to run inside your network.
What is not a red line
A few things look serious and usually are not:
- A few wrong flags. Every tool has them. Judge rate and tunability, not existence.
- Missing one framework you can cover with another control.
- A rough user interface, if the output is sound.
- A young company. That is a supplier risk to weigh, not an automatic no.
Keep the red lines to things you could not live with, so the list stays short enough to apply.
Make the decision usable
Record each red line as pass, fail or unknown. Unknown counts as fail until resolved, since a gap in answers is information. Share the list with the vendor at the start. Good vendors would rather know the criteria.
Also decide who can waive a red line. If a team really wants a tool that fails one, the exception should go to a named person, with a written reason and an end date, as in an exceptions process for an AI coding assistant policy.
After the trial
Keep the table with the vendor record. When the contract renews, rerun the red line checks, because tools change. A tool that passed last year may have gained a new integration or a new sub-processor.
Quick check
How many red lines is too many?
More than ten tends to be unusable. Keep only conditions that would genuinely end the evaluation.
What if the vendor passes all red lines but results are mediocre?
Red lines only exclude. Use your test set and metrics to decide whether it is worth adopting.
Safeguard publishes a free scan at gold.safeguard.sh/scan, and its policies, gates and guardrails are designed so enforcement stays under your control.
Last reviewed: 19 October 2026.
