A coding agent pilot should end with a written go or no go decision based on criteria agreed before it starts: controls working as designed, no unresolved security incidents, review quality holding up, and developers willing to keep using it. Set the thresholds first, so the result is a measurement, not a mood.
Why set criteria up front
Without criteria, the loudest voice wins. Enthusiasm or alarm after one memorable event drives the decision. Written criteria give you a calm way to say yes, no, or not yet.
NIST's AI Risk Management Framework is voluntary and organizes risk work around governing, mapping, measuring and managing risk. A go or no go review is a small instance of the measure and manage steps: you decide what to measure, then act on what you see.
Four groups of criteria
1. Control coverage. Did the controls you promised exist and work?
- Permission limits enforced in the environment, not only by prompts.
- Branch protection applied to every repository in scope.
- Action records available for every session.
2. Review quality. Did human review stay meaningful?
- Reviewers read the diffs, not just the summaries. Sample ten merged agent pull requests and re-read them.
- No pattern of approvals within seconds of opening.
3. Incidents and near misses.
- Count secrets exposed, unwanted dependencies added, and out of scope file changes.
- Every incident has an owner and a closed action.
4. Developer experience.
- The skeptic's concerns have an answer.
- Developers can describe when they should not use the agent.
A simple decision table
| Result | Decision |
|---|---|
| All four groups meet their thresholds | Go, move to the next ring |
| Controls fine, review or experience weak | Not yet, fix and extend two weeks |
| Controls missing or an unresolved serious incident | No go until fixed |
Write the thresholds in numbers where you can. For example, you might require zero open incidents and a review sample with no unexplained approvals. Choose your own numbers. This post does not give a universal one because teams differ.
Run the review meeting
- Circulate the data two days before.
- Start with incidents and near misses.
- Hear the skeptic and the reviewer first.
- Decide using the table, and write down the reasons.
- Record what changes before the next ring starts.
Avoid these traps
Measuring only speed. A faster pull request that a reviewer cannot trust is not a gain. Pair any speed number with a quality number such as changes requested or defects found after merge.
Moving the goalposts. If you must change a threshold, record why and who approved it.
Safeguard offers compliance and evidence automation, which can help keep the pilot records in a shape an auditor or customer can read later.
Quick check
Can a pilot end with a partial go?
Yes. Approve some repositories or teams and hold others, as long as the decision and the conditions are written down.
Who makes the final call?
The executive who owns the risk, advised by security and engineering. Name that person before the pilot starts.
Sources
Last reviewed: 19 October 2026.
