Measure five things for each category: how many proposals were accepted, how many were reverted, how much rework reviewers did, how long review took, and why rejected ones were rejected. Set thresholds before you start, so the data decides and enthusiasm does not.
Why a trial period needs numbers
A manual period without measurement is just delay. The point is to turn the question "do we trust it?" into figures you can read. It also gives you something to show a reviewer or auditor: a dated record of how the decision was made.
Run it for a fixed window or a fixed number of proposals, whichever is longer, per category. Do not mix categories, since a good result for patch updates says nothing about code changes.
The five measures
| Measure | How to calculate | What it tells you |
|---|---|---|
| Acceptance rate | Merged proposals divided by all proposals | Whether reviewers find the output usable |
| Revert rate | Merged proposals later reverted divided by merged | Whether accepted fixes hold up |
| Rework | Proposals that needed human edits before merge | How close the output is to final |
| Time to review | Open to decision, median and slowest | Reviewer load and queue health |
| Rejection reasons | Tagged categories for each closed proposal | Where the tool or scope is weak |
For the first two, see metrics for fix PR quality. Use the median for time, not the mean, because one stuck pull request skews an average.
Tag rejection reasons from day one
Give reviewers a short fixed list so tags are consistent:
- Breaks tests or build
- Changes behavior
- Out of scope files
- Unnecessary dependency
- Duplicate or already fixed
- Low value, no reachable use
- Other, with a note
After a month, the distribution tells you what to fix: scope rules, test coverage, or prioritization. If most rejections say changes behavior, the category is not ready, whatever the acceptance rate.
Set thresholds in advance
An example, to adapt, not a standard:
- At least a set number of proposals reviewed, enough that one outlier cannot swing the result.
- Acceptance rate above a level your team picks.
- Zero reverts that reached production, or a defined small number with root causes understood.
- No rejection tagged as out of scope files or unnecessary dependency in the last stretch.
- Median review time under a level reviewers say is sustainable.
Write the thresholds in the ticket that starts the trial. Changing them afterwards should need the same approval as the original.
Beware of small numbers
Ten proposals with eight accepted looks like eighty percent but tells you little. Report counts alongside rates, and do not promote a category on a thin sample. A category with few proposals may need a longer window or may simply never be a candidate.
What not to count
- Time saved, unless you measured it before. Estimates invite flattery.
- Number of vulnerabilities fixed, since a fix that should not have merged counts the same.
- Reviewer sentiment alone. Useful as color, weak as evidence.
After the window
Hold a short review: show the table, the rejection distribution, and the thresholds. Decide one of three outcomes: stay manual, extend the trial, or raise one category one step. Record the decision and the review date. If you raise it, keep measuring, since quality can drift.
Safeguard offers autonomous remediation with fix PRs, off by default with every category set to Manual, which gives you this manual period as the starting point.
Quick check
How long should the manual period last?
Long enough to review a meaningful number of proposals per category. Pick a window and a minimum count, and use whichever is longer.
Is a high acceptance rate enough to automate?
No. Check reverts and rejection reasons too. A high acceptance rate with reverts means reviewers were approving too quickly.
Last reviewed: 19 October 2026.
