Pilot a review bot on one or two repositories for 30 days. Record a baseline first, label every bot comment as useful, noise or wrong, and decide your exit criteria before you start. Without those three steps, the pilot will measure enthusiasm rather than value.
Week 0: set the baseline
Before turning anything on, pull four numbers from the last 60 days for the pilot repos: median time from pull request open to first review, median time to merge, number of review rounds, and the count of bugs fixed within two weeks of merge. Use your own tracker. Do not invent targets from a vendor page.
Pick repositories with steady traffic and a team willing to give feedback. A repo with three pull requests a month will not teach you anything.
Week 1: narrow the scope
Start with the default settings and one change at a time. Note which file types the tool excludes. GitHub's documentation for its review feature, for example, lists dependency manifests, logs and SVG files as not reviewed, so decide who covers those.
Tell the team plainly that the bot is a first pass. Its approval, if it can give one, should not count toward required reviews. GitHub notes that Copilot reviews do not count toward required approvals unless that setting is enabled.
Weeks 2 and 3: label every comment
Ask reviewers to react to each bot comment with one of three labels:
- Useful: it caught something worth changing.
- Noise: correct but not worth the time.
- Wrong: factually incorrect or misleading.
A shared spreadsheet is enough. At the end you can compute the share of each label, and you will have real examples to discuss.
Also record misses. When a human reviewer finds a significant bug the bot did not mention, log it. Misses tell you more about the tool than hits do.
Week 4: compare and decide
Re-pull the four baseline numbers. Be careful with attribution. Review time can fall because the team is reviewing less carefully, which is not a win. Check the bug count and review rounds alongside speed.
Write exit criteria in advance. A reasonable example is that the useful share must exceed the noise and wrong shares combined, and no significant bug class may go unreviewed because people trusted the bot. Your thresholds are yours to set.
Watch the human side
Look for rubber stamping. If approvals get faster and comments get shorter after the bot arrives, the team may be leaning on it. Google's review guidance tells reviewers to read every line of human written code assigned to them, and that expectation does not change because a bot has also looked.
Record what you could not measure
Note cost, setup effort and any data handling questions for your security team. Then decide: adopt, adopt on a smaller scope, or stop. Each is a valid result.
Quick check
How many repositories should a pilot include?
One or two with steady pull request volume. More repos dilute feedback and make labelling harder to keep up.
What is the most important thing to decide in advance?
The exit criteria. Agreeing on them before you see results keeps the verdict honest.
Sources
Last reviewed: 19 October 2026.
