Measure AI coding impact with a baseline, a comparison group and several paired metrics, and do not rely on how fast developers feel or report themselves to be. Feelings and measurements can point in opposite directions, and one controlled study showed exactly that.
What the evidence warns about
METR ran a randomized controlled trial in early 2025. Sixteen experienced developers worked on 246 real issues in large open source repositories they knew well. Each issue was randomly assigned to allow or disallow AI use. With AI allowed, developers took 19 percent longer. Before the work they predicted AI would speed them up by 24 percent, and afterward they estimated it had sped them up by 20 percent.
Treat that as a caution, not a verdict. The sample was small, the setting was experienced people on familiar code, and METR itself marks the post as out of date, pointing to a later update on newer tools. The lesson that holds is about method: perception is a weak instrument.
Step 1: baseline before you start
Record your delivery metrics for the previous quarter. DORA lists change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. They cover both throughput and instability, which matters because faster output with more breakage is not a gain.
Step 2: pair every speed metric with a quality metric
Pick at least one from each side:
- Speed: change lead time, time to first review.
- Quality: change fail rate, bugs found after merge, reviewer rework rounds.
If speed rises and quality falls, you have learned something real.
Step 3: use a comparison
Where you can, compare teams or periods with and without a tool, accounting for obvious differences in work type. If you can randomize at the task level, as METR did, you get far cleaner evidence, though few teams can afford it.
Step 4: ask people, but as a supplement
Surveys capture friction and trust, which numbers miss. Use them to explain the metrics, not to replace them.
Step 5: avoid the usual traps
DORA's guidance lists several. Do not set the metrics as goals, since targets invite gaming. Do not lean on a single metric. Do not use the numbers to rank teams. Compare like with like, since the metrics are meant per application or service.
DORA's 2025 report page frames AI as an amplifier of an organization's existing strengths and weaknesses. If your review and testing practices are weak, expect more output to expose them.
Step 6: report with humility
Share ranges, sample sizes and caveats. State what you could not measure. A modest, honest report builds more trust than a big claimed percentage.
Quick check
Is developer reported speedup enough evidence?
No. In the METR trial developers believed AI helped while measured completion times were longer.
Which metrics should be paired?
A speed metric such as change lead time with a stability metric such as change fail rate.
Sources
- METR, early 2025 AI and experienced open source developers study
- DORA, Software delivery performance metrics
- DORA, 2025 DORA report
Last reviewed: 19 October 2026.
