Do AI Coding Tools Make Developers Faster? What the Controlled Studies Measured, 2023 to 2026

The best controlled studies of AI coding tools disagree by a wide margin: one found developers 55.8% faster, another found experienced developers 19% slower. Both are careful, published and reproducible in method. They differ because they measured different people doing different work, and in 2026 the hardest part has become measuring at all.

This post is for engineering leaders who are asked what AI tools are worth, and for anyone quoting a productivity figure in a plan or a budget. It lays out six controlled studies from 2023 to 2026 with their designs and sample sizes, explains why the numbers diverge, covers what is known about code quality, and gives a script for measuring your own team.

The short answer

Controlled studies don't support a single productivity figure for AI coding tools. A 2022 lab task found developers 55.8% faster; a Google trial with 96 engineers estimated about 21% less time, though not significant at the usual threshold; field experiments with 4,867 developers at three companies found 26% more completed tasks; and METR's 2025 trial with 16 experienced open source developers found tasks took 19% longer. METR's 2026 follow-up suggests developers are faster now but says its own data can no longer measure it reliably, because many developers won't work without AI.

The studies

Two details in the table change how the headline numbers read. The Google estimate is the paper's best estimate, but its own 95% interval includes no effect. And METR's 2026 results come with the authors' warning that selection effects make them "only very weak evidence" for the size of any change.

Why the results disagree

The task. A greenfield HTTP server is close to ideal for an assistant: a well-known problem with a clear spec. METR's developers worked on their own repositories, which they'd contributed to for five years on average, under those projects' quality standards. An assistant has less to add when the human already knows the code better than any model does.

The people. Cui et al. found that less experienced developers adopted the tool more and gained more. Peng et al. found the same pattern for less experienced programmers. METR studied the opposite end: experts in mature projects.

The outcome. Time to finish one task, pull requests per week and later maintainability are different things. More pull requests can mean more work done, or smaller units of work. Cui et al. also report 13.55% more commits and 38.38% more builds, and they found no negative effect on build success rate.

What people believe. METR's 2025 developers forecast that AI would cut their time by 24%, and after the study estimated it had cut it by 20%. The measured effect went the other way. Self-reported speedups are the weakest evidence in this field, and they're the most commonly quoted.

The tools changed. The studies span completion-style assistants in 2022 to agentic tools in 2025 and 2026. A result from one generation says little about the next, which the Google authors note themselves.

Why measuring got harder in 2026

METR's February 2026 update is worth reading in full, because it describes a measurement problem rather than a result. Its second study, started in August 2025, ran into three effects:

• Developers opting out. More developers declined to take part because they didn't want to do half their work without AI, even when paid to work on tasks of their own choosing.
• Tasks held back. When surveyed, 30% to 50% of developers said they weren't submitting some tasks because they didn't want to do them without AI. The tasks most likely to benefit were the ones missing.
• Concurrent agents. Time per task became unreliable for developers running several agents at once.

METR also cut pay from $150 to $50 an hour, which it says likely added to the selection. Its conclusion is that developers are probably faster in early 2026 than in early 2025, that its estimate is likely a lower bound, and that it is changing the study design.

A May 2026 METR note adds a framing problem. AI changes which tasks people choose to do, not just how fast they do them. Measuring time saved on the old mix of tasks understates the value, and measuring it on the new mix overstates it. Under the note's assumptions, the true gain in value sits between the two.

What about quality and security?

Much less is measured here than speed. Borg et al. is the most direct test of a downstream effect: developers who later had to evolve AI-assisted code were no slower and produced no worse code, and the authors saw "no consistent warning signs" of degraded maintainability, while naming code bloat and cognitive debt as risks to study. Cui et al. found no drop in build success.

None of these six studies measured security outcomes such as vulnerabilities introduced, secrets committed or dependency choices. That gap matters more as agents write larger changes with less line-by-line review. If your case for AI tools rests on speed, plan to measure what the extra code does to your vulnerability backlog as well. Our post on coding agent security posture covers the controls side.

Measure your own team

Published studies tell you what happened to someone else's developers on someone else's tasks. A small internal trial tells you about yours. METR's design is simple to copy:

1. The developer writes down a task and how long they expect it to take.
2. Only then is the task randomly assigned to "AI allowed" or "AI not allowed".
3. They do it and record the actual time.

Comparing actual-to-forecast ratios between the two groups cancels out much of the difference between big and small tasks. This script does that comparison on a log scale and puts a bootstrap confidence interval around it, resampling by developer so one prolific person can't dominate. Standard library only:

#!/usr/bin/env python3
"""Estimate how much AI tools change task time from your own randomized log.

Input CSV columns: task_id, developer, condition (ai or no_ai), forecast_minutes,
actual_minutes. Randomize the condition per task AFTER the developer writes the
forecast, then record actual time. The estimate is the ratio of actual to
forecast time with AI versus without, on a log scale, with a bootstrap 95% CI
that resamples developers. Under 1.0 means AI tasks took less time.

Usage: python3 uplift.py tasks.csv
       python3 uplift.py --assign   (prints ai or no_ai for the next task)
"""
import csv
import math
import random
import statistics
import sys


def load(path):
    rows = []
    with open(path, newline="") as f:
        for r in csv.DictReader(f):
            if r["condition"] not in ("ai", "no_ai"):
                raise SystemExit(f"bad condition {r['condition']!r} in task {r['task_id']}")
            fc, act = float(r["forecast_minutes"]), float(r["actual_minutes"])
            if fc <= 0 or act <= 0:
                continue
            rows.append((r["developer"], r["condition"], math.log(act / fc)))
    return rows


def estimate(rows):
    ai = [x for _, c, x in rows if c == "ai"]
    no = [x for _, c, x in rows if c == "no_ai"]
    if len(ai) < 2 or len(no) < 2:
        return None
    return math.exp(statistics.fmean(ai) - statistics.fmean(no))


def bootstrap(rows, n=2000, seed=7):
    by_dev = {}
    for row in rows:
        by_dev.setdefault(row[0], []).append(row)
    devs = sorted(by_dev)
    rng = random.Random(seed)
    out = []
    for _ in range(n):
        sample = [r for d in rng.choices(devs, k=len(devs)) for r in by_dev[d]]
        e = estimate(sample)
        if e is not None:
            out.append(e)
    out.sort()
    return out[int(0.025 * len(out))], out[int(0.975 * len(out)) - 1]


def main():
    if sys.argv[1:2] == ["--assign"]:
        print(random.SystemRandom().choice(["ai", "no_ai"]))
        return
    rows = load(sys.argv[1])
    point = estimate(rows)
    if point is None:
        raise SystemExit("need at least two tasks in each condition")
    lo, hi = bootstrap(rows)
    devs = len({d for d, _, _ in rows})
    n_ai = sum(1 for _, c, _ in rows if c == "ai")
    print(f"{len(rows)} tasks ({n_ai} ai, {len(rows) - n_ai} no_ai) from {devs} developers")
    print(f"time with AI vs without: {point:.2f}x (95% CI {lo:.2f}x to {hi:.2f}x)")
    if hi < 1:
        print("AI tasks took less time, and the interval excludes no effect")
    elif lo > 1:
        print("AI tasks took more time, and the interval excludes no effect")
    else:
        print("the interval includes no effect; collect more tasks before concluding")


if __name__ == "__main__":
    main()

We ran it on three synthetic logs: a small team of 6 developers and a larger one of 20, both generated with a true effect of 0.8x (AI tasks take 80% of the time), and a team of 20 with no true effect:

== fx/team_small.csv
48 tasks (18 ai, 30 no_ai) from 6 developers
time with AI vs without: 0.77x (95% CI 0.54x to 1.11x)
the interval includes no effect; collect more tasks before concluding
== fx/team_large.csv
500 tasks (269 ai, 231 no_ai) from 20 developers
time with AI vs without: 0.75x (95% CI 0.68x to 0.82x)
AI tasks took less time, and the interval excludes no effect
== fx/team_null.csv
500 tasks (244 ai, 256 no_ai) from 20 developers
time with AI vs without: 1.00x (95% CI 0.92x to 1.09x)
the interval includes no effect; collect more tasks before concluding

The small team's estimate is close to the truth, but its interval is too wide to rule out no effect. That's the practical lesson: a few dozen tasks from a handful of people won't settle the question, and a few hundred from twenty people can. Expect the same selection problem METR hit. If people quietly stop submitting tasks they'd rather do with AI, your estimate drifts low.

Speed is only half the picture for a security team. Here's Safeguard's interactive walkthrough of finding and fixing a vulnerability, the kind of downstream work that extra AI-written code creates:

What to do with all this

1. Don't quote one number. If a plan says "AI makes developers X% faster", ask which study, which tasks and which developers.
2. Discount self-reports. In METR's 2025 study, the one here that measured both, beliefs and measurements pointed in opposite directions.
3. Expect different effects for different people. Less experienced developers gained more in two studies. Experts on familiar code gained least, or lost time, in 2025.
4. Run your own trial if the decision is large. Randomize after the forecast, collect enough tasks, and report the interval, not just the point estimate.
5. Measure quality and security alongside speed, since the published evidence on them is thin.

The fair view

Every study here has limits its authors state plainly: single tasks, small samples, one company's tools, a single year. The field moves faster than studies can be designed and published. A skeptic could fairly say the 2025 slowdown is already out of date, and METR's own update suggests it probably is.

A fair optimist should concede something too. The largest field evidence is a 26% gain in completed tasks with a wide error band, and the newest data is described by its authors as unreliable. Any figure much larger than these deserves a question about how it was measured. The honest position for late 2026 is that AI tools probably help most developers on many tasks, by an amount nobody has measured well, and that measuring it is now harder than it was a year ago.

Check yourself