So, Is AI Coding Faster?

Sometimes—but the evidence does not support a single percentage that travels unchanged from one software setting to another. The Copilot and METR studies measure the same broad subject, the effect of AI coding assistance on developer productivity, under different task, human, repository, tool, and completion boundaries. The practical judgment is therefore conditional: before adopting a headline claim, inspect the work that produced it and run a proportionate comparison in the workflow where the decision will matter.

Both results are valid because the boundary changed

The Copilot experiment provides credible evidence of a large speedup on a tightly defined task. Professional programmers received the same JavaScript HTTP-server assignment in a common repository, and the treatment group had Copilot while the control group could use ordinary resources such as search and Stack Overflow. The clock ran from repository creation to the first commit that passed twelve visible checks. Among participants who completed the task, average time was 71.17 minutes with Copilot and 160.89 minutes without it, a reported 55.8% reduction.

The result is not a marketing impression. Random assignment, a common task, automated checks, and recorded timestamps make it a meaningful causal estimate for that task and population. The confidence interval, reported as 21% to 89%, shows that the exact size of the effect remains uncertain even though the direction and practical magnitude are compelling. The study also reported a higher treatment-group success rate, but the difference was not statistically conclusive. Most importantly, the experiment did not measure code quality, maintainability, performance, security, or the work of getting a change accepted in a living codebase.

METR measured a different boundary. Sixteen experienced open-source developers completed 246 real issues in mature repositories to which they regularly contributed. Each issue was assigned to an AI-allowed or AI-disallowed condition before the work was done. In the allowed condition, a developer could use the tools they preferred or no AI at all. The primary outcome was total implementation time, including work before and after pull-request review. Under that definition, AI-allowed issues took 19% more time on average.

The METR result is also not a simple opinion about AI. Its issue-level randomization, pre-treatment difficulty forecasts, screen recordings, and review-aware time measure make it a serious field experiment. The paper reports that the pattern persists under several alternative estimators and outcome checks. Yet the participants and repositories were specialized: experienced maintainers, large and mature codebases, and high standards for contributions. That setting is closer to an important class of real work than the Copilot exercise, but it is not a sample of all software engineering.

The studies answer different conditional questions. Copilot asks what happens when developers receive an AI pair programmer for a bounded exercise with a visible automated target. METR asks what happens when experienced maintainers are allowed to incorporate contemporary AI tools into real repository work whose acceptance depends on local knowledge and review. Their percentages should not be averaged, ranked, or treated as competing estimates of one universal effect. They are observations at different points in the space of software work.

The apparent contradiction disappears once “faster” is unpacked. The first experiment stops at a passing test suite. The second includes the time needed to make a real issue acceptable through review. The first gives the assistant a common starting point and a clear target. The second gives the human a deeply familiar repository whose important rules may not be fully visible to the tool. The first demonstrates that AI can remove substantial implementation time under favorable conditions. The second demonstrates that generated code can fail to compensate for prompting, waiting, review, cleanup, and context-management costs.

Neither finding cancels the other. A positive result does not imply universal speedup. A negative result does not imply universal slowdown. Together they reject the idea that AI coding productivity is a property of the model alone.

A conditional decision model for new teams

The best transfer from these studies is not a forecast; it is a set of questions.