Overview

The GitHub Copilot experiment reported a 55.8% reduction in completion time on a standardized JavaScript HTTP-server task with visible tests, while the METR field experiment found AI-allowed issues took 19% more time for experienced developers in mature repositories after adjustment. Their different tools, populations, tasks, quality contexts, and endpoints make the results conditional evidence, not a universal productivity forecast.

Learning outcomes

The Copilot study reported a 55.8% time reduction on a standardized JavaScript HTTP-server task with twelve visible checks.
The METR study found AI-allowed issues took 19% more time after adjustment among experienced developers in mature repositories.
The studies support conditional productivity, not a universal forecast, because their tasks, tools, populations, contexts, and endpoints differed.

Keywords

GitHub CopilotMETRproductivityexperimentrepositoriescompletion

What the Productivity Studies Actually Show

Two studies often enter the agentic-coding debate as if they answered one question: does AI make software developers faster? They do not measure the same situation. This chapter reconstructs what the GitHub Copilot experiment and the METR field experiment actually randomized, measured, and left unknown. The practical judgment is to treat their different results as evidence about conditions and measurement, not as percentages that can be averaged into a universal productivity forecast.

The distinction matters because a productivity result is always a relationship between a tool, a person, a task, an environment, a quality bar, and an endpoint. Change any of those and the meaning of the number can change. A controlled task can isolate a treatment effect with unusual clarity. A real repository can expose context and review costs that a short benchmark cannot. Neither design is automatically the answer to every organizational decision.

A controlled task produced a large measured speedup

The GitHub Copilot study recruited 95 professional programmers through Upwork and randomly assigned 45 to a treatment group with access to Copilot and 50 to a control group without it. Thirty-five developers from the two groups completed the task and the exit survey. The assignment was deliberately standardized: participants had to implement an HTTP server in JavaScript from a provided skeleton, and a visible test suite contained twelve correctness checks. Both groups could otherwise use ordinary sources such as internet search and Stack Overflow.

The study measured two outcomes. Task success meant satisfying all twelve checks. Completion time ran from creation of the participant’s repository to the first commit that passed them all. Among participants who completed the task, the treatment group averaged 71.17 minutes and the control group 160.89 minutes. The reported 55.8% reduction in completion time had a p-value of 0.0017 and a 95% confidence interval for the improvement of 21–89%. The treatment group’s success rate was seven percentage points higher, but that difference was not statistically significant.

This is useful causal evidence for a narrow proposition: in this controlled programming task, access to Copilot was associated with a large reduction in time to a tested submission. The visible tests and fixed endpoint make the comparison legible. The experiment did not ask participants to discover an unfamiliar product requirement, negotiate an architectural change, preserve years of compatibility behavior, or take a pull request through a team’s review process. Its clarity is a strength, but it is also part of its boundary. A visible test suite also gives the agent and the human a particularly legible target: the required behaviors are concrete, the task is short enough to hold in working context, and completion has a clear stopping condition. That makes the result valuable for establishing that assistance can help under controlled conditions, while making it unsafe to assume that the same amount of help will survive a less specified task.

The study’s heterogeneous results add nuance without changing that boundary.

Published by DeepMox

FAQ

What did the GitHub Copilot experiment actually measure?

The GitHub Copilot experiment measured time from repository creation to the first commit passing twelve visible checks on a standardized JavaScript HTTP-server task. It randomly assigned 95 professional programmers, although 35 completed the task and exit survey, and it did not measure code quality, maintainability, performance, or security.

What did the METR field experiment find?

The METR field experiment found that AI-allowed issues took 19% more time after adjustment among 16 experienced developers completing 246 real issues in large, mature repositories. Its endpoint included work before and after pull-request review, and its tools represented the early-2025 frontier rather than every current or future system.

Why cannot the Copilot and METR results be averaged?

The Copilot and METR results cannot be averaged because they used different tools, developers, tasks, repositories, quality contexts, comparators, and completion endpoints. One isolated a standardized tested task, while the other measured the net effect of allowing AI in complex repository work that included post-review effort.

Previous9 minAgentic Coding: Speedup Is Not Value

Next9 minAgentic Coding Task Fit and Verification Tax

GitHub Copilot vs METR: Productivity Evidence

What the Productivity Studies Actually Show

A controlled task produced a large measured speedup

FAQ