When Real Software Work Gets 19% Slower
The METR study asks a harder and more practical version of the AI productivity question: what happens when experienced developers use contemporary AI tools on real issues in codebases they already know? Its answer was surprising. Across the study’s measured workflow, issues assigned to an AI-allowed condition took 19% more time than issues assigned to an AI-disallowed condition. Understanding that result requires more than repeating the percentage. It requires seeing how METR preserved a causal comparison while bringing repository history, pull-request review, and human judgment back into the task.
A field experiment built around real issues
The experiment involved 16 experienced open-source developers and 246 issues. These were not puzzles written for a benchmark. The developers contributed to the repositories in question and had an average of about five years of experience with their target repositories, including roughly 1,500 commits. The repositories were large and mature: on average they had about 23,000 stars, 1.1 million lines of code, 20,000 commits, and a high bar for accepting contributions.
An “issue” in this setting was a real piece of repository work: a bug report, feature request, or other item a contributor wanted to resolve. Developers supplied lists of issues and estimated how long each would take with and without AI. They were asked to keep issues relatively short—ideally no more than two hours, breaking up larger work when possible. That constraint made the experiment feasible, but it also matters for interpretation. The study measured a selected slice of repository maintenance, not every activity in open-source development. Design discussions, broad architectural work, long investigations, and pull-request review performed for other people were not represented in the same way.
After issue lists were assembled, METR randomly assigned issues to one of two conditions. In the AI-allowed condition, the developer could use any AI tools they chose, including no AI if they decided it would not help. In the AI-disallowed condition, generative AI tools were prohibited, although ordinary search remained available. Developers could work through their assigned issues in their preferred order, submit pull requests through the repository’s normal system, respond to review, and merge the completed change when it met the project’s standards.
This design combines two properties that are often difficult to hold together. The work is realistic enough to contain local conventions and accumulated knowledge, but assignment still occurs after the issues are defined and before the treatment is known. That timing limits a major source of confounding. If an AI tool could cause a developer to expand an issue after seeing it, or if the researcher selected only tasks that looked easy for the tool, a longer or shorter result would be harder to interpret. Defining the issue first does not remove every possible bias, but it gives the comparison a stable unit of work.
“AI allowed” was a permission, not a dose
The treatment label is easy to misread. METR did not require each participant to use the same model, the same interface, or the same prompting strategy. Developers could use Cursor Pro, a web-based model, autocomplete, or another available tool. They could also decline to use AI on an issue even when it was allowed. The measured effect therefore includes a human decision: whether the available assistance seemed worth incorporating into that particular task.
That makes the treatment closer to a workplace policy or workflow option than to a controlled quantity of generated code. It answers a question a team might actually face: if developers are permitted to use current AI tools while completing their normal work, does the work get completed faster? It does not answer whether a perfectly selected model output would be faster than no output, nor does it isolate the effect of forcing a specific agent to work for a fixed amount of time.
The study provided Cursor Pro subscriptions to most participants and gave them basic training. Cursor was relevant because it combined an ordinary code editor with chat and agent features that could inspect a repository, read documentation, run commands, edit files, and iterate. Participants were not starting from zero with language models: 93% had prior experience with tools such as ChatGPT, although only 44% had prior experience with Cursor. The experiment thus tested relatively capable users with a contemporary toolset, not people encountering generative AI for the first time.
The repositories also set a meaningful acceptance boundary. Developers generally worked through pull requests, and another contributor often reviewed their changes. The final result was expected to meet a project’s existing standards, not merely compile or pass a small visible test suite. A change that looked plausible but violated style, omitted a test, or disturbed an implicit compatibility rule still had work left to do.
How METR measured time
METR’s primary outcome was total implementation time: the active time a developer reported spending on an issue before and after pull-request review. The study included the work required to make changes in response to review, rather than stopping at the moment a first draft was ready for someone else to inspect. For a small number of issues that had not been reviewed or merged by the end of the study, the researchers imputed post-review time using the mean for the relevant treatment condition.
The paper expresses the treatment effect as the expected completion time with AI allowed divided by the expected completion time with AI disallowed, minus one. A negative value would indicate a reduction in time; a positive value indicates a slowdown. The main estimate came from a log-linear regression. Alongside the AI condition, the regression used each developer’s forecast of how long the issue would take without AI as a proxy for difficulty. Those forecasts were collected before the issue was assigned, so the treatment could not have changed them.
This adjustment is useful because random assignment does not guarantee that every small batch of issues has exactly the same difficulty. Imagine that the AI-allowed bucket happens to contain slightly more issues involving unfamiliar dependencies. A comparison of raw averages could then attribute some of that difference to AI. The pre-assignment forecast helps account for it without using information created after treatment. METR reports that forecasted difficulty was broadly balanced between the conditions, supporting the randomization, and that the forecasts predicted actual completion time.
The study completed 136 issues in the AI-allowed condition and 110 in the AI-disallowed condition. The primary analysis found that AI-allowed issues took 19% more time on average. This is a slowdown in the time required to complete the issue under the study’s total-time definition. It is not a claim that the developers produced 19% less code, that the AI models were 19% worse than humans, or that AI should never be used. The numerator and denominator are both human work performed under different access conditions.
What the result does—and does not—establish
The result is stronger than a survey because the condition was assigned rather than chosen solely by developers who already liked AI. It is also more relevant to ordinary software maintenance than a task designed to be solved in a few minutes with visible tests. The issue was fixed before the assignment, and the outcome was tied to the time needed to bring the work through the repository’s normal process.
But realism is not the same as universal representativeness. The participants were selected because they were highly experienced contributors to large open-source repositories. Their familiarity was unusually deep: they knew the local architecture, conventions, historical decisions, and likely review expectations. The repositories were also selected for size, maturity, popularity, and quality standards. A small greenfield project, a codebase unfamiliar to the developer, or a task involving routine boilerplate could present the assistant with a different balance of advantages.
The task mix was bounded as well. Developers were encouraged to break work into issues of two hours or less, and they noted that the study sampled toward the smaller end of their normal contributions. That choice is not a fatal flaw; it makes the unit of comparison more manageable and reduces the chance that a long task will cross too many unrelated states. It does mean that the study cannot tell us how AI affects extended design work or a multi-day change whose value emerges only after several pieces are integrated.
There are also measurement qualifications. Time was partly self-reported, and some post-review times were imputed. The study collected screen recordings, but only a subset was suitable for detailed labeling. Developers could choose their completion order, and some issues were dropped. These features create possible sources of bias even though the core assignment was randomized. The important question is whether those concerns are large enough to explain the whole slowdown.
Why the authors regard the slowdown as robust
METR tested the result against several alternative explanations. Twenty-five early issues had been randomized with a different block scheme before the researchers switched to a simulated fair coin flip. Excluding those issues still produced about a 20% slowdown. Using screen-recording duration instead of self-reported time on a comparable subset produced about a 25% slowdown, close to the estimate from self-reported time on that subset. Alternative estimators also gave similar results.
The authors examined whether an unfamiliar development environment was the main culprit. Some developers used comparable editors in both conditions, while others changed environments to avoid accidentally using AI when it was disallowed. The comparable-editor subset still showed a slowdown. They also found that basic noncompliance did not explain the pattern: AI was used on most labeled issues where it was allowed, and only a small number of AI-disallowed recordings showed apparent use of AI tools. Results remained similar under checks involving dropout.
These checks do not prove that every issue would slow down, or that no unmeasured artifact matters. They do make a single explanation—such as an awkward editor, widespread cheating, or a fragile statistical formula—less sufficient. The paper’s own language is careful. It evaluates 21 possible contributing factors, finding evidence for five, mixed or unclear evidence for ten, and evidence against six. The factor analysis was not powered to establish many independent effects through conventional multiple-comparison tests. It is a map of plausible mechanisms and boundaries, not a complete causal decomposition of the 19%.
The models used were also not obviously obsolete relative to the study period. Most issues were completed in February and March 2025, and the labeled recordings included Claude 3.7 Sonnet, Claude 3.7 Sonnet in thinking mode, Claude 3.5 Sonnet, GPT-4o, Gemini 2.5 Pro, and o1. The distribution was not a controlled model benchmark, and autocomplete was used separately, but the result cannot be dismissed simply as a test of an obviously outdated assistant.
A specialized result, not an anti-AI verdict
METR explicitly limits the claim. The study does not show that current AI tools fail to improve many developers, that they never speed up any individual task, or that they will not help in other domains. It does not establish that future models will fail in the same setting. Better reliability, lower latency, repository-specific training, more effective prompting, or more capable agent scaffolding could change the outcome.
Indeed, the paper reports that some developers experienced speedup on some tasks. It also leaves open the possibility of a high skill ceiling for using tools such as Cursor, while noting that the evidence for substantial gains after more than 50 hours of experience is sparse and underpowered. The negative aggregate result is therefore not a statement that AI assistance has no value. It is a statement that value was not reliably realized as shorter total completion time for this population and workflow.
The most important change from the Copilot experiment is not simply that the METR tasks were “more realistic.” Realism reintroduced responsibilities that the earlier test could hold constant or omit: the human had to understand a living codebase, the assistant had to work through incomplete context, generated changes had to meet a high local bar, and the clock continued through review. Once those responsibilities are inside the boundary, the question becomes: where did the time go?
The next analysis follows that time rather than treating the 19% as a mystery. METR’s recordings show developers spending less of their work on direct coding and more on prompting, waiting, reviewing, and cleaning. That workflow is the bridge between a model that can generate useful code and a system that can reliably shorten the path to an accepted change.