Is AI Coding Faster? The Answer Depends on the Work
This series examines a specific question: what do the GitHub Copilot and METR studies tell us about the effect of AI coding assistance on developer productivity? The question matters because the studies appear to disagree. One reports that developers completed a programming task 55.8% faster with Copilot. The other reports that experienced open-source developers took 19% longer when they were allowed to use contemporary AI tools. The useful conclusion is not that one study must be wrong. It is that a productivity result belongs to the work, people, tools, and definition of completion that produced it.
That distinction is easy to lose. “AI coding” can describe a suggestion that fills in a line, a chat session that explains an unfamiliar API, or an agent that searches a repository, edits several files, runs commands, and revises its own attempt. The modern category is therefore broader than code generation alone. Anthropic’s organizational guide describes agentic coding as tools that understand context, plan an approach, and execute coding tasks with limited human oversight. More capability can create more leverage, but it can also create more surface area for review, correction, and governance.
The subject here is not whether AI is impressive. It plainly can produce useful code. The subject is the relationship between AI assistance and the time required to produce software that a team can accept, maintain, and safely integrate. That relationship supports a practical judgment: before importing a headline percentage into a new team, ask what kind of work generated it, what the human already knew, what context the tool could access, and what counted as finished.
Two headline results, two kinds of work
The Copilot study used a deliberately standardized task. Researchers recruited 95 professional programmers through Upwork and randomly assigned 45 to a treatment group and 50 to a control group. Participants had to implement an HTTP server in JavaScript using a repository template. The treatment group received GitHub Copilot and a short introduction to it; the control group did not, although both groups could use ordinary resources such as internet search and Stack Overflow.
The researchers made completion concrete. Each repository contained twelve visible automated checks. The elapsed time ran from the creation of a participant’s repository to the first commit that passed all twelve tests. Among participants who completed the task, the Copilot group averaged 71.17 minutes, compared with 160.89 minutes for the control group. The paper reports a 55.8% reduction in completion time, with a p-value of 0.0017 and a 95% confidence interval for the improvement ranging from 21% to 89%.
That is a substantial result for the experiment it describes. It is evidence that, under those conditions, access to Copilot helped developers finish that common, testable programming exercise faster. It is not evidence that every software project becomes 55.8% faster. The assignment was a small standardized exercise rather than a long-lived product with requirements discovery, design negotiation, integration, review, and maintenance. The study also did not examine code quality, performance, or security in the way a production team would need to.
The METR study posed a different question in a different setting. It involved 16 experienced open-source developers completing 246 real issues in repositories to which they regularly contributed. The developers had, on average, about five years of experience with their target repositories and roughly 1,500 commits there. The repositories were large and mature, with high standards for contributions. Each issue was assigned to an AI-allowed or AI-disallowed condition. When AI was allowed, a developer could use the tools they preferred—or decide not to use AI if it did not seem helpful.
METR measured total implementation time, including work before and after pull-request review. Its primary estimate adjusted for each issue’s pre-assignment forecast of how long it would take without AI. Under that definition, AI-allowed issues took 19% more time on average. The result does not show that AI slows most developers or most software work. It does show that, in this particular real-world workflow, access to the early-2025 generation of AI tools did not translate into faster completion for experienced maintainers.
The contrast is therefore not “controlled study versus uncontrolled anecdote.” Both studies used random assignment. The difference is what was randomized, who did the work, how much context the work contained, what tools could do, and where the stopwatch stopped.
What “faster” is allowed to mean
The word “faster” sounds simple until the boundaries of the task are made explicit. In the Copilot experiment, it meant reaching the first commit that passed a fixed set of tests. In the METR experiment, it meant the time developers reported spending on an issue through the pull-request process, with some post-review time included and a small number of unreviewed cases handled through imputation. These are not interchangeable clocks.
A coding assistant can reduce the time spent typing while leaving the total delivery time unchanged. A developer may accept a generated function, then discover that it uses the wrong repository convention, misses an edge case, or changes an unrelated part of a file. They may spend the saved typing time reading the output, running tests, cleaning up style, restoring a previous version, or explaining the change during review. Those activities are not evidence that the tool failed in every respect. They are evidence that useful software work contains more than generation.
This is why “AI allowed” also needs careful interpretation. METR did not force developers to use one model for every issue. The condition changed whether generative AI tools were available. The resulting estimate includes the choices people made when deciding whether assistance was worth using. It is closer to the effect of permitting an AI workflow in a real setting than to a laboratory measurement of a fixed dose of model output.
The same care applies to the phrase “AI coding.” The Copilot study tested an AI pair programmer on a bounded exercise. METR participants could use interfaces such as Cursor’s chat and agent features, which could inspect files, run commands, edit code, and iterate, as well as other models and autocomplete. An organizational guide may discuss still broader agentic workflows, including incident response or legacy modernization. These interventions overlap, but they are not one standardized treatment. The more autonomy a tool has, the more important it becomes to define who approves changes, what tests establish success, and which risks remain with the human team.
The variables hidden inside a headline percentage
The two studies make several variables impossible to ignore.
Task structure. A small task with a clear target and visible tests gives an assistant a narrow path to a successful answer. Real maintenance work may involve ambiguous requirements, historical constraints, undocumented dependencies, or changes that touch several interfaces. A task can be technically small and still be difficult for a system that lacks the surrounding context.
Human baseline. Copilot’s paper reports that less experienced developers and some other subgroups benefited more in its experiment. METR found evidence that its experienced maintainers were slowed more on issues they already understood well. These observations are not universal laws, but they point to the same principle: assistance is valuable partly where it supplies capability or knowledge the human does not already have.
Accessible context. A maintainer may know why a seemingly redundant line protects backward compatibility, which helper a project uses for a particular test, or where a change belongs even when the documentation is incomplete. If that knowledge is not visible to the tool, the assistant can produce plausible but misplaced work. The human then becomes an interpreter and filter for context the system cannot reliably infer.
Reliability and latency. A suggestion that is correct often enough can save time. A suggestion that must be inspected line by line, repeatedly prompted, or substantially rewritten may impose a tax. Waiting for a generation can also interrupt the developer’s flow. The net effect depends on the amount of useful work that survives the review process, not on the amount of text the model emits.
Definition of completion. Passing an automated test suite is valuable evidence, but it is not the same as meeting a production team’s quality and security bar. Conversely, lines of code, commits, and pull requests can increase without representing more useful work. A fair comparison must state what is fixed in advance, what quality conditions apply, and whether review and integration are included.
These variables are the beginning of an explanation, not a formula that can predict an exact speedup for a new team. METR’s analysis of possible causes is deliberately cautious: the authors identify evidence for five contributors to the slowdown, mixed or unclear evidence for ten, and evidence against six. Their screen recordings and robustness checks make the workflow costs visible, but they do not isolate each mechanism as if it had been tested independently.
Why this matters beyond the two experiments
For an individual developer, the practical question is not “Which study should I believe?” It is “Which parts of my next task resemble the conditions of either study?” A bounded migration step with a clear test oracle may be a good candidate for assistance. A change that depends on tacit knowledge of a mature subsystem may still benefit from an AI conversation, but not necessarily from delegating implementation wholesale. In both cases, the relevant measure is the time to a correct, reviewed result—not the time to a first draft.
For a manager, the stakes are larger because a tool rollout changes the workflow around many people. A dashboard can show accepted lines, suggestion acceptance, daily activity, or spending. Those signals may help reveal adoption, but they do not establish that a team is delivering more valuable software. The manager also has to account for review capacity, security checks, onboarding, incident risk, and the possibility that people continue using a tool because it feels easier even when it does not reduce elapsed time.
That is why the Anthropic guide’s practical recommendations are best understood as interventions to test, not as proof that a productivity gain will follow. Shared repository guidance may make hidden context more accessible. Test-driven, incremental work may constrain an agent’s scope. Security policies and human review may prevent a fast change from becoming an expensive defect. Each practice has a plausible mechanism, but its value still depends on the target workflow and should be measured locally.
The comparison also matters for public claims about the future of software work. The Copilot result suggests that AI assistance can produce large gains in some tasks and may help narrow experience gaps. The METR result warns that impressive model capability and widespread enthusiasm do not guarantee faster real-world delivery. Neither study settles how future models, better scaffolding, repository-specific training, longer adaptation, or different task mixes will change the balance.
The most useful starting point, then, is neither optimism nor skepticism. It is a disciplined question about boundaries: what exactly was the task, what did the human and tool each know, what work did the tool add or remove, and what counted as done? The next part of this series examines the positive result first, in enough detail to see both why the Copilot experiment was convincing and why its precision should not be mistaken for universality.