AI Coding Productivity: GitHub Copilot vs METR

Overview

This series explains why GitHub Copilot produced a reported 55.8% faster completion time on a standardized JavaScript HTTP-server task while METR found AI-allowed issues took 19% longer for experienced maintainers in mature repositories. It traces how task structure, human expertise, accessible context, output reliability, review, and the definition of completion change the result, then turns those boundaries into a measurement and governance approach. The evidence supports local, end-to-end pilots—not a universal AI coding multiplier.

Learning outcomes

GitHub Copilot’s 55.8% gain applies to a bounded JavaScript task, not software delivery in general.
METR’s 19% slowdown applies to AI-allowed real issues for experienced maintainers in mature repositories.
Prompting, waiting, review, cleanup, and missing repository context can erase generation-time savings.
Teams should compare net time to a reviewed, secure change before scaling an AI workflow.

Keywords

AIproductivityGitHub CopilotMETRcodingagentscontexttestingreviewmeasurement

Chapters

GitHub Copilot vs METR: Is AI Coding Faster?

Why task, context, and completion boundaries matter

9 min · Why task, context, and completion boundaries matter

9 min

GitHub Copilot’s 55.8% Gain Explained

The task, stopwatch, and limits behind the headline

9 min · The task, stopwatch, and limits behind the headline

9 min

METR’s 19% Slowdown in Real Software Work

The field experiment behind the headline

10 min · The field experiment behind the headline

10 min

METR’s Hidden Cost of Missing Context

How prompting, review, and tacit knowledge add overhead

11 min · How prompting, review, and tacit knowledge add overhead

11 min

GitHub Copilot and METR: Measure Net Progress

Why activity, forecasts, and satisfaction are not enough

10 min · Why activity, forecasts, and satisfaction are not enough

10 min

GitHub Copilot and METR: A Governed Workflow

How context, feedback, and security bound agentic coding

11 min · How context, feedback, and security bound agentic coding

11 min

GitHub Copilot vs METR: When Is AI Coding Faster?

A conditional model for local decisions

12 min · A conditional model for local decisions

12 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-7 Premium

FAQ

Why do the Copilot and METR studies show opposite productivity results?

They show opposite results because they measured different work, users, tools, context, and completion boundaries. One used a bounded test-based task; the other included real repository work through review.

What does GitHub Copilot’s 55.8% result actually measure?

It measures completion time for a standardized JavaScript HTTP-server task among participants who completed it, ending at the first commit that passed twelve visible tests.

What does METR’s 19% slowdown mean?

It means AI-allowed issues took 19% more total implementation time on average in METR’s study of experienced open-source maintainers, including work around pull-request review. It does not establish that AI slows all developers or software work.

How should a team test whether AI coding is faster?

Define fixed work and a consistent finish line, compare AI-assisted and human paths, and measure time through review, quality, security, and remediation—not generated code alone.