Measure Net Progress, Not AI Activity
The central measurement problem in the Copilot and METR studies is not a shortage of numbers. It is the risk of counting the wrong thing. Developers and experts predicted that AI would save time, and a tool can generate or accept large amounts of code, while the time required to deliver a reviewed, safe change stays the same or increases. A defensible pilot therefore needs to measure the path from fixed work to completed work, using experience and activity data to explain the result rather than allowing them to stand in for it.
Why confident forecasts went wrong
METR asked developers to predict how much AI would change the time required for their issues. Before the work began, they forecast a 24% reduction. After participating, they still estimated a 20% reduction. The measured result went in the opposite direction: AI-allowed issues took 19% more time. Experts were even more optimistic, forecasting reductions of about 39% among economics experts and 38% among machine-learning experts.
This is not evidence that developers or experts were careless. Their forecasts contained useful information. The time developers predicted an issue would take was correlated with the time it actually took: the correlation was 0.64 for AI-allowed issues and 0.59 for AI-disallowed issues. In other words, participants could broadly recognize that one issue was harder than another. Their systematic error concerned the value of AI, not the relative difficulty of the work.
That distinction matters for decision making. A developer may correctly say that a large refactor will take longer than a small bug fix, then still underestimate the review and cleanup cost of asking an assistant to perform the refactor. An expert may understand the capabilities of a model on benchmarks and still lack evidence about how those capabilities interact with a maintainer’s local knowledge, a repository’s conventions, and a real pull-request process. Good intuition about the task does not automatically produce good intuition about the intervention.
The Copilot experiment shows the other direction of the same problem. Participants estimated about a 35% productivity gain, while their observed completion-time difference was 55.8% on the standardized task. Here the users underestimated the benefit; in METR’s setting, users overestimated it. The contrast is a reminder that post-hoc enthusiasm, disappointment, or confidence cannot replace a comparison with a defined baseline.
Why activity is not productivity
Software teams often reach for metrics that are easy to collect. Lines of code, commit counts, pull requests, accepted suggestions, usage frequency, and spending can reveal that a tool is being used. They can help identify where a workflow is active or where people may need support. None of them, by itself, establishes that more valuable software was delivered.
METR explains why output proxies can mislead. AI can make code more verbose without changing its function. Developers can split one unit of work into several commits or pull requests without increasing the total work completed. A high count may therefore describe a change in the shape of activity rather than a change in productivity. The same warning applies to generated or accepted lines: more text can mean more implementation, more boilerplate, or more material that a human must inspect and remove.
Spend has a similar ambiguity.