Measure the Repair Loop, Not the Lines

For the cloud Opus configuration and the private quantized-GLM configuration, the decisive quality question is not which one generated more activity. It is how much of that activity became accepted software without a separate repair burden. A repair-aware reading of the repository evidence helps distinguish usable output from lines, tokens, or commits that merely record motion.

This distinction is especially important for autonomous agents. The agent can create a change, run validation, interpret a failure, and create another change within the same work session. Some mistakes disappear inside that loop and never become separate commits. Others survive long enough to require a dedicated repair commit. The enterprise study uses the latter signal to compare the two configurations, then supplements it with change-footprint strata, defect categories, and timestamp patterns.

Code volume is a necessary but insufficient denominator

At first glance, the repository data might suggest that the cloud period produced more software. Period A recorded 129,556 inserted lines, compared with 93,281 in Period B. That comparison is misleading because the file mix changed. Markdown specification files accounted for 36.6% of Period A’s insertions but only 10.1% of Period B’s. Once the analysis excludes documentation, lockfiles, and other non-code material, code-only insertions are much closer: 75,738 for the Opus period and 80,258 for the GLM period.

The commit counts move in the opposite direction. The local period contains 367 non-merge commits, while the cloud period contains 246. More commits and similar code-only insertion volume do not automatically mean more progress. A repository can record many small corrections around a smaller amount of new logic. The measurement must therefore ask what kind of commit each one represents.

The study’s main signal is the Fix Commit Ratio, or FCR: the share of a configuration’s non-merge commits whose message is classified as a repair using a deterministic keyword rule. Messages containing terms such as “fix,” “bug,” “issue,” “patch,” or “correct” are counted as repair-oriented. Under that rule, the Opus period has 113 fix commits out of 246, an FCR of 45.93%. The GLM period has 275 out of 367, an FCR of 74.93%.

That does not mean that 74.93% of every line generated by the local agent was defective. It means that a much larger share of the observed commit stream was devoted to changes labeled as repairs. This is a more defensible claim because it stays at the level of the evidence. It is also the claim that matters economically: a repair commit consumes validation, review, and developer attention that a new feature commit might otherwise have used.

The paper uses a coarse companion measure, effective productivity, defined as one minus FCR. On that measure, 54.07% of Opus-period commits remain after discounting the repair share, versus 25.07% of GLM-period commits. This is not a claim that every non-fix commit has equal value or that every repair has zero value. It is a directional bound on how much of the commit activity is not classified as defect correction. A repair can improve the product, and a feature commit can still contain a defect; the measure should not be mistaken for a complete value function.

The choice of commits as the denominator is deliberate. A line-based defect rate would require deciding which lines are defective and how to assign a later correction to an earlier change.