Overview

AI programming agents can learn from executable tests when repository commits become reproducible environments, behavioral tasks, trajectories, and rewards. This chapter follows execution-grounded curation through supervised example selection and sparse reinforcement learning, including repository separation, stopping rules, task solvability, and infrastructure throughput. Passing checks provide a behavioral gate, not a complete definition of good engineering.

Learning outcomes

Execution-grounded curation can turn repository history into reproducible training environments, subject to infrastructure limits.
A passing test labels targeted behavior, but trajectory quality still determines whether a useful demonstration is learned.
Sparse rewards shape policy only when tasks are solvable enough and environments, tests, and stopping rules are reliable.

Keywords

trainingrewardstrajectoriesenvironmentscurationrepositories

Teach the Agent with Tests

For AI programming agents, an executable test can do more than reject a finished patch: it can determine which experiences enter training and which outcomes change the policy. This chapter follows that lifecycle from a repository commit to a reproducible environment, an execution-grounded task, an agent trajectory, and finally a supervised example or reinforcement reward. The judgment to carry forward is whether the learning signal preserves the behavior that matters without pretending that a binary test captures all of engineering quality.

The shift is subtle but consequential. In evaluation, the test is mainly a gate through which a candidate passes or fails. In training, the same test helps construct the world in which the agent learns what actions lead to success. A weak evaluator can mismeasure capability; a weak training environment can teach the wrong habit repeatedly. The executable specification has become part of the agent’s development process.

From a repository commit to a training example

Human-written issues and tests are valuable because they contain the history of what a team considered wrong and how the team verified the fix. They are also scarce. A repository can contain far more meaningful commits than documented issues with complete, runnable test cases. If training environments depend only on the latter, the data supply is limited by the rate at which people wrote and preserved those artifacts.

R2E-Gym and its SynGen curation recipe use a different starting state: repository commit history. The pipeline identifies relevant changes, constructs executable environments for historical commits, and collects tests that fail on the buggy state but pass on the fixed state. When a commit lacks a suitable test, the pipeline supplements it with a generated fail-to-pass test. The result is not merely a patch archive. It is an environment in which the agent can observe the old behavior, change the code, run the checks, and determine whether the proposed repair restores the targeted behavior.

The task description is generated only after that execution evidence exists. The curation process supplies the commit metadata and patch context along with failing test functions, error messages, expected behavior, and the difference between the old and new test outcomes. The language model then writes a concise issue description that describes the problem without simply revealing the solution. This is execution-grounded backtranslation: the prose is reconstructed from the behavior that the tests expose rather than from a generic summary of the changed lines.

That ordering matters. A code diff can show what a developer changed, but it does not necessarily state what a user experienced or which edge case made the change necessary. A generic summary can sound plausible while omitting the behavior that distinguishes the bug. A failing test and its transition to passing provide a directional constraint. They anchor the task description to a concrete discrepancy: this condition failed before, this condition passes after, and the agent must discover a repair that crosses that boundary.

The lifecycle also clarifies what must persist across the transition. The old and fixed repository states need a reproducible environment; the test must remain runnable against both states; the task description must retain the behavioral problem without giving away the patch; and the agent must receive observations from the same kind of workspace in which the training example was created. If one of those links changes, the trajectory may teach a different task from the one the test originally described. The patch is not the whole training object.

Published by DeepMox

FAQ

How can repository commits become coding-agent training tasks?

They can become tasks by pairing historical buggy and fixed states with reproducible environments, fail-to-pass tests, context, and execution-grounded descriptions.

Why is a passing test not enough for a training example?

A passing test is not enough because a trajectory may be brittle or wasteful and the test may miss regressions; outcome labeling needs trajectory selection and task-quality judgment.

What makes a sparse coding-agent reward useful?

It is useful when enough tasks are solvable, the environment runs reliably, the stopping rule is clear, and the reward is not easy to obtain through shortcuts.

Previous8 minAI Programming Agents: Designing Feedback Contracts

Next9 minAI Programming Agents: Verifying Candidate Patches

AI Programming Agents: Training with Tests

Teach the Agent with Tests

From a repository commit to a training example

FAQ