OmniCode: Seeing the Work a Bug-Fixing Score Hides
OmniCode is a benchmark for software development agents that asks a broader question than “can the system repair this issue?” This chapter examines how its task and language denominator exposes different capability profiles, and how its oracles make test generation, review response, and style fixing distinct from ordinary patch synthesis. The practical judgment is to read OmniCode's results as a structured profile of one selected workload—not as a universal ranking of models or agents.
One issue, four kinds of work
A conventional bug-fixing benchmark starts with an issue, asks an agent to modify a repository, and checks whether the resulting patch makes the relevant tests pass without breaking tests that already passed. That is a meaningful capability, but it is only one activity in software development. Engineers also write tests, respond to critique, enforce project rules, and preserve the behavior of a codebase while making changes.
OmniCode keeps a real repository issue as a common starting point and then builds several task types around it. Its base collection contains 494 issues from 28 repositories, distributed across 273 Python, 112 C++, and 109 Java instances. The resulting benchmark contains 1,794 tasks: 963 Python, 418 Java, and 413 C++. The authors selected issue-resolving pull requests that introduced at least one test, manually inspected them for quality, and built containerized environments in which dependencies and test suites could be run. Style-fixing tasks were created across the repositories with language-specific tools.
Those construction choices do two things at once. They widen the opportunities for an agent to demonstrate different skills, and they make the workload more controlled than an arbitrary sample of live engineering. The manual filters remove ill-defined, flaky, or non-runnable cases. The containers make execution reproducible. But the score consequently describes this curated, executable population. It does not estimate how an agent performs on every form of engineering work, including tasks that are mostly configuration, design discussion, performance investigation, or migration.
The four task types use different success conditions:
- Bug fixing asks the agent to resolve the issue described by the repository task. Tests that changed from failing to passing under the original fix establish the target behavior, while previously passing tests guard against regressions.
- Test generation asks the agent to write tests that distinguish the correct fix from plausible incorrect fixes. This is not simply the ability to make a test pass after the bug has been repaired; it is the ability to express behavior sharply enough to reject a family of wrong solutions.
- Review response starts with a meaningful but incorrect proposal and a review explaining its shortcomings. The agent must use the feedback to refine the proposal and produce a patch that satisfies the normal test suite.
- Style fixing supplies findings from language-specific checkers. The agent must resolve the requested violations while preserving functionality and avoiding a net increase in style errors.
These are related activities, but they exercise different points in the software-development loop.