Overview

c-CRAB evaluates code-review agents by translating selected, actionable human concerns into executable tests: the original pull request must fail and a corrected version must pass. This shifts evaluation from comment similarity to whether feedback leads a coding agent to fix the encoded issue. Automated tools passed 20.1%–32.1% of the benchmark’s tests, while a small manual sample found 84% useful comments; these measure different constructs, and the curated benchmark does not represent all review quality.

Learning outcomes

c-CRAB tests whether review feedback leads a coding agent to correct a human-identified concern, not whether comments share wording.
Its 20.1%–32.1% automated pass rates measure alignment with curated human concerns, not total review quality.
Fail-then-pass tests improve behavioral grounding but remain limited by human selection, downstream agent ability, and structural overfitting.

Keywords

c-CRABcode revieworaclestestsalignmentmaintainability

c-CRAB: Turning Review Intent into Executable Evidence

Code-review agents are evaluated on a particularly slippery question: did a comment identify a meaningful issue, or did it merely sound like something a human reviewer might have written? c-CRAB, the Code Review Agent Benchmark, focuses on the gap between wording and intent. This chapter follows how it turns selected human concerns into executable tests, then interprets the resulting human–automated gap without treating alignment with human reviews as the same thing as total review quality.

From a noisy comment to a testable issue

A human review discussion contains more than defects. It may include a question, a request for clarification, an acknowledgment, a stylistic preference, or a conversation about several possible fixes. Those comments can be useful to the people working on a change, but they do not all define a problem that can be translated into an objective test. c-CRAB begins by retaining comments that identify a specific, actionable, and objectively verifiable concern.

The benchmark then asks a different question of the retained comment: can its underlying concern be expressed as an executable oracle? The curation pipeline starts with a pull request, its candidate patch, and the associated human review. It constructs an execution environment containing the relevant repository state and dependencies. It aligns the comment with the affected code and supplies the review text, diff context, and file context to a test-generation process. The generated test must fail on the original pull request and pass on the version in which the reviewed issue has been addressed.

That fail-then-pass condition is the key transition. A comment is no longer treated as ground truth because of its vocabulary or position in a diff. Its claim is represented by a test that distinguishes the state under review from a corrected state. If the test cannot make that distinction, the comment or generated oracle does not become a reliable benchmark target. Execution feedback can be used to refine a candidate test, but the final condition remains behavioral: the test must fail before the issue is fixed and pass afterward.

sequenceDiagram participant H as Human review participant C as Curation pipeline participant E as Execution environment participant T as Executable test participant R as Automated review tool participant A as Coding agent H->>C: Retain actionable concern C->>T: Synthesize oracle from comment and code context C->>E: Run test on original PR E-->>C: Fail before correction C->>E: Run test on corrected version E-->>C: Pass after correction R->>A: Send generated review comments A->>E: Revise patch using review feedback E->>T: Execute validated oracle T-->>A: Pass or fail for the human-identified issue

The benchmark is therefore not measuring whether an automated reviewer reproduces a human sentence. It is measuring whether the review feedback can guide a coding agent toward a change that satisfies an executable representation of a human-identified concern. The coding agent is not shown the test during evaluation; the test is reserved for post-hoc verification. That separation matters because otherwise the agent could optimize directly for the oracle rather than interpret the review.

The process makes the test a durable boundary between review and correction. On the input side, the human comment supplies intent, but noisy or non-actionable conversation is filtered out. In the middle, the curation pipeline supplies repository state and execution infrastructure, and test synthesis translates natural language into an observable condition.

Published by DeepMox

FAQ

How does c-CRAB turn a human review comment into a benchmark test?

It retains a specific, actionable, objectively verifiable concern and creates a test that fails on the original pull request and passes after correction. The test is reserved for post-hoc verification rather than shown to the coding agent.

What does a c-CRAB pass rate tell you about a review tool?

It indicates how often the tool’s feedback leads a coding agent to resolve a curated human-identified concern under c-CRAB’s protocol. It does not measure every useful review finding or total review quality.

Why can c-CRAB show low alignment but useful comments?

Because alignment with selected human concerns and broader comment usefulness are different constructs. A tool can miss the benchmark’s encoded concerns while still identify valid issues in a small manual sample.

Previous11 minOmniCode Benchmark: Beyond Bug Fixing

Next12 minBuild a Coding-Agent Evaluation Portfolio

c-CRAB Benchmark for Code Review Agents

c-CRAB: Turning Review Intent into Executable Evidence

From a noisy comment to a testable issue

FAQ