Type: url
Original: https://arxiv.org/pdf/2603.23448
Status: generated
Summary
Introduces c-CRAB, a benchmark for evaluating automated code review tools that replaces textual-similarity and LLM-as-a-judge scoring with a test-based oracle: human review comments from real pull requests are converted into executable tests that fail on the original patch and pass once the issue is resolved, and a review is scored by whether it guides a coding agent to a fix that passes. The released benchmark holds 184 pull request instances and 234 validated review comments, each with an isolated Docker environment. Evaluating four review tools, the authors report pass rates of 20.1%-32.1% against a human rate of 100%, with 41.5% of tests passed by at least one tool, and argue the gap reflects automated tools raising different categories of concern rather than producing low-quality reviews.