This paper asks whether LLMs can reliably judge whether a code implementation satisfies its task description, a check software engineers increasingly delegate to them. Using widely adopted benchmarks and a unified prompt design, the authors report a systematic failure: LLMs frequently misclassify correct implementations as non-compliant or defective — the “overcorrection” of the title.
The paper also finds that prompt design pushes the error in an unexpected direction: more detailed prompts, particularly those that require the model to give explanations and propose corrections, lead to higher misjudgment rates. The authors analyse the mechanisms behind these failures and evaluate how reliable rationale-required judgments are. As a remedy they propose a Fix-guided Verification Filter, which treats the fix a model proposes as executable counterfactual evidence and validates both the original and the revised implementation using benchmark tests and spec-constrained augmented tests.
Key Points
- LLMs used to check code against natural-language requirements frequently label correct code as non-compliant or defective.
- More detailed prompts — especially ones requiring explanations and proposed corrections — produce higher misjudgment rates, not lower ones, which the authors present as a critical reliability issue for LLM-based code assistants.
- The proposed Fix-guided Verification Filter uses the model’s own suggested fix as executable counterfactual evidence, validating both the original and revised implementations with benchmark tests and spec-constrained augmented tests.
- The authors frame their results as exposing under-explored limitations of LLM-based code review and as practical guidance for adding safeguards when LLM reviewers are integrated into automated review and development pipelines.
Notes
The findings are drawn from widely adopted benchmarks with a unified prompt design. The paper bears on the use of LLMs as automated reviewers in Agentic Code Review and on the reliability of model-based judging more generally, as in LLM-as-a-Judge.