Type: url
Original: https://arxiv.org/pdf/2408.14354
Status: generated
Summary
The paper introducing SWE-bench-java-verified, a Java version of the SWE-bench GitHub issue resolving benchmark built as a first step toward multilingual issue resolving evaluation, with a Docker-based evaluation environment and leaderboard. It describes a five-phase construction process that narrowed 1,979 crawled issue instances to 91 manually verified issues across 6 Java repositories, and reports resolved rates between 1.10% and 9.89% for SWE-agent with GPT-4o, GPT-4o-mini, DeepSeek-V2, DeepSeek-Coder-V2 and Doubao-pro.