This paper extends the issue resolving benchmark SWE-Bench, which asks a model to produce a patch from an issue description and the buggy repository, beyond Python. The authors argue that SWE-bench’s focus on Python limits its evaluation to fields such as data processing and artificial intelligence, leaving out areas like web, mobile and system programming that rely on other languages. As a first step toward a multilingual benchmark they build a Java version, choosing Java for its popularity in industry and, over C and C++, because they believe language models are not primarily designed to address the performance issues those languages are chosen for.
The resulting benchmark, SWE-bench-java-verified, follows SWE-bench’s construction workflow in five phases: collecting candidate repositories, crawling issue instances from pull requests, determining each issue’s runtime environment, extracting fail-to-pass tests, and a questionnaire-based manual verification following the SWE-bench Verified annotation guidelines. The paper also describes problems found when migrating the pipeline to Java — an error in the original SWE-bench script that sometimes crawls the wrong base commit, redundant downloads of repositories and dependencies, and compilation breaking during incremental compilation — and how the authors addressed them.
To check the benchmark’s reliability, the authors run SWE-Agent with several models. Resolved rates are low across the board, which they read as evidence that the benchmark is challenging and discriminates between models.
Key Points
- The benchmark is presented as a first step toward multilingual GitHub issue resolving evaluation, starting with Java; the authors plan to add languages such as Go, Rust, C and C++.
- Construction narrowed 70 candidate repositories to 19, crawled 1,979 issue instances from them, kept 308 that compile under the determined environment, 137 with at least one fail-to-pass test and no pass-to-fail test, and after manual verification 91 issues across 6 repositories.
- Manual verification by 10 Java developers rated issue clarity, test coverage and major flaws, keeping only issues rated clear, well covered and free of major flaws.
- The authors report fixing a bug in the original SWE-bench collection script, which sometimes took the wrong base commit by ignoring branch differences, by using the git commit graph.
- With SWE-agent, resolved rates range from 1.10% (GPT-4o-mini, Doubao-pro) to 9.89% (DeepSeek-V2) of the 91 issues, with GPT-4o at 6.59% and DeepSeek-Coder-V2 at 7.69%.
- The authors observe that DeepSeek-V2 outperforms DeepSeek-Coder-V2 on the repository with the most extensive issue descriptions and the reverse on the one with the least text, which they take to suggest that more detailed task descriptions demand more natural-language understanding.
Notes
The authors state that they implemented SWE-agent hastily, without configuring the runtime environment for all Java issues, which affects the agent’s ability to reproduce issues and may make its results lower than its normal performance. The dataset, evaluation environment and leaderboard are open-sourced, and the paper’s project links point to multi-swe-bench.github.io and a Hugging Face dataset named Daoguang/Multi-SWE-bench. The work is set among benchmarks for Software Issue Resolution, and the paper cites the multilingual code generation benchmarks HumanEval-X, MBXP and MultiPL-E as prior multilingual efforts outside issue resolving.