This arXiv preprint addresses a gap it identifies in how issue-resolving systems are measured. It defines the task first — the task of issue resolving is to modify a codebase to generate a patch that addresses a given issue — and then states the problem: existing benchmarks, of which the authors name SWE-Bench as an example, focus almost exclusively on Python, which the authors argue makes them insufficient for evaluating large language models across diverse software ecosystems.
Its answer is Multi-SWE-bench, a multilingual issue-resolving benchmark covering Java, TypeScript, JavaScript, Go, Rust, C and C++. The paper gives both its scale and how that scale was arrived at: 1,632 high-quality instances, carefully annotated from 2,456 candidates by 68 expert annotators, which the authors offer as the reason the benchmark can provide an accurate and reliable evaluation.
On that basis the authors evaluate a series of state-of-the-art models using three representative methods, which they name as Agentless, SWE-Agent and OpenHands, and present what they describe as a comprehensive analysis with key empirical insights.
The paper’s second contribution shifts from evaluation to training. The authors launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning training datasets for issue-resolving tasks, and as an initial contribution release a set of 4,723 well-structured instances spanning seven programming languages. They also open-source the entire data production pipeline along with detailed tutorials, framing this as an encouragement for the open-source community to continuously contribute and expand the dataset. The authors close by envisioning Multi-SWE-bench and the growing Multi-SWE-RL community as catalysts for advancing reinforcement learning toward its full potential.
On arXiv the preprint is filed under Software Engineering (cs.SE), with cross-listings to Artificial Intelligence (cs.AI) and Computation and Language (cs.CL). It was submitted on 3 April 2025 and carries the arXiv-issued DOI 10.48550/arXiv.2504.02605.
Key Points
- The paper’s stated problem is that existing issue-resolving benchmarks focus almost exclusively on Python, which the authors argue makes them insufficient for evaluating LLMs across diverse software ecosystems.
- It introduces Multi-SWE-bench, covering seven languages beyond Python: Java, TypeScript, JavaScript, Go, Rust, C and C++.
- The benchmark’s 1,632 instances were annotated from 2,456 candidates by 68 expert annotators, which the authors present as what makes the evaluation accurate and reliable.
- The authors evaluate state-of-the-art models on it using three representative methods — Agentless, SWE-agent and OpenHands — and report a comprehensive analysis with key empirical insights.
- They launch the Multi-SWE-RL open-source community for building large-scale RL training datasets for issue resolving, releasing 4,723 well-structured instances across seven programming languages as an initial contribution.
- The entire data production pipeline is open-sourced with detailed tutorials, with the stated aim of letting the community continuously contribute and expand the dataset.
Notes
The paper carries two contributions with different purposes — an evaluation benchmark and a training data effort — and links them through a shared task definition and a shared production pipeline. The authors present the released pipeline, rather than the released instances alone, as what makes the second contribution extensible. The claim about existing benchmarks is stated at the level of language coverage; the paper’s own numbers describe Multi-SWE-bench and the initial Multi-SWE-RL release, not the benchmarks it contrasts itself with.