Multi-SWE-bench is a benchmark for issue resolving — the task of modifying a codebase to generate a patch that addresses a given issue — built to cover programming languages beyond Python. It is introduced in Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving, whose motivation for it is that existing benchmarks, of which that paper names SWE-Bench as an example, focus almost exclusively on Python and are therefore insufficient for evaluating large language models across diverse software ecosystems.
Contents
The benchmark spans seven languages: Java, TypeScript, JavaScript, Go, Rust, C and C++. It holds a total of 1,632 instances, which the introducing paper describes as high-quality. One instance poses the issue-resolving task the benchmark is built around — a codebase to be modified, and an issue the resulting patch must address.
Provenance
The 1,632 instances were selected rather than collected wholesale: the introducing paper states they were carefully annotated from 2,456 candidates by 68 expert annotators, and offers that annotation effort as the reason the benchmark can provide an accurate and reliable evaluation.
The authors also open-source the entire data production pipeline behind it, along with detailed tutorials, with the stated aim of encouraging the open-source community to continuously contribute and expand the dataset.
Use
The introducing paper evaluates a series of state-of-the-art models on Multi-SWE-bench using three representative methods — Agentless, SWE-Agent and OpenHands — and reports a comprehensive analysis with key empirical insights.