This paper, by authors from Stanford University, Princeton University, Cornell University, the University of Tübingen and Meta AI, asks whether AI systems for software engineering generalize beyond the setting they are usually evaluated in. SWE-Bench draws only on Python repositories, and the paper notes that only 5.6% of its tasks contain an image, even though many domains of software development — user interface design, games, data visualization — rely on visual assets. To probe this, the authors build SWE-bench Multimodal (SWE-bench M): real GitHub issues from user-facing JavaScript libraries, filtered so that every task has an image or video in its problem statement or tests, and checked by human annotators.
The authors then try to run the top open-source systems from the SWE-bench leaderboard on it. Some proved so tailored to Python and SWE-bench that they could not be adapted: AutoCodeRover and Moatless rely on Python-specific program analysis, and the authors chose not to benchmark them, while Agentless needed a JavaScript parser written from scratch. They evaluate RAG, three configurations of SWE-Agent (Base, a JavaScript-adapted editor, and a multimodal version with a browser, screenshots and image viewing) and their adapted Agentless JS, with GPT-4o and Claude 3.5 Sonnet.
The interactive SWE-agent configurations clearly outperform the others, and the paper draws a design conclusion from this: generalizable LM-based systems for software engineering should emphasize interaction rather than fixed problem-solving workflows, leaving the burden of problem solving on the LM instead of manually engineered, language-specific pipelines.
Key Points
- SWE-bench M contains 619 task instances from 17 JavaScript repositories, split into a 517-instance test set from 12 repositories and a 102-instance development set from 5 repositories (the abstract states 617 instances).
- Human annotators judged the images necessary for solving the task in 83.5% of the 557 instances whose problem statements contain images.
- Annotators estimated that SWE-bench M tasks take longer to solve than SWE-bench tasks, and its reference solutions edit more files, functions and lines; 28% of them modify two or more file types.
- Several leading SWE-bench systems could not be adapted without extensive redesign, because their localization steps depend on Python-only parsing; the paper argues that such rigid workflows fail under minor distribution shifts.
- On the test set the SWE-agent configurations resolve 9.2–12.2% of tasks, against 3.1–6.2% for Agentless JS and 5.0–6.0% for RAG.
- Swapping the underlying LM, or adding JavaScript-specific and multimodal tooling to SWE-agent, made little difference to overall resolution in the main results, and the authors describe the effect of the multimodal tools as ambiguous overall.
- Removing images from the input lowered performance for both RAG and SWE-agent JS on the development set, especially where annotators judged the images necessary.
- A temporal analysis found no evidence that solutions leaked into the models’ training data gave an advantage on the test set.
Notes
The paper’s related-work section places SWE-bench M at the meeting point of repository-level software engineering benchmarks and multimodal code generation, and argues that web navigation and software engineering agents have usually been studied separately. The authors list as limitations that the benchmark covers only JavaScript, only images and videos among possible media, and a limited set of domains, and that adding tasks was labor-intensive. They also ask practitioners to iterate on the development split and evaluate on the test split only once an approach is final. The multimodal SWE-agent extends the Agent-Computer Interface idea with browser and image tools.