Type: url
Original: https://arxiv.org/pdf/2410.03859
Status: generated
Summary
The paper introducing SWE-bench Multimodal (SWE-bench M), a benchmark of 619 task instances collected from 17 user-facing JavaScript repositories in which every task has at least one image or video in its problem statement or unit tests, built to ask whether AI systems evaluated on the Python-only, largely text-only SWE-bench generalize to visual software domains. The authors find that several top open-source SWE-bench systems were so tailored to Python that they could not be adapted, and that on SWE-bench M the interactive SWE-agent resolves about 12% of tasks against 6% for the next best system, from which they argue that generalizable systems should emphasize LM-driven interaction over fixed, language-specific workflows.