Type: url
Original: https://arxiv.org/abs/2509.16941
Status: generated
Summary
“SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” (Deng, Da, Pan, He, et al.; arXiv:2509.16941) introduces SWE-Bench Pro, a benchmark of 1,865 problems drawn from 41 actively maintained repositories spanning business applications, B2B services and developer tools, partitioned into a public set of 11 repositories, a held-out set of 12 and a commercial set of 18 proprietary repositories. Its long-horizon tasks may take a professional engineer hours to days and often require patches across multiple files; all are human-verified and given sufficient context to ensure resolvability. The authors also cluster the failure modes observed in collected agent trajectories.