This arXiv preprint introduces SWE-Bench Pro, presented as a substantially more challenging benchmark that builds upon the best practices of SWE-Bench while being explicitly designed to capture realistic, complex, enterprise-level problems beyond that benchmark’s scope.
The paper gives the benchmark’s composition in detail. It contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services and developer tools, and is partitioned three ways: a public set with open access to problems sourced from 11 repositories; a held-out set of 12 repositories; and a commercial set of 18 proprietary repositories, which the authors state they hold under formal partnership agreements with early-stage startups. Problems in the held-out and commercial sets are not publicly accessible, though the authors state they release results on the commercial set.
What distinguishes the tasks is their length. The authors describe the benchmark as featuring long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. They also state that all tasks are human-verified and augmented with sufficient context to ensure resolvability.
Alongside the benchmark the authors report an analysis of how agents fail on it: they cluster the failure modes observed in the collected agent trajectories, for what they describe as a clearer characterization of the error patterns exhibited by current models. They summarize the whole as a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.
The preprint is filed under Software Engineering (cs.SE), with a cross-listing to Computation and Language (cs.CL). It was first submitted on 21 September 2025 and revised on 14 November 2025 as version 2, and carries the arXiv-issued DOI 10.48550/arXiv.2509.16941.
Key Points
- The paper introduces SWE-Bench Pro as a substantially more challenging benchmark that builds on SWE-bench’s best practices while targeting enterprise-level problems beyond its scope.
- The benchmark holds 1,865 problems from 41 actively maintained repositories spanning business applications, B2B services and developer tools.
- It is partitioned into a public set (11 repositories), a held-out set (12 repositories) and a commercial set (18 proprietary repositories held under formal partnership agreements with early-stage startups); the latter two are not publicly accessible, though results on the commercial set are released.
- Its tasks are long-horizon: the authors state they may require hours to days of work from a professional software engineer, and often involve patches across multiple files and substantial code modifications.
- All tasks are human-verified and augmented with sufficient context to ensure resolvability.
- The authors cluster the failure modes observed in collected agent trajectories in order to characterize the error patterns current models exhibit.
- The authors describe the benchmark as a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development — their own characterization, not a measured result stated in the abstract.
Notes
Two of the benchmark’s three partitions are stated to be outside public access, and the commercial set rests on formal partnership agreements with early-stage startups. The preprint went through two versions between September and November 2025.