SWE-Bench Verified is a human-validated, 500-task subset of SWE-Bench, discussed in Agentic Software Engineering: Foundational Pillars and a Research Roadmap. It was released by OpenAI to address concerns that some of the original SWE-Bench’s tasks were ambiguous or underspecified, with OpenAI working directly with SWE-Bench’s own authors on the curation.
Contents
The subset holds 500 tasks drawn from SWE-Bench and validated by humans as well-specified and solvable.
Provenance
SWE-Bench Verified was introduced by OpenAI in collaboration with the authors of SWE-Bench. Agentic Software Engineering: Foundational Pillars and a Research Roadmap reports that OpenAI later warned the benchmark was increasingly exposed to data contamination and recommended SWE-Bench Pro as a more reliable evaluation instead.
Use
Agentic Software Engineering: Foundational Pillars and a Research Roadmap cites results on this subset as illustrating rapid progress on agentic coding benchmarks: GPT-4o solved 33% of tasks at its August 2024 launch, while leading agentic solutions exceeded 70% of tasks by mid-2025.