SWE-bench Lite-S is a filtered variant of the SWE-bench Lite benchmark, constructed for the purpose of more rigorous evaluation and comparison of systems that resolve software issues. It is introduced in Agentless: Demystifying LLM-based Software Engineering Agents alongside the Agentless approach that paper’s headline results are measured with.
Contents
The dataset is SWE-bench Lite with a subset of its problems removed. The authors state the exclusion criterion in terms of what they found wrong with those problems: issues whose ground truth patch is exact, and issues whose descriptions are insufficient or misleading.
Provenance
The filtering is the result of a manual pass over the benchmark rather than an automated one: the authors of the Agentless paper report that they manually classified the problems in SWE-bench Lite, and constructed SWE-bench Lite-S by excluding the problematic issues they identified. The stated motivation is that the original benchmark’s defects affect what a comparison between systems measures, so a filtered version is offered as the basis for that comparison instead.