SWE-Bench turned real GitHub issues into an evaluation for language models and coding agents, and a number of later datasets reuse its format while changing something about it. This page sets those datasets side by side, draws out what each one changes, and collects what the rest of the wiki records about how the family is used, built on and criticised. It does not rank the benchmarks.
What they share
Every dataset here keeps the same basic task: an instance gives a codebase together with an issue to
resolve, and the system under evaluation produces a patch. Software Issue Resolution
is the name the wiki uses for that task, and
Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey states it formally: an
instance I = (D, C, T) holds an issue description, a codebase and tests, only the description and
codebase are observable while resolving, and the patch is evaluated by running the tests. Its
aggregate metric is the Resolved Rate, the mean of per-instance binary outcomes; the benchmark’s own
site reports % Resolved for each member of the family it hosts.
For most of the datasets, tests decide whether the patch counts. SWE-Bench checks correctness with unit tests and continuous integration; SWE-bench Multimodal uses fail-to-pass and pass-to-pass tests from the corresponding pull request; and SWE-bench-java-verified counts an issue as resolved only if all the given test cases pass. SWE-Bench+ states that each of its instances follows the SWE-bench format, and was collected with SWE-bench’s own open-source scripts. The pages for SWE-Bench Verified, SWE-bench Lite-S, SWE-Bench Pro and Multi-SWE-bench do not describe how patches are checked.
Side by side
The first eight rows are datasets with their own page in this wiki. The last three are members the benchmark’s own site lists, recorded on the SWE-Bench page, without pages of their own.
| Dataset | Size as its page gives it | Language | Who built it | What it changes relative to SWE-bench |
|---|---|---|---|---|
| SWE-Bench | 2,294 problems from 12 Python repositories | Python | The authors of SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | — (the original) |
| SWE-Bench Verified | 500 tasks | Python | OpenAI, with the SWE-bench authors | A human-validated subset, addressing tasks that were ambiguous or underspecified |
| SWE-bench Lite-S | Not stated | Not stated on its page | The authors of the Agentless paper | A manually filtered version of SWE-bench Lite, for more rigorous comparison |
| SWE-Bench+ | 548 tasks | Python (the SWE-bench projects except Django) | The authors of SWE-Bench+: Enhanced Coding Benchmark for LLMs | Issues created after the evaluated models’ training cut-offs, screened for solutions in the issue text |
| SWE-Bench Pro | 1,865 problems from 41 repositories | Not stated on its page | The authors of SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? | Long-horizon, enterprise-level tasks, with partitions that are not publicly accessible |
| Multi-SWE-bench | 1,632 instances | Java, TypeScript, JavaScript, Go, Rust, C, C++ | The authors of Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving | Languages beyond Python |
| SWE-bench-java-verified | 91 issues from 6 repositories | Java | The authors of SWE-bench-java: A GitHub Issue Resolving Benchmark for Java | A Java version, screened with the SWE-bench Verified annotation guidelines |
| SWE-bench Multimodal | 619 instances from 17 repositories (paper); 480 (benchmark site) | JavaScript / TypeScript | The authors of SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? | Every task contains visual content |
| SWE-bench Lite | 300 instances | Python | Listed on the SWE-bench site | A subset curated for less costly evaluation |
| SWE-bench Multilingual | 300 instances from 42 repositories | 9 programming languages | Listed on the SWE-bench site | Tasks across several languages |
| Bash Only | The same 500 instances as Verified | Python | Listed on the SWE-bench site | The default view of the Verified leaderboard, in which every model runs in the same Mini-SWE-Agent environment |
One size is given differently by different sources: for SWE-bench Multimodal, the introducing paper reports 619 task instances (its abstract states 617), split into a 517-instance test set and a 102-instance development set, while the benchmark’s site lists 480 instances. For SWE-bench Lite, MASAI: Modular Architecture for Software-engineering AI Agents adds that its 300 issues come from 11 Python repositories.
Order of appearance
The pages give dates for several members, which puts them in this order:
- October 2023 — the SWE-bench paper is first submitted, on 10 October 2023.
- March 2024 — SWE-bench Lite is released, according to the benchmark’s site.
- June 2024 — the site records SWE-bench being Docker-ized for easier evaluation.
- July 2024 — the Agentless paper, which constructs SWE-bench Lite-S, is first submitted on 1 July 2024.
- August 2024 — SWE-Bench Verified is announced as a collaboration with OpenAI.
- October 2024 — SWE-bench Multimodal is introduced, according to the site.
- April 2025 — the Multi-SWE-bench paper is submitted on 3 April 2025.
- September 2025 — the SWE-Bench Pro paper is first submitted on 21 September 2025.
SWE-Bench+ collects issues created from 2023-11-01 to 2024-08-22; its page gives no release date. SWE-bench-java-verified’s page gives none either.
Where they differ
The variants answer different complaints about the original, and they fall into roughly four groups by which complaint they take up.
Whether a task is well specified
SWE-Bench Verified was released to address concerns that some SWE-bench tasks were ambiguous or underspecified; its 500 tasks were validated by humans as well-specified and solvable. SWE-bench Lite-S deals with the same kind of problem through its authors’ own classification: the Agentless authors manually classified the problems in SWE-bench Lite and excluded issues whose ground-truth patch is exact and issues whose descriptions were insufficient or misleading. SWE-bench-java-verified carries the Verified approach into Java: ten developers experienced in Java rated issue clarity, test coverage and major flaws following the SWE-bench Verified guidelines, and only instances meeting all three criteria were kept, taking 137 candidates down to 91. SWE-Bench Pro states that all of its tasks are human-verified and augmented with sufficient context to ensure resolvability.
Other pages record how much underspecification remains a factor on the older sets. Of the 207 SWE-bench Lite issues SpecRover: Code Intent Extraction via LLMs left unresolved, it counts 107 with ambiguous issue descriptions.
Whether the answer is already available to the model
Two different leaks are at issue here. Solution Leakage is the name the SWE-Bench+
authors give to an instance whose issue report or comments already spell out the fix; because
SWE-bench gives both to the model as input (the comments as hints_text), a model can copy the fix
rather than work it out. The other concern is that issues predating a model’s training cut-off may
have been seen in training; the SWE-Bench+ paper states that 94% of SWE-bench’s issues and pull
requests were created before the cut-offs of the models it considers.
SWE-Bench+ addresses both at once: it takes issues created from 2023-11-01 to 2024-08-22, starting a month after the latest cut-off among the models its authors used, and checks every instance by hand to remove those with clear solution details in the issue report. SWE-Bench Pro addresses exposure differently, by withholding: of its 41 repositories, 11 form a public set, 12 a held-out set and 18 a commercial set of proprietary repositories, and problems in the held-out and commercial sets are not publicly accessible. Its authors describe it as contamination-resistant.
Agentic Software Engineering: Foundational Pillars and a Research Roadmap describes a progression from SWE-Bench to SWE-Bench Verified, addressing tasks that were ambiguous or underspecified, and then to SWE-Bench Pro, which OpenAI recommended after warning that SWE-Bench Verified was increasingly exposed to data contamination.
The variants do not all find contamination to matter in the same way. The SWE-bench Multimodal paper reports a temporal analysis that found no evidence that solutions leaked into the models’ training data gave an advantage on its test set. FeatureBench’s paper observes that task performance depends far more on the amount of code required than on a task’s commit date. Several system papers raise the concern about their own results: Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration names the possibility that the models it used saw parts of the test repositories in training, and SpecRover checks for memorization by counting patches syntactically identical to the ground truth, finding 9 of its 93 resolved Lite issues. The DeepSWE post, DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL, filtered its training tasks to exclude repositories that also appear in SWE-Bench-Verified to avoid contamination.
Which languages and domains are covered
The original is restricted to Python repositories. Multi-SWE-bench covers seven other languages, and its introducing paper motivates it by saying that existing benchmarks, SWE-bench among them, focus almost exclusively on Python. SWE-bench-java-verified is presented by its authors as a first step toward multilingual issue-resolving evaluation; its paper argues that SWE-bench’s Python focus confines it to fields such as data processing and artificial intelligence, leaving out web, mobile and system programming, and says the authors plan to add Go, Rust, C and C++. That paper’s project links point to multi-swe-bench.github.io and a Hugging Face dataset named Daoguang/Multi-SWE-bench. The benchmark’s site also lists a SWE-bench Multilingual set covering 9 languages.
SWE-bench Multimodal changes both language and domain: its tasks come from user-facing JavaScript libraries, every task has visual content in its problem statement or tests, and it was built to ask whether systems developed for the Python-only SWE-bench generalize to other languages and to problems that have to be understood visually. Its paper notes that only 5.6% of SWE-bench’s tasks contain an image. A subset of 69 of its tasks is checked with pixel-level visual testing that compares rendered screenshots.
Moving off Python exposed how tightly some systems had been fitted to the original. The Multimodal paper reports that AutoCodeRover and Moatless rely on Python-specific program analysis and were not benchmarked, and that Agentless needed a JavaScript parser written from scratch; it attributes Agentless’s weak results there to a localization module designed around Python.
What kind and size of task
SWE-Bench Pro is the variant built around the size of the task: its introducing paper describes its problems as long-horizon ones that may take a professional engineer hours to days, often patching multiple files with substantial modifications, from repositories spanning business applications, B2B services and developer tools. By comparison, the survey cited on the SWE-bench page characterizes SWE-bench’s own mix as 65% function-level tasks, 25% module-level and under 10% project-level. AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities, using SWE-Bench as a case study, reports that commonly used coding benchmarks are heavily biased toward Python and typically evaluate small, self-contained, or function- and module-level problems. The Multimodal paper’s annotators estimated its tasks take longer to solve than SWE-bench tasks, with reference solutions editing more files, functions and lines.
Task kind is a separate axis from size. The FeatureBench paper, FeatureBench: Benchmarking Agentic Coding for Complex Feature Development, puts feature requests at only about 18–22% of SWE-bench instances and argues that pull-request-based collection cannot capture features that span several pull requests.
How the curation was done
The variants also differ in how their instances were chosen:
- Human validation: SWE-Bench Verified (validated by humans as well-specified and solvable).
- Manual verification by annotators: Multi-SWE-bench (1,632 instances annotated from 2,456 candidates by 68 expert annotators, offered by its authors as the reason it can provide an accurate and reliable evaluation), SWE-bench-java-verified (ten Java developers).
- Manual inspection by the authors themselves: SWE-bench Lite-S, SWE-Bench+ (every instance, for solution details), and SWE-bench Multimodal (every remaining instance, removing 24 judged impossible).
- Stated as human-verified and augmented with context: SWE-Bench Pro.
Several variants also changed the collection pipeline itself. SWE-Bench+ reused SWE-bench’s attribute and execution filters. The SWE-bench-java paper reports fixing a bug in the original SWE-bench collection script, which sometimes took the wrong base commit by ignoring branch differences. SWE-bench Multimodal added Node.js and Chrome support to SWE-bench’s Docker setup and removed tests whose results were inconsistent across repeated runs. Multi-SWE-bench’s authors open-source their whole data production pipeline, with the stated aim that the community keep extending the dataset, and pair the benchmark with Multi-SWE-RL, a community effort releasing 4,723 instances across seven languages as reinforcement-learning training data.
What the variants report about the original
Several of these pages record findings about SWE-bench that go beyond the complaint each variant was built around. The SWE-Bench+ paper reports that, of 251 SWE-Agent+GPT-4 patches that passed all tests on the full SWE-bench, 32.67% were solution leaks; filtering suspicious patches drops that system’s resolution rate from 12.47% to 3.97% in its abstract (its section 2.2 gives 5.49%). It also identified instances with the solution in the issue in SWE-bench Lite (18) and SWE-bench Verified (37), and reports suspicious fixes reducing SWE-Agent+GPT-4’s rates from 18% to 9.33% on Lite and from 22.4% to 10.0% on Verified — so human validation for specification quality, as the Verified page describes it, did not on its own remove solution leakage.
Filtering for one defect did not remove the other, either. On SWE-Bench+, solution leakage no longer appears but weak tests persist: about 67.72% of instances marked resolved did not truly resolve the issue, and the systems’ validated resolution rates are far below their reported SWE-bench figures. The SWE-bench page cites the same line of findings through Agentic Software Engineering: Foundational Pillars and a Research Roadmap: 29.6% of “plausible” fixes introduced regressions or were incorrect on retesting, and passing tests alone does not establish that a patch is merge-ready.
A system paper reached a related measurement from the other side. SpecRover’s authors inspected the 93 SWE-bench Lite patches it resolved and found 56 (60.2%) semantically equivalent to the developers’ patches, with 29 of the remaining 37 modifying the same methods as the ground truth.
How the rest of the wiki reads the family
Pages beyond the variants themselves take up the family in five ways.
As a measurement whose score depends on more than the model
- Quantifying infrastructure noise in agentic coding evals and Infrastructure noise report a crossover experiment on SWE-bench, varying available RAM up to 5x across 227 problems, that moved the score by 1.54 percentage points — the same monotonic effect as on Terminal-Bench 2.0 at a smaller magnitude, which the term’s page attributes to SWE-bench’s tasks being less resource-intensive.
- From Question Answering to Task Completion: A Survey on Agent System and Harness Design compiles SWE-bench Verified leaderboard data showing that, within one model, harness choice moves Claude 3.5 Sonnet from 33.6% with SWE-agent to 53.6% with PatchPilot, and cautions that the table is observational rather than a controlled experiment.
- Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering describes SWE-bench, among others, as producing a single number from a single model, harness and environment, with no signal at the level of individual components.
- Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering compiles a SWE-bench Verified timeline from vendor reports and papers, rising from 1.96% in October 2023 to 78.4% in April 2026, and reads the gain as dominated by scaffolding rather than raw model capability; it marks its figures as approximate.
- Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — A Source-Code Study of Eleven Systems distinguishes an evaluation harness such as SWE-bench’s, which wraps an agent rather than a model, from an agent harness.
- Devstral separates comparisons made under the same scaffold from comparisons with models evaluated under any scaffold.
As a benchmark whose scores need qualifying
- Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey reports that agent success rates are frequently inflated by solution leakage, ambiguous issue descriptions and weak test suites, that the field is shifting toward automated validation by model-based consensus because manual cleanup is too costly at scale, and that contamination threatens reliability as benchmarks approach saturation. It also frames SWE-bench as a departure from software generation toward maintenance and evolution.
- Vibe Coding: Practice, Performance, Productivity, and Risk—A State-of-the-Art Review reads the climb on SWE-Bench Verified as qualified by lower scores on contamination-resistant and independently run evaluations.
- Human-In-the-Loop Software Development Agents argues that evaluating functional correctness should go beyond passing unit tests.
- Demystifying evals for AI agents gives SWE-bench Verified as an example of the deterministic, test-based grading that is natural for coding agents.
As the substrate for a different benchmark
- SWE-Review-Bench derives its instances from the 500 issues of SWE-Bench Verified, which supplies the executable tests its metrics depend on.
- RepoComplianceBench reuses SWE-bench’s issue-to-patch substrate and contamination discipline, but measures compliance with contribution rules rather than repair quality.
- FeatureBench is introduced as a harder, feature-focused counterpart to SWE-bench; on a subset restricted to repositories shared with SWE-bench, its paper reports Claude Opus 4.5 resolving 5.2% of tasks, against the 74.40% it lists for that model on SWE-bench Verified.
- SWE-Compass positions itself against SWE-Bench Verified, SWE-Bench Pro and Multi-SWE-bench, on the grounds that they cover fewer languages and, in most cases, only bug fixing.
- LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation argues that SWE-bench and its variants remain terminal — judged by final task success — and builds LoopsBench around dependency structure instead.
- TraceLab: Characterizing Coding Agent Workloads for LLM Serving says capability benchmarks such as SWE-bench contain relatively few narrowly scoped tasks, and so do not capture what shapes the cost of serving coding agents.
As a source of training signal
Critic Model and Learning to Verify AI-Generated Code record that earlier critic models were trained on SWE-bench and SWE-Gym, where unit tests supply verified rewards; OpenHands reports that a critic trained only on benchmark-style data scored an AUC of about 0.45–0.48 on production outcomes. Multi-SWE-RL releases issue-resolving instances as reinforcement-learning training data, and the DeepSWE post casts software-engineering tasks as reinforcement-learning environments, excluding repositories that appear in SWE-Bench-Verified.
As a name used in passing
Some pages use SWE-bench only as a reference point. The OpenHands paper, OpenHands: An Open Platform for AI Software Developers as Generalist Agents, names it as an example of the software-engineering tasks the platform incorporates; OpenHands and One Year of OpenHands: A Journey of Open Source AI Development list accuracy on benchmarks such as SWE-Bench among the project’s aims. The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE places agentic-coding benchmarks such as SWE-bench mainly in its rings 1–3. Building effective agents names a coding agent for SWE-bench tasks as one of Anthropic’s own examples. Magentic-UI: Towards Human-in-the-loop Agentic Systems lists SWE-Bench-style tasks among those it struggles with. Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward covers benchmark progress on SWE-bench in its deployment axis. Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned names SWE-bench evaluation as future work. SWE-Agent’s repository presents SWE-bench alongside the team’s related projects, and the benchmark’s own site groups SWE-Agent, mini-SWE-agent, SWE-smith, SWE-ReX and a SWE-bench CLI with the benchmarks as one family.
Which member each result is reported on
The pages report results on different members, and the member is part of what a figure means. The table lists where each page’s reported result was measured, in that page’s own terms; figures from different rows are not measured under the same conditions, and several pages say so themselves.
Several of these pages carry their own caveat about the figure. The Devstral and Devstral 2 posts’ figures are Mistral’s own; Mini-SWE-Agent’s page notes that the figures it cites are self-reported, on different models and dates; the AutoCodeRover paper compares against SWE-agent’s reported efficacy rather than a run of its own; SpecRover’s baselines are the figures other tools reported; and MASAI’s authors note that SWE-bench Lite’s issues are limited to those that can be validated with tests and are all in English, and that some compared methods used extra inputs such as hints text.